Agent task execution runtime

A task executor built as a foundation for agent runtimes / harnesses, focused on concurrency control, timeouts, cancellation propagation, child process group cleanup and pipe deadlocks when an agent calls local commands and tools.

The project is developed in collaboration with AI: I own the semantic boundaries, the acceptance matrix, failure reproduction and part of the root-cause analysis (and wrote the combined fault injection regression tests myself); a coding agent does most of the implementation and built the mutation testing tool.

Six core problems solved

1. Pressing Esc twice, the second cancellation doesn’t take effect (about 1.7 seconds of extra waiting)
   • Found: in the acceptance matrix the design intent expected an exit in 0.6 seconds, but it actually took 2.31 seconds.
   • Root cause: once Python’s asyncio.TaskGroup sets _aborting on the first cancellation, it no longer forwards later cancellations to child tasks; the existing unit tests called the low-level function directly and bypassed the batch composition layer.
   • Fix: the batch executor now awaits child tasks itself and forwards every external cancellation (without forwarding again on an internal fail-fast); after the fix it measured 0.62 seconds.
2. After one timeout an indexing task never runs again (a stale lock file)
   • Root cause: on timeout SIGKILL was sent straight away, so the tool’s cleanup that deletes its lock file never got to run.
   • Fix: send SIGTERM to the whole process group first, return as soon as the group is empty within the grace period, and send SIGKILL to whatever is still running when it expires.
3. Esc shows “cancelled”, but the background process keeps writing
   • Root cause: the cancellation exception is raised directly at the await, and the original code only caught TimeoutError, so all cleanup was skipped.
   • Fix: process group cleanup runs in a separate task protected by asyncio.shield; if another cancellation arrives during cleanup the grace period is cut short and escalated to SIGKILL; the cancellation is re-raised only after the cleanup result is written to the record and handed to the caller.
4. A failed build recorded as a success, and cleanup status misreported
   • Root cause: the scheduling layer and the executor each kept their own timers and did their own cleanup, and only checked whether an exception was raised, not the child process’s return code.
   • Fix: the executor is solely responsible for timing and cleanup, and the scheduling layer only translates results; the task result (state) and the resource cleanup outcome (cleanup) are recorded as two independent dimensions, and a backstop timer identifies a hang in the executor itself.
5. Deadlock on large output, or a background descendant making the read wait forever for EOF
   • Root cause: waiting for the process to exit before reading its output lets the OS pipe fill up and deadlock; for commands like sh -c "sleep 20 & echo ok", the background child inherits the write end of the stdout pipe, so EOF never arrives.
   • Fix: the direct child’s exit marks the end of the command; output is read continuously during execution, keeping only the last 64 KiB; the whole process group is cleaned up afterwards; if a descendant escapes the process group with setsid and holds the pipe, reading is abandoned after at most 0.5 seconds and cleanup is honestly reported as INCOMPLETE.
6. How do you prove the tests actually catch bugs?
   • Combined fault injection: tests built from “never-seen combinations of failures” — for example a tool with slow cleanup releasing an escaping process during cleanup (F1), and 12 tasks that ignore SIGTERM queued in three waves with Esc pressed twice midway through the second wave (F2).
   • Mutation testing: a self-built, targeted mutation testing script runs all 114 tests under Python 3.14, and all 45 code mutants are detected (0 survive).
   • Sandboxed code review: a small local model (qwen2.5-coder 7B) wrote cleanup logic from the first draft of the requirements; it was run in a user + PID namespace sandbox and reviewed by hand to find logic holes the tests did not cover.

Current status and known limits

• Status: wrapping up v0.1 (private repository; the full source and ADR records can be shared for interviews or conversations).
• Known limits:
  • descendants that escape the process group with setsid can currently only be reported honestly as INCOMPLETE; Linux cgroup v2 is planned for forced cleanup;
  • pure Python coroutine tools cannot be force-terminated the way subprocesses can (untrusted tools must run as subprocesses);
  • task records are kept in memory for now; SQLite persistence and restart recovery are planned for v0.2.