A timeout that stops watching is not a timeout.

During a large automated test batch, one job hit its seven-minute polling deadline. The runner marked it failed and moved to the next item, exactly as written. Ninety minutes later we noticed the job was still running — it was recording a live stream, and it would have kept recording until the stream ended. The timeout had stopped our watching. It had not stopped the work.

Abandonment is not abortion

The bug is embarrassingly common once you look for it. A supervisor process starts a remote job, polls for completion, and gives up after a deadline. Giving up ends the supervisor's involvement — the poll loop, the log line, the status update. The remote job never hears about any of it. From the platform's point of view nothing changed: a client stopped asking questions, which clients do all the time.

For most jobs the distinction is invisible, because most jobs finish on their own a few seconds after the deadline anyway. It becomes visible at the worst possible time: on the job whose whole nature is to run indefinitely. A stream recorder does not converge. It records until the stream ends or someone kills it, and we had just guaranteed that nobody was watching to kill it.

The fix is one API call, which is the point

The correction was almost insultingly small: when the deadline passes, call abort before walking away. One request. The gap between the broken and correct versions was never effort — it was noticing that "timeout" quietly meant two different things, and that we had implemented the cheap one.

We also swept every recent job on both accounts for strays and found none, which was luck rather than design. The recorder had been the only unbounded job in a batch of a hundred and forty; a batch with five of them would have been five times the bill.

Unbounded work needs a different contract

The deeper lesson sits upstream of the bug. Jobs divide into those that terminate on their own — fetch, transform, respond — and those that run until stopped. The second kind should never enter an unattended pipeline under the first kind's contract. Our batch runner now refuses jobs it recognizes as unbounded, and the two stream recorders in our catalog are excluded from automated testing entirely, with the reason written next to the exclusion: an honest test of "record until the stream ends" costs the length of the stream.

That refusal is itself the general form: automation should decline work whose cost it cannot bound, rather than accept it and rely on a watcher that might stop watching.

Supervision is part of the job

We write a lot about systems that report success without doing the work. This was the mirror image — a system doing work after we stopped listening — and it completes the symmetry: a supervisor owes its jobs the same guarantees it demands from them. Every claim of "this batch is done" now implies a second claim, checked against the platform rather than our own bookkeeping: nothing we started is still running.

← All notes