Human Spot-Check Sampling
In addition to automated evaluation, sample a small fraction of live agent outputs for human review on a rolling basis.
This page is the complete instruction page for one pattern called "Human Spot-Check Sampling." It explains the problem this pattern solves, the idea behind the solution, when you should (and should not) use it, and what happens afterward — both the good effects and the costs.
This matters because building AI agents is not just about making them clever. It is also about making them safe and predictable. Following a well-tested pattern like this one helps avoid common mistakes, and shows you exactly which safety rules and regulations it connects to, listed under "Standards Mesh" on this page.
Context
Automated evals catch known failure modes. Humans catch unknowns — subtle tone issues, emergent biases, cultural mistakes, things the evaluator did not think to check for.
Problem
Automated evaluation alone converges on a narrow definition of "good".
Forces
- Review throughput vs agent volume
- Statistical power vs reviewer fatigue
Solution
Sample a stratified slice of live outputs (by risk, by cohort, by confidence). Route to trained reviewers with a structured rubric. Aggregate findings into a quality dashboard and feed new failure modes back into the automated eval suite.
Applicability
- Post-launch agents
- Agents serving diverse user cohorts
- Any agent where "good enough" is partly subjective
Anti-Patterns
- "100% human review" at scale (not scalable)
- Sampling only happy-path outputs
Consequences
- +Catches unknown unknowns
- +Feeds test-suite improvement
- −Staffing cost
- −Sampling bias if not stratified