Offline Evaluation Suite (Golden Set)
Maintain a versioned suite of tests (prompts, expected behaviours, edge cases, adversarial inputs) that the agent must pass before any deployment.
This page is the complete instruction page for one pattern called "Offline Evaluation Suite (Golden Set)." It explains the problem this pattern solves, the idea behind the solution, when you should (and should not) use it, and what happens afterward, both the good effects and the costs.
This matters because building AI agents is not just about making them clever. It is also about making them safe and predictable. Following a well-tested pattern like this one helps avoid common mistakes, and shows you exactly which safety rules and regulations it connects to, listed under "Standards Mesh" on this page.
Context
Agents regress silently. A prompt change, a model upgrade, a tool update can break behaviour that worked yesterday. Tests catch regressions before users do.
Problem
Shipping agent changes without an evaluation suite is shipping blind.
Forces
- Test maintenance cost vs regression catch rate
- Coverage breadth vs depth
Solution
Build a golden evaluation set covering: happy-path cases, edge cases, known failure modes, adversarial probes, safety violations. Run on every material change. Track pass-rate per category; block deployment on regressions. Version the test suite alongside the code.
Where this pattern acts in the two lifecycles
02 Reason / Plan. Interpret goal, decompose, select strategy. 13 of 61 catalogue patterns act at this stage.
03 Evaluate & Red-Team. Pre-deployment testing, safety evals. 3 of 61 catalogue patterns act at this stage.
Stages and counts from the catalogue data. The Runtime lens is the loop the agent runs each turn; the AgentOps lens is the lifecycle the team runs the agent through. See the Matrix for every pattern at each stage.
Applicability
- All production agents
- Any agent whose behaviour must be stable across updates
Anti-Patterns
- "We test manually"
- Evaluation set that grows but is never curated
Consequences
- +Regression detection
- +Confidence in model upgrades
- −Authoring and maintaining tests
- −Tests lag behind new behaviours
