Online A/B & Shadow Evaluation
Compare agent variants in production using A/B, shadow, or canary deployments with statistical rigour, not anecdotes or vibe-checks.
This page is the complete instruction page for one pattern called "Online A/B & Shadow Evaluation." It explains the problem this pattern solves, the idea behind the solution, when you should (and should not) use it, and what happens afterward, both the good effects and the costs.
This matters because building AI agents is not just about making them clever. It is also about making them safe and predictable. Following a well-tested pattern like this one helps avoid common mistakes, and shows you exactly which safety rules and regulations it connects to, listed under "Standards Mesh" on this page.
Context
Offline evaluation captures known-good behaviour; production reveals the unknown-unknowns. Changes to prompts, models, or tools need production evidence before full rollout.
Problem
Shipping changes on offline evidence alone misses real-user failure modes.
Forces
- Speed of iteration vs statistical rigour
- User-visible risk vs validation quality
Solution
Deploy variants in shadow mode (A runs, B observes) or A/B (split traffic). Define primary metrics (task success, latency, cost) and guardrail metrics (refusal rate, escalation rate, safety violations). Reach statistical significance before rollout. Document the result regardless of outcome.
Where this pattern acts in the two lifecycles
05 Observe. Tool result ingestion, state update. 7 of 61 catalogue patterns act at this stage.
05 Operate & Monitor. Runtime telemetry, drift detection. 18 of 61 catalogue patterns act at this stage.
Stages and counts from the catalogue data. The Runtime lens is the loop the agent runs each turn; the AgentOps lens is the lifecycle the team runs the agent through. See the Matrix for every pattern at each stage.
Applicability
- High-volume agents
- Any agent where small changes can meaningfully affect outcomes
- Continuous-improvement programmes
Anti-Patterns
- Ship-and-see without instrumentation
- Ignoring guardrail metrics once primary metric improves
Consequences
- +Evidence-based rollout
- +Catches regressions automatic evals miss
- −Experiment infrastructure required
- −Statistical discipline required
