Evaluation & Testing Reference — Claude Cert Academy

A practical guide to evaluating Claude outputs, building test suites, and maintaining quality as models and prompts evolve.

Evaluation Taxonomy

• Unit evals: test a single prompt on a fixed input set — fast, deterministic, run on every change
• Integration evals: test a multi-step workflow end-to-end — slower, catch emergent failures
• Regression evals: compare before/after a prompt or model change — catch quality degradation
• Adversarial evals: test with tricky inputs, edge cases, and injection attempts — find failure modes

Metric Types

Automatic Metrics

• Exact match: does the output match a reference string exactly? (Good for structured output)
• ROUGE / BLEU: overlap-based metrics for summarization (use as rough signal only)
• JSON schema validation: does the output conform to the expected schema?
• Regex pattern match: does a key value appear in the correct position or format?
• Length check: is the output within the expected token or character range?

LLM-as-Judge Metrics

Use a Claude call to evaluate a Claude output. More flexible than exact match, but adds cost.

eval_prompt = f"""
Evaluate the following customer service response on a scale of 1-5 for:
- Empathy (1-5)
- Accuracy (1-5)
- Clarity (1-5)

Customer issue: {issue}
Agent response: {response}

Return JSON: {{"empathy": N, "accuracy": N, "clarity": N, "reasoning": "..."}}
"""

Human Evaluation

• Use for: high-stakes tasks, nuanced quality judgments, initial benchmark creation
• Structured rubric: define what 1, 3, and 5 look like for each dimension
• Sample size: 50-200 examples for statistically reliable results
• Inter-rater reliability: have 2+ raters for 10% of samples and measure agreement

Golden Dataset Construction

• Start with 50-100 real production examples covering the full input distribution
• Include edge cases: empty inputs, very long inputs, adversarial inputs, ambiguous cases
• Label expected outputs — not just correct/incorrect but the ideal response
• Version the dataset — treat it as code and track changes over time
• Expand over time: add examples for every new failure mode discovered in production

Evaluation Pipeline

# Minimal eval pipeline
import anthropic

client = anthropic.Anthropic()

def evaluate(prompt_template, test_cases, metrics):
    results = []
    for case in test_cases:
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt_template.format(**case)}]
        )
        output = response.content[0].text
        scores = {m: metrics[m](case, output) for m in metrics}
        results.append({"input": case, "output": output, "scores": scores})
    return results

Model Upgrade Evaluation Protocol

Use this protocol when Anthropic releases a new model version.

• Run full golden dataset on both old and new model
• Compare average scores per metric across all test cases
• Flag any case where new model scores more than 10% lower (regression)
• Flag any case where new model scores more than 10% higher (improvement to validate)
• Spot-check 20 flagged cases manually
• Decision threshold: proceed if no critical regressions (accuracy < -5%) on any high-stakes metric

Prompt Change Testing Protocol

• Before any prompt change: snapshot current eval scores as baseline
• Run the changed prompt on the full golden dataset
• Compare delta per metric — accept only if no regression on any critical metric
• For significant changes: run an A/B test with 5-10% of production traffic
• Add new golden examples for any new behavior the prompt is supposed to address

Eval Anti-Patterns

• Testing only happy path: eval suite that only tests normal inputs will miss real failures
• Overfitting to the eval: optimizing prompts for the test set produces brittle systems
• No versioning: eval suite should be version-controlled alongside prompt code
• Manual-only evals: any metric you compute manually should be automated once you have a definition
• Ignoring cost: LLM-as-judge evals add cost; cache judge calls and batch efficiently

Eval Infrastructure Checklist

• CI/CD integration: run evals on every prompt change (like unit tests)
• Results storage: persist eval scores with prompt version and model version for trend analysis
• Alert thresholds: alert when any critical metric drops below a defined floor
• Dashboard: visualize score trends over time per metric and per test case category
• Failure analysis: for every eval failure, record the input, expected output, actual output, and root cause

Continue to Claude Cert Academy