Evaluating LLM Code Generation: From Pass@K to Sandbox Regression Testing

Early benchmarks for code-generating models (such as HumanEval and MBPP) measured isolated function synthesis against rudimentary unit tests. In real-world enterprise repositories, however, synthetic Pass@1 metrics provide almost zero correlation with whether an autonomous agent can successfully resolve production tickets.

Modern evaluation suites have shifted from single-function Pass@K to Hermetic Repository-Level Regression Testing.

flowchart LR
    Bug["GitHub Issue / Production Alert"] --> Agent["Autonomous Coding Agent"]
    Agent --> Patch["Synthesized Git Patch"]
    Patch --> Sandbox["Isolated Docker Micro-VM"]
    Sandbox --> Build["npm/cargo/mvn Build"]
    Sandbox --> ExistingTests["Pre-existing Test Suite (Regression Guard)"]
    Sandbox --> Reproducer["Fail-to-Pass Repro Test"]
    ExistingTests --> Gate{All Tests Pass?}
    Reproducer --> Gate
    Gate -->|Yes| Accepted["Merge Candidate Approved"]
    Gate -->|No| Rejected["Diagnostic Rollback Triggered"]

1. The Flaws of Static Pass@K

The unbiased Pass@k metric:

$$\text{Pass}@k = \mathbb{E}\left[1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}\right]$$

assumes independent sample evaluations without context or compiler interaction. This creates severe distortion:

• Hallucinated Dependencies: A model may write syntactically elegant code that imports nonexistent packages or outdated API methods.
• Context Window Contamination: In benchmarks where test asserts are visible in prompt prefixes, models frequently overfit to specific mock variables.
• Silent Regressions: A patch may fix the reported symptom while silently breaking adjacent modules.

2. The SWE-Bench Evaluation Paradigm

Enterprise evaluation harnesses (such as SWE-Bench and custom repository sandboxes) evaluate complete Git diffs against full project workspaces:

3. Best Practices for Production Test Harnesses

• Always isolate execution in disposable micro-VMs: Never execute LLM-synthesized code on the host machine.
• Enforce Fail-to-Pass (F2P) and Pass-to-Pass (P2P) dual-gating: A change is only verified if it both resolves the defect and keeps all legacy tests green.
• Track Token Efficiency per Resolved Defect: Model performance must be evaluated not just on accuracy, but on token consumption and Wall-Clock inference cost.