Why this matters
Coding-agent quality is not adequately described by solving one short English problem. Evaluation must also cover context retention, repository conventions, safe tool use, and the tendency to mistake partial completion for finished work.
In environments where instructions are Korean and repositories include Korean documents and conventions, a score from an English public benchmark cannot be treated as product fit. This report addresses how to design the evaluation system before publishing a score.
Key takeaways
- 01
Eight workload classes prevent improvement in one task family from dominating the aggregate.
- 02
A scorer is valid only when it fails on the seed state and passes on the reference solution.
- 03
The 50 SWE-bench Verified instances are a selected external anchor, not a substitute for the full leaderboard.
- 04
Long-session context retention and Korean-instructed work require dedicated A/B harnesses.
Public benchmarks are insufficient in two ways: potential contamination and limited representativeness. Many agent failures occur in context management, tool use, and partial completion rather than algorithm generation.
The corpus is stratified across feature implementation, bug fixing, refactoring, test authoring, migration, documentation synchronization, debugging, and code-review follow-up. Difficulty distributions are maintained by class.
Anti-guess design is a core rule. Tasks should resist solution by contextual guesswork, and every scorer must fail on the seed input and pass on the reference solution.
The context-retention harness measures degradation as sessions grow by comparing retained-context and cold-restart conditions.
The SWE-bench Verified component uses only 50 selected instances. Its results are not directly comparable to the full leaderboard, and corpus counts and distributions must be versioned with future scores.
Decision implications
Organizations selecting a coding agent should require an evaluation portfolio that reflects their repositories, language, session length, and tool permissions instead of relying on one pass rate.
Published results should include task construction, scorer validation, failure definitions, and exclusion boundaries so model or prompt changes can be compared honestly.
Methodology
Development workflows are divided into eight classes. Each task fixes a seed repository and reference solution, and the scorer must satisfy fail-on-seed/pass-on-reference validation.
A selected 50-instance SWE-bench Verified set remains as an external anchor. Long sessions compare retained-context and cold-restart conditions. Internal and public benchmark scores are not merged into one number.
Measurement conditions
- End-to-end workload classes
Classification used in the internal benchmark suite.
- SWE-bench Verified subset
A selected subset with narrower coverage than the full benchmark.
Limitations
This document describes corpus design and validation discipline; it does not publish a Korean performance score for any model. The 50-instance subset is not directly comparable to the full SWE-bench leaderboard.
Internal task distributions do not represent every customer repository and cannot prove the complete absence of contamination. Task versions and execution conditions must accompany every future result.
References
- SWE-bench
Software engineering benchmark built from real GitHub issues.
- SWE-bench research paper
Primary paper describing the benchmark and evaluation.
- Benchmark engineering
Patty evaluation design principles and implementation process.