Series II: From infrastructure to agentic projects in production. Episode 11 of 20.
A coding agent can impress in a demonstration yet produce changes that are difficult to accept in a company's project. For the series' fictional B2B portal, the decision is concrete: which tasks can we delegate, and with which checks? Answering requires a project evaluation set with observable results, a work budget and someone responsible for acceptance.
In the episode on shared memory, we established how to retain confirmed decisions. We now use those decisions as controlled context. We evaluate the agents that develop the portal; any agents integrated into the application will have their own evaluations, focused on users' actions.
What a score tells us and what needs local verification
A benchmark is a set of tasks and a measurement protocol used for comparisons. “State of the art”, meaning leading performance, is meaningful only when we specify the task, protocol and comparison date. A public score can justify investigating a tool, but does not by itself establish what is worth delegating in our portal.
SWE-bench applies generated changes to real repositories, meaning code stores, and runs tests in Docker, a platform for running applications in containers. The result describes task resolution through those tests. The UTBoost paper, published at ACL in July 2025, identified changes in its evaluation that insufficient tests accepted even though the issue remained unresolved. The practical conclusion is that the strength of the evidence also depends on the evaluator. The study does not demonstrate that all scores or all generated changes are wrong.
We evaluate the complete configuration: the model, instructions, context, tools and environment. The system that connects the model to tools and manages execution is sometimes called a harness. Changing it can affect results even when the model stays the same. Local comparison therefore starts with an identifiable configuration, rather than the agent's commercial name.
An initial set for the B2B portal
We propose six tasks inspired by the portal's functions. This is an evaluation scenario, with no results attributed to a client. Each task starts from a known code version and has a description, synthetic data, expected behaviour and acceptance criteria prepared before execution.
Task | Minimum evidence for acceptance |
|---|---|
Validating a request form | Valid data is accepted; missing mandatory fields produce useful messages. |
Correcting access to documents | The owning organisation can read them; another organisation is denied access. |
Paginating the request list | Ordering is stable; the fixed data produces no omissions or duplicates. |
Exporting a CSV report, a tabular file with values separated by delimiters | Columns, diacritics and values containing the delimiter are preserved correctly. |
Refactoring a component, meaning reorganising code without changing its behaviour | Existing behaviour remains verified, without changing the public contract. |
Investigating an incompletely described error | The agent reproduces the issue or requests the necessary information before making a speculative change. |
The set combines ordinary cases, boundaries and behaviours that must be rejected. Six tasks are a proposed starting point, not a sufficient sample for general statistical conclusions. We extend the set with relevant issues found during work. We separate tasks used to adjust instructions from those reserved for final verification, to avoid optimising exclusively for examples already seen.
Before comparison, a developer prepares a reference solution and confirms that the evaluator accepts it. They also check an intentionally incorrect version, which must be rejected. The criteria describe the required behaviour without unnecessarily demanding the same implementation as the reference. For pagination, for example, we specify the order to preserve and what happens on the last page. The agent can choose a different solution if it respects the contract. A new requirement discovered in review becomes part of a new task version, rather than a retroactive penalty.
Verification must be independent
An evaluation compares the actual result with the established criteria. The message “I fixed the issue” is not evidence. We run the code, inspect the resulting data and examine the code difference, called a diff. Acceptance tests are managed separately from the agent being evaluated; it can propose additional tests, but cannot weaken the official checks to obtain a favourable result.
A regression is the deterioration of behaviour that previously worked. Alongside the test for the new requirement, we run relevant checks for existing functions. We add type analysis, style checks and security controls appropriate to the project. Passing them does not demonstrate the absence of every defect. The human reviewer checks readability, adherence to the architecture and any unnecessary changes.
An AI evaluator can flag issues with explanations or structure using an explicit rubric. We periodically compare its verdict with a specialist's. Anthropic's guide of 9 January 2026 recommends combining code, model and human evaluators according to the task. For the document access rule, the test result and implementation review remain mandatory; a favourable style score does not compensate for unauthorised access.
Fair comparisons and repeated attempts
We record the code version, model, tool version, instructions, context sources and available memory. We fix time and resource limits and the policy for human interventions. We use an isolated environment, synthetic data and test credentials. Each attempt starts clean, without solutions or conversations left over from the previous attempt.
Reproducibility means that someone else can repeat the protocol and understand the result; it does not promise an identical answer on every run. We propose a few attempts for each task, with the same budget, reporting all results. Retaining only the best attempt would hide the variation the team will encounter.
We distinguish an incorrect change from a test that cannot run because of the environment. Anthropic's publication of 5 February 2026 shows, in its own experiments, that resources and enforcement of limits can influence scores. Our report also retains the total number of attempts and infrastructure errors; we do not remove them without explanation. If a requirement is ambiguous or the evaluator rejects a valid solution, we correct the protocol and reevaluate the affected configurations.
The record supporting the delegation decision
For each task, the record documents the accepted or rejected result, tests, regressions, interventions, execution time, review time and usage reported by the tool. Cost is calculated from actual usage and applicable commercial terms. We track the time to an accepted change, including corrections, rather than only the speed at which the first solution appears.
We compare configurations on the same tasks and against the team's current workflow. A configuration may be suitable for form validation while still unsuitable for authorisation changes. The useful decision is a list of tasks that can be delegated with the necessary checks, rather than blanket approval of the agent. An access issue between organisations blocks acceptance of that change regardless of the aggregate result.
This episode's outcome is a small evaluation set and a reusable acceptance record. We revisit them when the model, tools or instructions change and retain the configuration history. The next episode adds checks on dependencies and component provenance: working code is only one part of the delivery decision.
Sources
Documentation and papers verified on 12 October 2026:
- SWE-bench, Evaluation Guide, for evaluating changes and interpreting results.
- Anthropic, Demystifying evals for AI agents, 9 January 2026, an engineering guide to evaluators and repeated attempts.
- Yu and colleagues, UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench, ACL, July 2025, for the limitations of insufficient tests in the tasks investigated.
- Anthropic, Quantifying infrastructure noise in agentic coding evals, 5 February 2026, for the effect of execution conditions in the authors' experiments.
Want to test coding agents on tasks representative of your product? Talk to the i8 team about acceptance criteria and a verifiable pilot.
