How to Measure AI Coding Agents
Design a useful way to measure AI coding agents: define the work unit, denominator and evidence before deciding what the metrics can show.
TraceYield Insights
8 min read · Updated October 6, 2026
Start with the decision, not an agent score
To measure AI coding agents, first say what decision the evidence should support: whether access has reached the intended group, where a workflow needs investigation, or whether a defined change in practice is worth continuing. “How well is the agent performing?” is too broad until the work and the meaning of performance are specified.
This page is about measuring specified aspects of agent-assisted work for a defined engineering decision. Prompts, tokens, generated lines, merged pull requests and later incidents are different observations. None is a universal score for an agent, developer or team.
Choose the unit before the measure
An interaction or suggestion can answer a tool-use question. A session can help describe one captured stretch of interaction. A task or work episode can connect activity to a bounded piece of work. A code change or pull request records an artifact and its review disposition. A deployment or service-period supports delivery questions. A team-period can describe aggregate patterns. These units do not map one-to-one: one task may span several sessions, one session may touch several tasks, one pull request may bundle tasks, and several people or agents may contribute to one change.
A proposed comparison unit can be work selected under explicit inclusion rules because its task type, scope, context, risk and observation window are sufficiently similar for the particular comparison. “Sufficiently similar” depends on the decision; this is a practical selection rule, not a validated standard or TraceYield metric. State the rules, exclusions and ambiguous joins. Keep failed, rejected and abandoned attempts visible when the question concerns all attempted work, rather than quietly counting only completed cases.
TraceYield’s product description presents session, project and profile views of development trajectories. It does not establish that every task, session, pull request and deployment can be joined completely. Repository and delivery records may need to come from other systems.
Write a measurement contract
Before comparing a number, record what it literally observes, its unit, denominator or eligible population, observation window and capture coverage. Then name the decision it may inform, the claim it can support, the claim it cannot establish and the main differences that could explain the result. The denominator may mean more than the bottom of a fraction: it can be the eligible population, exposure, sampling frame or observation period.
For accepted suggestions, identify which suggestions were offered, which interface or feature was measured, which users were eligible and when. For an agent completion status, define a task and count eligible attempts, including failures and abandonment; specify independent verification criteria. For merged pull requests, define the eligible PR population and examine size, task mix, repository and contribution mix. For defects, state the affected change population, exposure period, severity and attribution limits.
GitHub’s current documentation defines adoption, engagement, acceptance, lines-of-code and pull-request lifecycle fields, and notes that coverage varies by telemetry source and product surface. Use it to identify what its fields count. Its interpretations of “trust,” “value” or “impact” are vendor interpretations, not independent proof of those outcomes. GitHub’s Copilot metric definitions are a useful example of why coverage and scope belong beside a metric.
Active users
- Unit / denominator
- Unique users meeting the vendor definition / eligible licensed or intended users, in a stated window.
- May support
- Whether recorded use reached a defined population on covered surfaces.
- Cannot establish
- Whether use was useful, sustained across unmeasured tools or productive.
- Context needed
- Seat source, user eligibility, surface coverage, telemetry and period.
Accepted suggestions or generated LoC
- Unit / denominator
- Suggestions shown and accepted under a defined feature; or tool-counted lines in scope.
- May support
- A directional signal of tool interaction or output on that surface.
- Cannot establish
- Useful retained changes, quality, time saved or productivity.
- Context needed
- Feature definition, eligible suggestions, edits, languages, model and window.
Agent-reported completion
- Unit / denominator
- Defined agent attempts / all eligible attempts, including failed or abandoned attempts.
- May support
- How often the agent reported its assigned task complete under that definition.
- Cannot establish
- Verified task success or accepted engineering work.
- Context needed
- Task boundary, completion rule, retries, abandonment and human checks.
Merged PRs or review time
- Unit / denominator
- Eligible pull requests; elapsed time from explicitly defined events.
- May support
- PR disposition and a bounded view of review or repository flow.
- Cannot establish
- Comparable task throughput, quality or an effect caused by AI.
- Context needed
- Task mix, size, queue time, policy, contributors and repository.
Verification and defects
- Unit / denominator
- Named check runs; later confirmed events among changes with defined exposure.
- May support
- Check outcomes for their scope, or a reason to investigate later behavior.
- Cannot establish
- Universal correctness or that AI caused a defect change.
- Context needed
- Assertions, environment, severity, exposure window, change population and attribution.
Developer-reported experience
- Unit / denominator
- Responses to defined questions / eligible respondents in a stated period.
- May support
- Reported friction, confidence or perceived time saving among respondents.
- Cannot establish
- Observed hours saved, productivity change or experience of nonrespondents.
- Context needed
- Question wording, response coverage, task mix and survey timing.
| Metric | Unit / denominator | What it may support | What it cannot establish | Context needed |
|---|---|---|---|---|
| Active users | Unique users meeting the vendor definition / eligible licensed or intended users, in a stated window. | Whether recorded use reached a defined population on covered surfaces. | Whether use was useful, sustained across unmeasured tools or productive. | Seat source, user eligibility, surface coverage, telemetry and period. |
| Accepted suggestions or generated LoC | Suggestions shown and accepted under a defined feature; or tool-counted lines in scope. | A directional signal of tool interaction or output on that surface. | Useful retained changes, quality, time saved or productivity. | Feature definition, eligible suggestions, edits, languages, model and window. |
| Agent-reported completion | Defined agent attempts / all eligible attempts, including failed or abandoned attempts. | How often the agent reported its assigned task complete under that definition. | Verified task success or accepted engineering work. | Task boundary, completion rule, retries, abandonment and human checks. |
| Merged PRs or review time | Eligible pull requests; elapsed time from explicitly defined events. | PR disposition and a bounded view of review or repository flow. | Comparable task throughput, quality or an effect caused by AI. | Task mix, size, queue time, policy, contributors and repository. |
| Verification and defects | Named check runs; later confirmed events among changes with defined exposure. | Check outcomes for their scope, or a reason to investigate later behavior. | Universal correctness or that AI caused a defect change. | Assertions, environment, severity, exposure window, change population and attribution. |
| Developer-reported experience | Responses to defined questions / eligible respondents in a stated period. | Reported friction, confidence or perceived time saving among respondents. | Observed hours saved, productivity change or experience of nonrespondents. | Question wording, response coverage, task mix and survey timing. |
Constructed example: deprecated date parsing
Constructed example — not customer data, TraceYield telemetry or research evidence. A team wants to know whether agent use changes its workflow for small backend maintenance changes involving a deprecated date-parsing dependency.
Weak dashboard reading: prompt count, generated lines and merged PR count rise or fall, so the team concludes that agents made this work faster. Those totals do not reveal which eligible changes were attempted, what was rejected or abandoned, how much work was bundled into each PR, or whether the team’s process changed.
Decision: understand whether the workflow for this bounded maintenance class changed. Eligible work: tickets replacing or removing the deprecated parser for an existing service while preserving its stated date and timezone behavior. Exclusions: new date semantics, data migrations and multi-service redesigns; report how many were excluded and why. Unit: one ticket-level work episode from first recorded work through accepted, rejected or abandoned disposition. Sessions and PRs can be linked where records support the join, with unmatched records left visible.
Evidence and context: retain attempts whether they pass or fail; describe agent and model availability, repository and risk, captured interaction or steering, checks and their results, review disposition, and any later correction within a stated exposure window. This describes the observed work. It does not establish that the agent caused a workflow change or saved time. An unanswered question might be whether developers spent less time on implementation while spending more time on review; that requires a credible comparison of time, not prompt or line counts.
Treat before-and-after results as a prompt to investigate
A pre/post difference may be useful, but it is not by itself an AI effect. Task mix, repository maturity, developer experience, staffing, test maturity, review policy, release cadence, model version, simultaneous process changes, seasonality, learning effects and changed metric definitions may also explain movement.
Segmented views, a longitudinal baseline, within-team comparisons, explicit inclusion rules and carefully matched work can make a comparison easier to interpret. None guarantees causality. State what changed alongside the tool and what evidence would distinguish competing explanations. Describe survey responses about time as developer-perceived time saving; call it observed time saved only when a credible counterfactual supports that claim.
DORA defines measures such as deployment frequency, change lead time, change fail rate, failed deployment recovery time and deployment rework rate at the software-delivery level. These can inform a service or team question; they are not individual coding-agent metrics. DORA’s metric definitions also caution against disparate comparisons. The SPACE paper describes why developer productivity cannot be captured by a single metric; it does not validate this article’s measurement design.
Keep experience and economic claims in their scope
A survey can record what respondents report about friction, confidence or perceived time saving. Treat the question wording, response coverage and period as part of the evidence. Research on developer experience can inform how such constructs are studied, but a local survey is still local evidence. See DevEx in Action for an example of research into developer experience and outcomes.
If the question is whether the broader developer/work system became more productive, use the developer productivity guide. If the question is whether economic benefits exceeded full incremental costs against a credible alternative, use the AI coding ROI guide. Neither conclusion follows from activity or trajectory evidence alone.
What TraceYield can add to the review
TraceYield’s published product description presents development trajectories through recorded session context, project relationships and patterns over time. Where available, those records may help a team examine task framing, agent interactions, steering, retries, corrections and verification around work. Repository, review, deployment and operational outcomes may require other systems; missing records or joins should remain explicit rather than being filled by assumption.
TraceYield does not calculate productivity, prove time saved, rank developers, attribute causality, calculate ROI or establish business impact. For the distinction between evidence states, see AI Coding Agent Output vs. Engineering Outcomes. For a particular change’s completion checks, see Why Agent Completion Is Not Engineering Completion. For quality evidence, see How to Measure AI-Generated Code Quality.