How to Compare AI Coding Usage Across Engineering Teams

Normalize the work context

At minimum, record team responsibility, repository or service, task type, project maturity, technology, risk, and expected verification. A platform team handling migrations will not have the same trajectory as a product team making routine UI changes.

The goal is not to remove all variation. It is to prevent the most obvious category errors. Compare similar work first, then expand the comparison when the evidence supports it.

Compare distributions and ranges

Team averages hide the shape of work. Use ranges, medians, sample sizes, and representative episodes. A small number of difficult tasks can dominate a mean. A team with consistent moderate usage may have a healthier pattern than a team with a low average and several extreme rework cases.

Always show the time window and the definition of an episode. “Monthly spend per active developer” and “cost per comparable completed task” answer different questions and should not be placed on the same ranking table.

Add workflow and outcome context

After normalizing the work, inspect exploration, retries, context, model selection, verification, review, rework, and result. High spend with strong verification may be expected. Low spend with repeated corrections may deserve more attention. The same total can support different actions.

Compare outcome and quality signals at the team or service level: delivery flow, reliability, review burden, defects, and developer experience. Avoid attributing every difference to AI when staffing, architecture, planning, or incident load also changed.

Example: platform team versus product team

A platform team spends more AI capacity per completed task while migrating a shared deployment system. Its sessions include broad repository exploration, test updates, and extensive verification. A product team spends less on small feature changes but has a higher rate of post-merge corrections.

The correct conclusion is not that the platform team is inefficient. It is that the teams have different work profiles and different signals worth reviewing. The comparison can still identify practices to share, but only after the context is visible.

Use comparison to choose a next action

Good comparisons end with a question or experiment: can a team share a context guide, change a model default, improve requirements, add a verification step, or reduce repeated failure? If no action follows, the dashboard may be creating noise rather than insight.

Compare again after the intervention. A one-time difference is a clue. A recurring pattern with measured change is stronger evidence.

What not to conclude

Do not rank teams by tokens, spend, agent turns, or generated lines without task context. Do not infer that a team with lower usage has better engineering. Do not hide sample size, missing telemetry, or tool differences. Do not treat a correlation as proof that one workflow caused another team’s outcome.

A careful comparison is slower than a leaderboard but more useful for management. It shows where the organization can learn without turning context into an excuse for avoiding measurement.

Sources and further reading

SPACE and DORA support multidimensional, context-aware measurement. GitHub’s metrics API documentation is a practical example of the aggregation and access boundaries that affect team comparisons.

Normalize before you compare

Begin with a common description of the work: task type, service or repository, project phase, risk, team responsibility, and tool configuration. Compare distributions within a reasonably similar group before comparing team averages. Report sample size and missing data beside the result.

If teams work on fundamentally different systems, the honest conclusion may be that a direct ranking is not meaningful. A portfolio view can still show where measurement coverage, cost concentration, or workflow variation deserves attention.

Use comparison to find questions

A team with higher usage may be doing more discovery, supporting a legacy platform, or using an agent for routine work. A team with lower usage may have better context, stricter policy, or less suitable tasks. Treat the difference as a prompt to investigate, not as a performance verdict.

Review comparable examples with the teams involved. Their explanations are evidence about the work and can reveal which normalization fields the next analysis needs.

Report ranges and context

Team comparison is more honest when it shows a distribution, sample size, task mix, and confidence in the classification rather than a single ordered list. A median and range can reveal that two team averages are driven by very different patterns. A note about project phase may explain a temporary shift better than a new ranking.

If the purpose is capacity or budget planning, compare the relevant cost and workload units. If the purpose is enablement, compare workflow patterns. If the purpose is performance management, stop and redesign the question: raw AI activity is not a defensible individual productivity measure.