Measuring AI Impact the Right Way
The core question is not how much code an agent produced. It is whether faster execution is turning into better delivery without degrading quality. A number reported without a pre-AI baseline is a story, not evidence.
Report Against a Baseline, Not a Vibe
Every AI-adoption metric is a comparison. "Cycle time is down" or "review time is up" only means something next to a "before" - a number captured while the team was still working without AI, or before the current round of expansion. Skip that step and every later report is a comparison against a memory instead of a measurement.
If a clean pre-AI baseline does not exist, say so explicitly in every report rather than back-filling. A missing baseline is a real limitation of the data, and hiding it behind an estimated "before" number turns a measurement report into a guess dressed up as evidence.
Capture the Baseline First
Capture these while the team is still working without AI, or at least before the next expansion of tool usage. A baseline collected after the fact is storytelling, not measurement.
Cycle Time, Ticket to Production
Track both the median (p50) and the 85th percentile (p85). The median hides the tail, and the tail is where queues form.
PR Review Turnaround
Review time in absolute terms, and review time relative to PR size, since a bigger diff should take longer to review honestly.
Change Failure Rate
The share of deploys that need a hotfix, a rollback, or an incident response.
Bug Escape Rate
Defects that reach production per release, counted the same way before and after adoption.
Test Coverage and Mutation Score
Coverage by module, plus a mutation score on the modules that matter most, so a coverage number cannot hide untested logic.
Deployment Frequency
How often the team actually ships to production, independent of how much code was written.
Developer Satisfaction Survey
Perceived productivity, captured now, so a later survey can measure the gap between what people feel and what the other six numbers show.
Report These
Each metric below earns its place on a dashboard because it names a specific failure it would catch. If a metric cannot fail, it is not measuring anything.
| Metric | Why | Failure It Detects |
|---|---|---|
| Cycle time, ticket to production (p50 and p85) | The outcome that matters | Queue growth hidden behind per-developer output |
| Review latency and review time per line | The constraint agents shift rather than remove | The verification tax |
| Change failure rate | Held flat is the honest claim | Speed bought by lowering the bar |
| Escaped defect rate, by tier | Whether tiering is calibrated correctly | Tier 1 auto-merge scoped too broadly |
| Incidents per deploy | Faros AI's 2026 Acceleration Whiplash report found this figure up roughly 3x | Unreviewed merges reaching production |
| PRs merged with no human comment, by tier | Oversight erosion | Rubber-stamped review |
| Rework rate after review | Review effectiveness | Shallow review that misses real issues |
| Mutation score on critical modules | Test quality, not just coverage | Test theater - tests that run but do not verify |
| Duplication and cross-file reuse trend | Maintainability, per GitClear's 2026 Maintainability Gap report | Deferred structural maintenance, in GitClear's own framing |
| Refactoring share of all changes | Maintainability, the same signal from a different angle | Same as above |
| Agent spend per merged PR, per workflow | Unit economics | Runaway agent loops |
| Percent of releases with a complete evidence pack | A compliance byproduct worth tracking | Audit fire drills returning |
| Regulatory or requirement change lead time | The product-facing benefit, where it applies | - |
Do Not Report These as Success
These are the numbers a leadership deck reaches for first, and the ones most likely to mislead:
- AI adoption percentage
- AI-authored lines or PRs as a share of total
- Suggestions accepted
- Tasks completed per developer, alone
They go up while quality goes down, and they hand leadership the wrong dashboard. Track adoption for coverage and cost planning, not as an outcome.
The Quarterly Paradox Test
Run this test every quarter. If PRs per developer are up and any two of the following are also true, the team has a queue and a risk, not a gain:
- Deployments per week flat or down
- Median review time up more than PR count
- Change failure rate or incidents per deploy up
- No-comment merges up
More PRs per developer plus two or more of these means execution sped up while delivery and quality did not - a queue and a risk, not a gain.
The Measurement Stack
No single tool produces this dashboard. Six sources feed it, and every number is read against the baseline, broken out per tier and per lane.
Source Control
PR size, review time, comments, no-comment merges, agent identity.
CI/CD
Deploy frequency, failure rate, mutation score, build repairs.
Incident and Monitoring
Incidents per deploy, escaped defects, time to detect and restore.
Ticketing
Cycle time, lane, tier.
Vendor Usage and Cost Export
Spend per workflow and per merged PR.
Quarterly Survey
Perceived productivity, trust in the tooling.
Where It Fits in AIDLC
Measurement is not a phase of its own - it runs alongside all five and reports on how they are performing. Launch is where most of the raw data originates: deploy frequency, change failure rate, and incidents per deploy all come from what actually reaches production. Curate owns the loop end to end - it is where the dashboard gets read against the baseline, the quarterly paradox test gets run, and what the numbers show gets fed back into how Analyze, Ideate, and Develop scope the next round of work.
Frequently Asked Questions
Capture a pre-AI baseline covering cycle time (p50 and p85), PR review turnaround relative to size, change failure rate, bug escape rate, test coverage and mutation score on critical modules, deployment frequency, and a developer satisfaction survey. Without this baseline, any later comparison is a story rather than evidence, and the report should say so explicitly instead of back-filling numbers.
Adoption percentage, AI-authored line share, suggestions accepted, and tasks completed per developer can all rise while quality falls. These figures measure usage, not outcomes, and reporting them as wins hands leadership the wrong dashboard. Track adoption for coverage and cost planning, and report cycle time, change failure rate, and review latency as the real success signals.
The verification tax is the human review time it costs to check AI-generated output before it ships. Faros AI's 2026 Acceleration Whiplash report found median PR review time rose roughly 5x as adoption scaled, even as tasks completed per developer rose too. Reporting review latency next to deployment frequency shows whether speed gains are real or simply shifted onto reviewers.
Measure Before You Scale
A baseline and an honest dashboard turn adoption from a bet into a decision you can defend.