# Measuring AI Impact in Engineering - Metrics That Matter > Which pre-AI baseline to capture, which metrics to report, which vanity numbers to skip, and the quarterly test that flags a queue, not a gain. Source: https://aidlc.guru/measuring-impact/ --- Measurement # Measuring AI Impact the Right Way The core question is not how much code an agent produced. It is whether faster execution is turning into better delivery without degrading quality. A number reported without a pre-AI baseline is a story, not evidence. ![A gauge beside a chart where a rising line pulls away from a dashed baseline, with a ruler beneath](https://aidlc.guru/images/pages/measuring-impact.png) ## Report Against a Baseline, Not a Vibe Every AI-adoption metric is a comparison. "Cycle time is down" or "review time is up" only means something next to a "before" - a number captured while the team was still working without AI, or before the current round of expansion. Skip that step and every later report is a comparison against a memory instead of a measurement. If a clean pre-AI baseline does not exist, say so explicitly in every report rather than back-filling. A missing baseline is a real limitation of the data, and hiding it behind an estimated "before" number turns a measurement report into a guess dressed up as evidence. ## Capture the Baseline First Capture these while the team is still working without AI, or at least before the next expansion of tool usage. A baseline collected after the fact is storytelling, not measurement. 1 ### Cycle Time, Ticket to Production Track both the median (p50) and the 85th percentile (p85). The median hides the tail, and the tail is where queues form. 2 ### PR Review Turnaround Review time in absolute terms, and review time relative to PR size, since a bigger diff should take longer to review honestly. 3 ### Change Failure Rate The share of deploys that need a hotfix, a rollback, or an incident response. 4 ### Bug Escape Rate Defects that reach production per release, counted the same way before and after adoption. 5 ### Test Coverage and Mutation Score Coverage by module, plus a mutation score on the modules that matter most, so a coverage number cannot hide untested logic. 6 ### Deployment Frequency How often the team actually ships to production, independent of how much code was written. 7 ### Developer Satisfaction Survey Perceived productivity, captured now, so a later survey can measure the gap between what people feel and what the other six numbers show. ## Report These Each metric below earns its place on a dashboard because it names a specific failure it would catch. If a metric cannot fail, it is not measuring anything. Metric Why Failure It Detects Cycle time, ticket to production (p50 and p85) The outcome that matters Queue growth hidden behind per-developer output Review latency and review time per line The constraint agents shift rather than remove The verification tax Change failure rate Held flat is the honest claim Speed bought by lowering the bar Escaped defect rate, by tier Whether tiering is calibrated correctly Tier 1 auto-merge scoped too broadly Incidents per deploy Faros AI's 2026 Acceleration Whiplash report found incidents per merged pull request up roughly 3x Unreviewed merges reaching production PRs merged with no human comment, by tier Oversight erosion Rubber-stamped review Rework rate after review Review effectiveness Shallow review that misses real issues Mutation score on critical modules Test quality, not just coverage Test theater - tests that run but do not verify Duplication and cross-file reuse trend Maintainability, per GitClear's 2026 Maintainability Gap report Deferred structural maintenance, in GitClear's own framing Refactoring share of all changes Maintainability, the same signal from a different angle Same as above Agent spend per merged PR, per workflow Unit economics Runaway agent loops Percent of releases with a complete evidence pack A compliance byproduct worth tracking Audit fire drills returning Regulatory or requirement change lead time The product-facing benefit, where it applies \- ## Do Not Report These as Success These are the numbers a leadership deck reaches for first, and the ones most likely to mislead: - AI adoption percentage - AI-authored lines or PRs as a share of total - Suggestions accepted - Tasks completed per developer, alone They go up while quality goes down, and they hand leadership the wrong dashboard. Track adoption for coverage and cost planning, not as an outcome. ![Report: cycle time, PR review turnaround, change failure rate, bug escape rate, deployment frequency. Not success: adoption, AI lines, accepts, tasks.](https://aidlc.guru/images/figures/measuring-impact-report-vs-avoid.png) ## The Quarterly Paradox Test Run this test every quarter. If **PRs per developer are up** and any two of the following are also true, the team has a queue and a risk, not a gain: - Deployments per week flat or down - Median review time up more than PR count - Change failure rate or incidents per deploy up - No-comment merges up More PRs per developer plus two or more of these means execution sped up while delivery and quality did not - a queue and a risk, not a gain. ## The Measurement Stack No single tool produces this dashboard. Six sources feed it, and every number is read against the baseline, broken out per tier and per lane. ### Source Control PR size, review time, comments, no-comment merges, agent identity. ### CI/CD Deploy frequency, failure rate, mutation score, build repairs. ### Incident and Monitoring Incidents per deploy, escaped defects, time to detect and restore. ### Ticketing Cycle time, lane, tier. ### Vendor Usage and Cost Export Spend per workflow and per merged PR. ### Quarterly Survey Perceived productivity, trust in the tooling. Engineering health dashboard Compared against the baseline, reported per tier and per lane ## Where It Fits in AIDLC Measurement is not a phase of its own - it runs alongside all five and reports on how they are performing. [Launch](https://aidlc.guru/phases/launch/) is where most of the raw data originates: deploy frequency, change failure rate, and incidents per deploy all come from what actually reaches production. [Curate](https://aidlc.guru/phases/curate/) owns the loop end to end - it is where the dashboard gets read against the baseline, the quarterly paradox test gets run, and what the numbers show gets fed back into how [Analyze](https://aidlc.guru/phases/analyze/), [Ideate](https://aidlc.guru/phases/ideate/), and [Develop](https://aidlc.guru/phases/develop/) scope the next round of work. ## Related Pages ### The Productivity Paradox The mechanism behind more output and more incidents at the same time. [Read more](https://aidlc.guru/productivity-paradox/) ### AI Code Review at Scale The review layer that determines whether rework rate and rubber-stamping stay under control. [Read more](https://aidlc.guru/code-review/) ### Cost Model and Build vs Buy Where agent spend per merged PR fits into the wider unit economics. [Read more](https://aidlc.guru/cost-model/) ### Rollout Playbook Exit criteria for each rollout phase, built on the same baseline and metrics. [Read more](https://aidlc.guru/rollout-playbook/) ### Maturity Model Why Level 3 and Level 4 require exactly this kind of provable measurement. [Read more](https://aidlc.guru/maturity-model/) ## Frequently Asked Questions Capture a pre-AI baseline covering cycle time (p50 and p85), PR review turnaround relative to size, change failure rate, bug escape rate, test coverage and mutation score on critical modules, deployment frequency, and a developer satisfaction survey. Without this baseline, any later comparison is a story rather than evidence, and the report should say so explicitly instead of back-filling numbers. Adoption percentage, AI-authored line share, suggestions accepted, and tasks completed per developer can all rise while quality falls. These figures measure usage, not outcomes, and reporting them as wins hands leadership the wrong dashboard. Track adoption for coverage and cost planning, and report cycle time, change failure rate, and review latency as the real success signals. The verification tax is the human review time it costs to check AI-generated output before it ships. Faros AI's 2026 Acceleration Whiplash report found median PR review time rose roughly 5x as adoption scaled, even as tasks completed per developer rose too. Reporting review latency next to deployment frequency shows whether speed gains are real or simply shifted onto reviewers. ## Measure Before You Scale A baseline and an honest dashboard turn adoption from a bet into a decision you can defend. [The Rollout Playbook](https://aidlc.guru/rollout-playbook/) [The Curate Phase](https://aidlc.guru/phases/curate/)