Measure AI's impact on engineering by recording a baseline: human-touched share of merged code, workflow per PR, intent-to-deploy time and cost per outcome.


To measure the impact of AI coding tools and agents on an engineering org, you need a dated record of how the system worked before the change and a continuous record of how it works now, per repository and team. The core measures are how much of the merged code a human actually wrote, edited or directed, which workflow produced each PR, how long work takes from intent to deploy, and what each useful outcome costs. Without that baseline, any claim of improvement is a feeling, not a finding.
This is part 1 of a 5-part series on the AI SDLC: how software delivery changes when agents do a growing share of the work, and what engineering leaders need to keep it grounded. We start with measurement because every later part (preserving speed, keeping quality, governing workflows) depends on having a record to compare against.
Every engineering org I talk to has an AI story. "We're 30% faster." "Review time dropped." "Agents write half our code now." Ask one follow-up question: compared to when?
Usually there's no answer. The rollout happened, the tools changed three times, a reorg moved two teams, and nobody captured what the system looked like before. The claim floats. It can't be pinned to a date, a team or a repository, so it can't be defended in a budget review and it can't be used to decide anything.
Floating claims also hide the uncomfortable cases. An org-wide "faster" can be true while one critical service is quietly accumulating code nobody on the team fully understands. If you only have the headline, you never see the service.
The hard part about measuring an AI SDLC is not computing metrics. It is that the raw evidence is unusually perishable. Three things vanish by default:
Six months later you cannot reconstruct any of this. You can only guess, and guesses tend to agree with whoever is presenting them.
You do not need fifty metrics. You need a small number of records that are hard to game and that answer the questions leadership actually asks.
For each merged change, estimate how much a human wrote, meaningfully edited or directly steered, versus what an agent generated and a human accepted as is. Aggregate per repository and per month. This is the single most useful number for understanding ownership: it tells you where the team still knows the code and where it is drifting toward code it merely approved.
For example, a ledger service that was 61% human-touched in March and 31% today is not automatically a problem. It is a fact a leader can act on: maybe review depth on ledger needs to rise, maybe the team needs time to read what was merged, maybe it's fine because the generated code is mostly tests. Without the March number, there is nothing to discuss.
Record whether a PR came from a person working alone, a person pairing with an assistant, an agent run started from an issue, or a multi-step agent workflow. Link the PR back to the agent sessions that produced it. When a checkout change adding multi-currency support causes an incident, the first question is "how was this built?", and you want an answer rather than an archaeology project.
Measure from the moment work was intended (issue created or picked up) through first commit, PR opened, review, merge and deploy. AI usually compresses the build stage and often expands review and validation. A single cycle-time number hides that shift; a stage breakdown shows where the time went.
Token spend per developer is a budget line, not a measure. Attribute agent cost to the PRs and work items it produced, then to outcomes: merged and still in place after 30 days, not reverted, not reworked. A cheap session that produced a reverted PR was expensive.
Alongside these four, track quality signals across all PRs, AI-assisted or not: churn, rework, reverts and change-failure rate. Measuring only AI-tagged work lets problems hide in whatever goes untagged.
The best time to start was before your first agent rollout. The second best time is this sprint. Most orgs are mid-rollout, and that is workable, because a surprising amount of the past is still recoverable.
Within a month you will have a real baseline for everything going forward and a reasonable, clearly labeled reconstruction for the period before.
Lines generated, suggestions accepted and acceptance rate measure tool activity, not engineering outcomes. A high acceptance rate can mean the tool is good or that reviewers stopped reading. Neither number tells you whether the merged code was kept, reverted or rewritten a week later.
If engineers believe the data ranks them, the data will degrade: people will route work around capture or stop using tools openly. Measure the system, aggregate at team and repository level, be explicit about what is collected and why, and never use authorship share as an individual performance score. A low human-touched share is a property of a codebase, not a verdict on a person.
An org-wide average of 55% human-touched can hide one payments repository at 20% and falling. Risk concentrates in specific services, so report per repository and highlight the steepest trends rather than the mean.
If what counts as "human-touched" or "deployed" changes, earlier numbers stop being comparable. Version your definitions and note the date of any change on every chart.
Leaders do not need a dashboard tour. They need a few dated sentences they can act on, each tied to a repository or team:
In March, ledger was 61% human-touched. Today it is 31%, review time per PR is flat, and rework is up. We are adding a second reviewer for ledger and scheduling a code walkthrough.
We built Sparkey to make this record automatic. Lens, a lightweight CLI and hook, captures agent sessions (prompts, tool calls, tokens and cost) and links them to the commits and PRs they produced. Sparkey computes the human-touched share per repository, team and month, tracks quality signals across all PRs, and attributes cost to outcomes. It also backfills what git, PR and CI history can still tell you, so the baseline reaches back before capture started.
Measurement gives you the record. The next question is what that record shows after the first burst of speed fades. Part 2 of this series picks that up: "AI makes it easy to go fast. Staying fast is the hard part."
It is the proportion of merged code that a human wrote, meaningfully edited or directly steered, as opposed to code an agent generated and a human accepted unchanged. It is tracked per repository and month to show where a team still owns its code and where ownership is thinning.
Yes, partially. Git, PR and CI history let you reconstruct cycle time, churn, reverts and change-failure rate for the months before the rollout. Authorship and agent sessions can only be estimated for the past, so label them as estimates and capture them properly from now on.
No. Acceptance rate measures how often suggestions were taken, not whether the resulting code survived review, stayed in place or caused rework. Measure outcomes such as kept changes, reverts and cost per outcome instead.
Aggregate at team and repository level, publish what is collected and why, and never use authorship share to rank individuals. Present the data as a description of how the system works, used to decide where to add review, training or tooling.
Monthly per repository and team is a good default, with a quarterly summary for leadership. The point is to watch trends and slopes, so consistency of cadence and definitions matters more than frequency.
It is agent spend (mostly tokens) attributed to the PRs it produced and divided by the outcomes that held up, such as changes still merged after 30 days without revert or heavy rework. It shows which workflows are worth their cost, not just how much was spent.

Sparkey shows how AI coding tools and agents are changing your teams, your code and your costs.