AI-Assisted Software Delivery Evidence Review 2026

Executive Summary

The question “Does AI make developers faster?” is too coarse. Studies measure different tasks, people, tools, time periods, and endpoints. A clearly delimited JavaScript task, real issues handled by experienced Open Source maintainers, organization-wide delivery metrics, and vendor telemetry on pull requests are not interchangeable experiments.

This report therefore does not assess a universal productivity effect. It classifies eleven evidence building blocks by study design and endpoint and connects them to an operating model for review capacity, Secure Software Development, and cost per accepted production change.

The evidence is heterogeneous:

• A randomized experiment on a bounded task reported a substantial speedup.
• The METR field experiment with experienced maintainers in familiar repositories found a 19 percent slowdown for early-2025 tools; a 2026 update showed weak, selection-affected indications of improvements with newer tools.
• DORA found organization-wide associations that caution against equating adoption with delivery performance.
• A longitudinal 2026 enterprise case with 802 developers and 196,212 pull requests reached 2.09 times the earlier per-capita throughput, while the review load per reviewer grew roughly in step. Because adoption was not randomized, this is not a universal causal figure.
• Security and code quality studies show both room for improvement and relevant weaknesses, depending on task, tool, tests, and review.

The operational conclusion is not “avoid AI code”. It is: generation capacity must not be scaled faster than specification, review, tests, security, and production feedback. The correct economic metric is not Lines of Code or PR Count, but cost and lead time per accepted, stable production change.

Citable key findings

1. [OBSERVED] The 2025 METR RCT reported 19 percent longer completion time in its specific setting when AI was permitted, even though participants subjectively perceived a speedup.
2. [OBSERVED] METR’s 2026 update provided weak indications of a possible speedup for newer tools, but names selection and participation problems that prevent a strong interpretation.
3. [OBSERVED] The longitudinal enterprise case by He et al. analyzes 802 developers and 196,212 pull requests from January 2024 to April 2026; per-capita throughput reached 2.09 times the pre-mandate value, while the review load rose roughly in step.
4. [INTERPRETATION] More output per developer is not the same as more value, quality, or stability per unit of time.
5. [OBSERVED] The frequently cited figure of 8.1 million pull requests belongs to the LinearB benchmark corpus; the +242.7 percent incident presentation comes from a separate Faros vendor report. The sources must not be conflated.
6. [INTERPRETATION] Review is a finite service capacity. If the PR arrival rate grows faster than substantive examination, then queue age, superficial approvals, or the automated review share increase.
7. [RECOMMENDATION] AI-assisted changes require the same or stricter risk-based gates as human-written code: small batches, tests, SAST, dependency review, secret scanning, independent approval, canary, and observability.
8. [RECOMMENDATION] Companies should measure AI impact with controlled local rollouts and outcome metrics, not through self-assessment or generated lines.

---

1. Review question and protocol

Which robust statements can be made as of July 2026 about the influence of AI-assisted development on throughput, time, review, quality, security, and delivery?

1.1 Inclusion criteria

• publicly traceable methodology;
• a concrete sample or corpus description;
• a measurable endpoint;
• documented limitations;
• direct relevance to software generation, review, or delivery;
• primary source, or transparently classified as vendor telemetry.

1.2 Evidence grades

Grade	Type	permissible language
A	randomized controlled experiment	“caused in this setting”, with scope
A-	randomized, but with relevant selection/participation problems	“weak/uncertain evidence”
B	quasi-experimental or controlled empirical benchmark	“is associated with / showed in the design”
C	observational data, vendor telemetry, descriptive corpus	“reported / observed”; no universal causality

1.3 Primary endpoints

• completion time;
• accepted throughput;
• review waiting and handling time;
• rework and reverts;
• change failure and recovery;
• security findings;
• test quality;
• cost per accepted production change.

---

2. Structured evidence review

2. Review protocol

2.1 Inclusion criteria

Sources were included if they meet at least one of the following:

• randomized or controlled comparison group
• quasi-experimental identification strategy
• real engineering telemetry with a stated sample
• empirical security/quality analysis with a reproducible unit
• official benchmark corpus with a clear sample size

2.2 Exclusion criteria

The following were not used as core evidence:

• pure vendor testimonials
• surveys without reference to observable outcomes
• anecdotal “10x developer” claims
• model benchmarks without a human development process
• individual examples without a comparison condition
• metrics without a traceable source

2.3 Evidence grades

Grade	Design	Interpretation
A	randomized controlled experiment	high internal validity for the setting studied; assess external generalization separately
A-	randomized design with substantial selection/measurement problems	causal idea strong, concrete effect size uncertain
B	quasi-experimental or controlled technical benchmark	robust for the defined population/endpoint, but with confounding or transfer limits
C	observational telemetry or descriptive benchmark	real-world breadth, but no reliable causality
D	self-report, vendor claim without sufficiently verifiable methodology	hypothesis generation, not a robust effect

[CRITICAL] Vendor funding does not automatically lead to a downgrade. It is documented as a conflict of interest and a replication limit. Likewise, an independent preprint is not automatically causally robust.

---

---

3. Evidence matrix

ID	Source	Design	Sample	reported key finding	Grade
E01	Peng et al. / Microsoft Research & GitHub	randomized controlled experiment	recruited developers; HTTP server task in JavaScript	treatment group completed the clearly delimited task 55.8 % faster	A
E02	METR	randomized controlled field experiment	16 experienced OSS developers, 246 real issues in familiar repositories	permitting AI led to 19 % longer completion time in this setting; developers had expected a 24 % speedup beforehand and perceived a 20 % speedup afterwards	A
E03	METR update	continued randomized experiment with selection problems	returning and newly recruited OSS developers; late-2025 tools	returners: estimated 18 % speedup, CI from 38 % speedup to 9 % slowdown; new developers: 4 % speedup, CI from 15 % speedup to 9 % slowdown	A-
E04	DORA / Google Cloud	observational organization-wide analysis	software development organizations in the DORA data set	25 % higher AI adoption was associated with 1.5 % lower throughput and 7.2 % lower stability	C
E05	Song, Agarwal & Wen	quasi-experimental OSS analysis with proprietary Copilot usage data	collaborative Open Source projects and developers	+5.9 % project contributions, +3.4 % participation, +2.1 % individual productivity, and +8 % coordination time	B
E06	Faros AI	observational engineering telemetry / before-after comparison	two years of telemetry, 22,000 developers, 4,000 teams	vendor reports +51 % PR size, +28 % bugs/PR, 5x median review time, 3x incidents/PR, and +242.7 % incidents/PR in one presented high-adoption analysis	C
E07	GitHub	randomized controlled code quality experiment	202 valid submissions from experienced Python developers; 1,293 blind reviews	Copilot access increased the probability of passing all ten tests by 53.2 %; several quality metrics improved slightly	A
E08	Pearce et al.	controlled security prompt benchmark	89 scenarios, 1,689 generated programs	around 40 % of the generated programs were classified as vulnerable	B
E09	Fu et al.	empirical analysis of real Copilot code snippets	733 snippets from GitHub projects	29.5 % of the Python and 24.2 % of the JavaScript snippets affected; 43 CWE categories; warning messages enabled up to 55.5 % remediation	B
E10	LinearB	large descriptive benchmark corpus	8.1+ million pull requests from 4,800+ organizations/teams in 42 countries according to the vendor	corpus demonstrates a large underlying data set; vendor frames code review as the bottleneck between AI adoption and measurable engineering impact	C

Machine-readable version:

data/FDR-2026-07_ai_delivery_evidence_matrix.csv

---

---

4. Why the results appear contradictory

4.1 Different tasks

A clearly specified API or HTTP server task is not the same work as an issue in a codebase of millions of lines grown over years. AI benefits from:

• explicit requirements
• short context windows
• known framework patterns
• automatically verifiable results
• little implicit domain logic

It is strained more by:

• local conventions
• unwritten architecture rules
• historically grown dependencies
• non-functional requirements that are hard to measure
• review, documentation, and migration obligations

4.2 Different users

A beginner, a productive generalist, and a long-standing maintainer of a highly complex codebase have different bottlenecks. AI can strongly compensate for missing syntax knowledge, while an expert spends time on prompting, checking, and correcting that would not have been needed with familiar code.

4.3 Different endpoints

faster first draft
≠ faster accepted pull request
≠ faster merge
≠ faster production
≠ fewer incidents
≠ higher business value

The studies each measure different parts of this chain. A positive effect at the start can be consumed by review and rework. Conversely, a slower individual task can lead to better knowledge transfer or wider reach.

4.4 Tool and time dependence

Models, agents, IDE integration, and usage patterns change within a few months. A 2021 security benchmark matters as a risk pattern, but is not a direct estimate for a model in July 2026. METR’s opposing early and later results show exactly this dynamic, and at the same time how hard it is to measure cleanly.

---

---

5. Evidence by outcome

5.1 Individual task speed

The strongest positive figure comes from the controlled HTTP server experiment: 55.8 percent faster. The strongest negative controlled figure comes from METR’s real OSS issues: 19 percent slower.

Both can be correct at the same time. The first setting examines a bounded task. The second examines experienced developers in familiar, large repositories with implicit standards.

[INTERPRETATION] An organization must not adopt an external uplift as a forecast without segmenting its own task distribution.

5.2 Project output and coordination

The OSS analysis by Song, Agarwal, and Wen reports:

• +5.9 % project contributions
• +3.4 % participation
• +2.1 % individual productivity
• +8 % coordination time

This is a central mechanism: AI can increase the number of contributors and changes while joint coordination becomes slower.

5.3 Delivery throughput and stability

For 25 percent higher AI adoption, DORA reports an association with:

• 1.5 percent lower delivery throughput
• 7.2 percent lower delivery stability

The data do not prove causality. They are, however, consistent with a queueing logic: larger batches and more changes strain review and integration.

5.4 Code quality

GitHub’s controlled experiment with 202 valid submissions reports a 53.2 percent higher probability of passing all ten unit tests. In addition, small, statistically significant improvements in readability, reliability, maintainability, and conciseness were reported.

The correct interpretation is not “AI code is of higher quality”. It is:

In this standardized task and sample, the treatment group produced better measured results on average.

5.5 Security

Pearce et al. generated 1,689 programs in 89 security-relevant scenarios and classified around 40 percent as vulnerable. A later analysis of real Copilot snippets found weaknesses in 29.5 percent of the Python and 24.2 percent of the JavaScript snippets; up to 55.5 percent of the detected problems could be remediated with concrete warning messages through Copilot Chat.

[INTERPRETATION] AI produces security risk and security remediation at the same time. Without scanners, tests, and domain review, the direction remains undetermined.

5.6 Engineering telemetry

For 22,000 developers and 4,000 teams, Faros reports among other things:

• +51 % PR size
• +28 % bugs per PR
• 5x median review time
• 3x incidents per PR
• 10x code churn
• +242.7 % incidents per PR in one visualized high-adoption analysis

These values matter, but as vendor telemetry with raw data and models that are not fully open, they are grade C. The report describes the findings as reported by Faros, not as independently verified causality.

---

---

6. The LinearB/Faros attribution correction

Two figures frequently cited together come from different sources:

• 8.1+ million pull requests and 4,800+ organizations/teams: LinearB Software Engineering Benchmarks Report 2026.
• +242.7 percent incidents per PR: visualized analysis in the Faros AI report “The Acceleration Whiplash”.

[CRITICAL] The incident figure must not be attributed to LinearB. Likewise, the size of the LinearB corpus does not prove the Faros causality claim.

In this review, LinearB is recorded as a large descriptive benchmark corpus. Its vendor thesis that code review becomes the bottleneck is consistent with several other sources. Consistency, however, is not a substitute for identical measurement or independent replication.

---

---

7. An economic model of the review bottleneck

7.1 Production and inspection rate

λ = volume of arriving changes per unit of time
μ = volume of reviewed and integrated changes per unit of time

As long as λ < μ, the queue remains controllable. As λ approaches review capacity, waiting time rises disproportionately. If λ permanently exceeds μ, open PRs, context loss, and risk pressure grow.

AI can increase λ through:

• more PRs
• larger PRs
• more parallel tasks
• more developers/agents making contributions

Review capacity μ does not increase automatically. It depends on:

• reviewer time and domain knowledge
• test automation
• architecture boundaries
• observability
• PR size
• change risk
• quality of the specification

7.2 Hidden work

AI output creates additional work that does not appear as “coding time”:

• validate context
• detect hallucinations
• dependency and license checks
• security scanning
• write or correct tests
• identify edge cases
• rebase and merge conflicts
• incident and rework costs

A productivity measurement that counts only author time shifts this work onto reviewers and operations.

---

---

8. Measurement framework for companies

8.1 Four levels

Level 1 – Task

• completion time
• accepted output
• correction loops
• tool cost

Level 2 – Pull request

• size and complexity
• pickup/review time
• number of reviewers
• review comments
• rework/churn
• test and security findings

Level 3 – Delivery

• Lead Time for Changes
• Deployment Frequency
• Change Failure Rate
• Failed Deployment Recovery Time
• Batch Size

Level 4 – Business

• delivered functionality
• usage/revenue impact
• support and incident costs
• maintainability
• compliance/audit effort

8.2 Core formulas

Accepted AI Contribution Rate
= accepted AI-influenced changes / AI-influenced changes

Review Amplification
= additional review and rework time / saved author time

Stable Delivery Yield
= changes without rollback or incident in the window / delivered changes

AI Engineering ROI
= value of stably delivered changes
  - tool cost
  - review and rework cost
  - incident and operating cost

No formula should automatically assume that every AI-influenced line would not have been written without AI.

---

---

9. Control architecture for AI code

Before generation

• task and acceptance criteria in writing
• permitted repositories, data, and tools
• secret and PII protection
• dependency/license rules
• risk and criticality class

During generation

• small, bounded deltas
• traceable agent/model version
• log tool and file accesses
• no direct production without a controlled path
• generate tests in parallel, but validate them independently

Before merge

• human owner
• automated tests
• SAST, dependency, and secret scanning
• architecture/domain review on critical paths
• check provenance and new dependencies
• limit or split PR size

After deployment

• feature/release telemetry
• error budget and incident linkage
• rollback capability
• measure churn and hotfixes
• feed findings back into prompts, policies, and tests

[RECOMMENDATION] AI-generated code does not necessarily need a different SDLC. It needs an SDLC that actually measures its own capacity and evidence.

---

---

10. What decision-makers should not claim from the evidence

Blanket statements such as these are not robust:

• “AI makes developers 55.8 percent faster.”
• “AI makes experienced developers 19 percent slower.”
• “AI causes 242.7 percent more incidents.”
• “AI code is more secure/less secure than human code.”
• “Code review is always the only bottleneck.”

Each of these figures is bound to a design, a population, a time, a tool, and an endpoint.

More robust is:

Controlled and observational studies show strongly heterogeneous effects. More code production can come with higher review, coordination, stability, and security requirements. Organizations must measure the net effect in their own delivery system.

---

3. Evidence matrix and attribution protection

The complete, machine-readable evidence matrix is located in data/FDR-2026-07_ai_delivery_evidence_matrix.csv. Every public figure must be cited with design, population, endpoint, and limitation.

4. New evidence 2026: the review bottleneck in an enterprise case

The study “AI Writes Faster Than Humans Can Review”, published in July 2026, extends the evidence base with an unusually large longitudinal corporate case. It analyzes 802 developers and 196,212 pull requests from January 2024 to April 2026.

4.1 What was observed

According to the authors:

• per-capita throughput reached 2.09 times the pre-mandate level in April 2026;
• the increase was distributed across seniority groups, but concentrated more strongly in newer code;
• the review load per reviewer grew roughly in step;
• automated review overtook human review activity;
• merge and revert rates remained stable in the aggregation examined.

4.2 What does not follow from this

Adoption and usage intensity were not randomized. The company was inclined toward AI, had an explicit 2x target, and represents a specific organizational context. The following statement is therefore not permissible:

“AI doubles the productivity of software teams.”

Permissible is:

“In a longitudinal, non-randomized enterprise case, measured per-capita PR throughput rose to 2.09x while the review load grew roughly in step.”

4.3 The bottleneck shifts

A delivery pipeline can be represented as a chained capacity system:

specification → code generation → local validation → review → CI → release → production feedback

Total capacity is limited by the slowest effective stage. If code generation is accelerated but specification, review, and test do not grow with it, no proportional outcome gain arises. Instead, the following grow:

• open pull requests;
• reviewer context switches;
• average diff size;
• waiting time until substantive feedback;
• automated approvals without sufficient calibration;
• rework after merge;
• production risk that coarse revert metrics may not capture.

---

5. Review Capacity Model

5.1 Basic equation

PR arrival rate λ =
new review-ready changes per working day

Review service rate μ =
reviewers × available review minutes / minutes per change

If λ ≥ μ, the backlog grows. With fluctuating arrivals and handling times, the queue can become unstable well before 100 percent utilization.

5.2 Example

Before AI adoption:

λ = 40 PR/day
5 reviewers × 90 review minutes / 10 minutes = μ 45 PR/day
utilization ρ = 0.89

After doubling generation:

λ = 80 PR/day
μ remains 45 PR/day
ρ = 1.78

Even if automated pre-screening reduces human handling time to seven minutes:

μ = 5 × 90 / 7 = 64 PR/day
ρ = 1.25

The queue keeps growing. The organization must either reduce arrivals, shrink batch size, raise quality before review, expand reviewer capacity, or route by risk.

5.3 Measurable review SLOs

Metric	Purpose
Time to First Substantive Review	detects a queue rather than merely a bot comment
Review Queue Age p50/p90	shows the age of critical changes
Review Minutes per 100 Meaningful Lines	roughly normalizes effort by diff
Reviewer Load Distribution	detects concentration on a few experts
Rework after Review	measures the quality of the feedback/author cycle
Escaped Defects by Risk Class	links review to production impact
Automated vs Human Review Coverage	makes the shift visible
Diff Size and Concern Count	controls batch risk

“Meaningful Lines” must exclude generated lockfiles, formatting, and vendored code, or report them separately.

---

6. Cost per accepted production change

6.1 Full formula

Accepted production change cost =
AI tool cost
+ inference/compute
+ author time
+ reviewer time
+ CI/preview
+ security review
+ rework
+ expected defect/incident cost

divided by:

changes that were accepted in production
and required no corrective reversal
within the defined window

6.2 Why PR count is unsuitable

A pull request can:

• contain only formatting;
• be generated automatically;
• never be merged;
• artificially split a larger PR;
• be corrected after a short time;
• create technical debt or security risk;
• deliver valuable or worthless functionality.

PR count is useful as a throughput signal, but is not a sole measure of productivity or value.

6.3 Expected defect cost

Expected defect cost =
Σ (defect probability × impact × detection/recovery cost)

A team should distinguish at least:

• pre-merge finding;
• post-merge/pre-release finding;
• production defect without customer impact;
• incident with availability impact;
• security finding;
• data or compliance deviation.

AI-assisted provenance may be documented in the root cause analysis, but must not be treated as the sole cause. Requirements, architecture, review, tests, and release controls remain part of the system.

---

7. Control architecture for AI-assisted development

The file data/FDR-2026-07_ai_delivery_governance_controls.csv contains 20 minimum controls. The following gate structure is recommended for tier 1 systems.

Gate A - Input and access

• approved tools, models, and data classes;
• no production secret or impermissible personal content in the prompt;
• least privilege for repository and context;
• retention and training terms documented;
• limit repository/file scopes.

Gate B - Change Design

• acceptance criteria before code generation;
• one functional concern per change;
• diff and file limits;
• AI assistance provenance in the PR;
• no autonomous change to auth, crypto, permissions, or payment without a specialist review.

Gate C - Deterministic CI

• unit, integration, and contract tests;
• type checking and linting;
• SAST and secret scanning;
• dependency review, lockfiles, and SBOM;
• test quality control, mutation testing where appropriate;
• reproducible builds and signed artifacts.

Gate D - Independent review

• the reviewer is not the author;
• risk-based reviewer assignment;
• queue age and load visible;
• automated review as support, not an invisible substitute;
• security triggers for sensitive areas;
• justification on override.

Gate E - Release and production

• ephemeral/preview environment;
• canary or progressive delivery;
• health gates and rollback;
• deployment markers in logs/metrics/traces;
• change owner and on-call assignment;
• record escaped defects and rework.

This architecture fits NIST SSDF because it integrates secure development practices into the entire lifecycle instead of treating AI code as a separate category without existing accountability.

---

8. Local experiment design: 90 days instead of an ideological battle

8.1 Hypothesis

Example:

“For clearly delimited backend changes of medium complexity, AI assistance reduces the median time to stable production by at least 15 percent without worsening change failure, security findings, or reviewer time.”

8.2 Randomization or staged rollout

Possible designs:

• randomized at task level, if tasks are comparable;
• staggered introduction by team;
• matched pairs by repository, complexity, and developer experience;
• A/B in time-boxed sprints;
• difference-in-differences with a clean pre-period.

Self-selection must remain visible. Developers who voluntarily use AI intensively can differ systematically from others.

8.3 Pre-Registration Light

Record before the start:

• primary endpoint;
• secondary endpoints;
• inclusion/exclusion rules;
• measurement window;
• risk classes;
• analysis plan;
• handling of dropouts and tool changes;
• minimum sample or duration.

This prevents only favorable metrics from being selected after the fact.

8.4 Recommended metric set

From data/FDR-2026-07_measurement_framework.csv:

• accepted change lead time;
• review queue age;
• review minutes per 100 changed lines;
• rework rate;
• change fail rate;
• failed deployment recovery time;
• escaped security findings;
• test mutation score delta;
• AI-assisted PR share;
• accepted outcome cost.

8.5 Segmentation

At minimum by:

• developer experience;
• repository familiarity;
• task type;
• risk;
• language/stack;
• tool/model version;
• greenfield vs legacy;
• PR size;
• test coverage.

An aggregated average can mask positive effects in simple tasks and negative effects in complex legacy areas.

---

9. Stop/scale rules

Scale

Expand AI use if, across several measurement windows:

• accepted lead time falls;
• the review queue does not grow;
• rework and change failure are stable or better;
• security findings do not increase;
• cost per accepted change falls;
• teams comply with the controls without circumvention.

Stop or limit

• the review queue exceeds its SLO;
• large AI diffs without a traceable specification;
• secret/data policy violations;
• the security finding rate rises;
• tests are generated, but the mutation score falls;
• rework merely shifts into later phases;
• tool cost grows faster than accepted outcomes;
• a model/tool change renders comparison data unusable.

Degrade Mode

Between full use and a ban:

• explanation, tests, or documentation only;
• no autonomous merges;
• no sensitive repositories;
• shorter context windows;
• only local or contractually suitable models;
• mandatory pairing for high-risk changes.

---

10. Risk classes instead of a uniform AI policy

A single rule for autocomplete, test generation, authentication logic, and production-adjacent infrastructure is too coarse. FW Delta recommends four risk classes.

Class A - assistive, non-executable content

Examples:

• explanation of existing code;
• drafting internal documentation;
• rewording comments;
• non-production examples;
• search and navigation assistance.

Minimum controls:

• approved tool;
• no prohibited data;
• human plausibility check;
• no unchecked adoption of external sources or licenses.

Class B - low-risk code with strong determinism

Examples:

• simple data conversion;
• boilerplate;
• tests for clearly defined functions;
• UI components without sensitive logic;
• internal scripts with limited scope.

Minimum controls:

• normal code review;
• deterministic tests;
• lint/type check;
• secret and dependency scan;
• small diffs.

Class C - business- or data-relevant logic

Examples:

• pricing, payroll, invoicing, CRM routing;
• personal data;
• external APIs with write effects;
• data migrations;
• important background jobs.

Additional controls:

• written acceptance criteria;
• contract/integration tests;
• independent domain approval;
• reconciliation;
• preview/canary;
• AI assistance provenance;
• rollback and data correction plan.

Class D - security, safety, or system boundary

Examples:

• authentication and authorization;
• cryptography;
• secret management;
• payments;
• infrastructure and IAM policies;
• security-critical parsers;
• regulated high-risk functions.

Additional controls:

• specialist review;
• threat model;
• negative and adversarial tests;
• SAST/DAST/fuzzing depending on context;
• dependency/SBOM policy;
• no autonomous approval;
• documented human accountability;
• staged rollout and close monitoring.

The classes assess the impact of the change, not the presumed quality of the model. The same model output can be harmless in class A and existential in class D.

---

11. Specification quality as an upstream bottleneck

AI can translate poor requirements into code faster. That is not a productivity gain. For measurable delivery, the specification must contain at least:

• the desired behavior;
• non-goals;
• inputs, outputs, and error states;
• data and permission boundaries;
• acceptance tests;
• performance/reliability requirements;
• migration and rollback behavior;
• observability;
• the accountable decision-maker.

11.1 Specification Readiness Score

Criterion	0	1	2
Outcome	unclear	feature described	measurable business/user outcome
Scope	open	rough boundary	explicit in/out-of-scope list
Interfaces	implicit	partial	versioned contracts
Error States	missing	common cases	complete risk classes
Tests	none	examples	automatable acceptance criteria
Security/Data	not mentioned	generic	concrete data and trust boundaries
Rollback	missing	manual	tested plan
Observability	missing	logs	metrics, traces, SLO/alerts

A task below a defined threshold should not be delegated to an agent but clarified first. This prevents high output speed from increasing the number of misunderstood changes.

11.2 A prompt is not a contract

A prompt can be part of the interaction, but the durable specification should be:

• versioned;
• reviewable;
• tool-independent;
• testable;
• linked to code and decision.

This keeps the development process traceable even when models change.

---

12. Automated review: calibration instead of blind trust

Automated review can reduce queue pressure, but it produces its own false positives and false negatives. A bot comment is not a substantive review.

12.1 Calibration data set

For each relevant finding class:

• 100 to 500 historical or synthetic examples;
• confirmed positive and negative cases;
• risk severity;
• language/framework;
• fixability;
• human gold rating.

12.2 Metrics

Precision = confirmed findings / all findings
Recall = relevant cases found / all relevant cases

In addition:

• mean time to triage;
• override rate;
• recurring false alarm classes;
• escaped findings;
• reviewer time the bot saves or additionally creates;
• effect after a model/rule update.

A tool with low precision can worsen the queue even though it produces more comments. A tool with high precision but very low recall must not be treated as a complete control.

12.3 Autonomy limits

Action	possible automation	precondition
Style/Lint Fix	auto-commit	deterministic and reversible
Dependency Update	PR creation	lockfile, tests, policy
Low-risk Finding	suggestion	human acceptance
Security Finding	block/route	calibrated rule and owner
Merge	very narrow scope only	branch protection, tests, low risk class
Production Deploy	progressive automation	health gates, rollback, accountability

---

13. Statistical analysis plan for the local experiment

13.1 Primary endpoint

accepted_change_lead_time as a median or log-transformed time, because delivery times are frequently skewed.

13.2 Secondary endpoints

• review queue age;
• reviewer minutes;
• rework within 14/30 days;
• change failure rate;
• recovery time;
• security findings;
• cost per accepted change.

13.3 Estimation

Depending on the design:

• difference in means/medians with a bootstrap interval;
• mixed-effects model with developer and repository as random effects;
• difference-in-differences with staggered introduction;
• survival analysis for lead time;
• negative binomial/Poisson model for count metrics;
• sensitivity analysis without extreme diffs or bot PRs.

13.4 Confounders

• task complexity;
• repository familiarity;
• developer seniority;
• model/IDE version;
• team changes;
• parallel initiatives;
• season/release phase;
• test infrastructure;
• voluntary adoption;
• changes to review rules.

13.5 Minimum interpretation standard

Every results slide contains:

1. effect size;
2. uncertainty interval;
3. sample;
4. time period;
5. design;
6. relevant confounders;
7. practical significance;
8. decision and next measurement.

A p-value without effect size and context is insufficient for a delivery decision.

---

14. Data protection, IP, and work organization

14.1 Data minimization

AI provenance should not degenerate into permanent performance monitoring of individual developers. Record what is necessary for security and impact measurement:

• tool/model class;
• material AI assistance yes/no;
• repository/risk class;
• change/outcome metrics;
• no unnecessary prompt content;
• limited retention and clear access rights.

14.2 Confidential information

Clarify before approving a tool:

• training/retention;
• region and sub-processors;
• admin/support access;
• IP and output terms;
• logging;
• deletion;
• enterprise security controls;
• permitted repository and data classes.

14.3 Accountability

The human author or owner remains responsible for the submitted change. At the same time, an incident is no reason to shift responsibility solely onto the individual person. Tool selection, specification, review capacity, CI, and release policy are management and system decisions.

14.4 Skill retention

A team must be able to operate critical systems even without the same model. Measures:

• regular manual debug/review exercises;
• document architecture knowledge;
• rotation of reviewer roles;
• no black-box automation without a runbook;
• postmortems and learning time;
• models as assistance, not the only source of knowledge.

---

15. 30/60/90-day implementation

Days 0–30 - Baseline and boundaries

• adopt a tool and data policy;
• define risk classes;
• secure a baseline of at least four weeks;
• introduce AI assistance provenance;
• make review queue and rework measurable;
• exclude tier D changes initially;
• select 2–3 clearly delimited task types.

Gate: measurement data, owner, hypothesis, and stop rules are in place.

Days 31–60 - Controlled rollout

• randomized or staggered introduction;
• enforce small diffs;
• tighten CI/security gates;
• test automated review against the calibration set;
• weekly outcome analysis;
• capture qualitative feedback in a structured way;
• version tool/model updates.

Gate: no negative trend in security, change failure, and review queue.

Days 61–90 - Scale or limit

• calculate effect sizes and intervals;
• identify segments with positive/negative effects;
• determine cost per accepted change;
• adjust controls according to findings;
• expand only suitable task types;
• register the next 90-day hypothesis;
• publish an executive decision record.

Gate: scaling follows outcome, not adoption rate.

---

16. Executive Scorecard

Dimension	Key question	Traffic-light example
Adoption	Is the tool used within the permitted scope?	information, not a target value
Throughput	Do accepted changes reach production faster?	trend plus interval
Review	Do queue and effort remain manageable?	p90 queue below SLO
Quality	Is rework falling or rising?	14/30-day cohort
Stability	Do change failure and recovery remain stable?	DORA-adjacent metrics
Security	Are findings detected earlier?	by severity and phase
Economics	Is cost per accepted change falling?	full cost, not tool price
Learning	Does the team know limits and failure modes?	drills, postmortems, updates

No single traffic light may obscure the rest. Higher throughput with growing security exposure is not a green overall result.

---

17. Visualization specification

Chart 1 - Evidence Landscape

• file: data/FDR-2026-07_ai_delivery_evidence_matrix.csv;
• X: year;
• Y: evidence grade;
• size: sample/corpus categorical only, because the units are heterogeneous;
• color/symbol: endpoint;
• no meta average.

Chart 2 - Effect and uncertainty

• only studies with a comparable time endpoint and intervals;
• METR 2025 and 2026 separately;
• define positive/negative direction clearly;
• do not mix vendor telemetry into the same causal forest plot.

Chart 3 - Review Queue

• simulated λ/μ scenarios;
• queue age over time;
• assumptions visible;
• shows a capacity bottleneck, not an empirical industry result.

Chart 4 - Control Pipeline

• five gates from input to production;
• controls from FDR-2026-07_ai_delivery_governance_controls.csv;
• evidence artifact per gate.

Chart 5 - Outcome Scorecard

• before/after or control/treatment;
• Lead Time, Review, Quality, Security, Stability, Cost;
• no single composite “AI productivity” figure without weighting.

---

18. Reproducibility

Files

• data/FDR-2026-07_ai_delivery_evidence_matrix.csv
• data/FDR-2026-07_ai_delivery_governance_controls.csv
• data/FDR-2026-07_measurement_framework.csv

Review rules

1. Never omit study design and endpoint.
2. Label vendor telemetry as such.
3. Do not phrase correlation as causality.
4. Do not aggregate different tasks and populations into a single average effect.
5. Name the tool/model point in time.
6. Present confidence intervals and author caveats.
7. Do not conflate LinearB and Faros figures.
8. Do not present arXiv preprints as conclusively peer-reviewed.
9. Version local experiments and archive the pre-period.
10. Design AI provenance appropriately under data protection and employment law.

---

19. Limits

1. The evidence base is developing rapidly.
2. Studies differ strongly in tasks and populations.
3. Vendor funding or proprietary data can limit replication.
4. PR and code metrics capture business value only indirectly.
5. Security benchmarks of older models are not directly transferable to 2026 systems.
6. A local experiment can contain Hawthorne, learning, and selection effects.
7. AI assistance is often hard to measure precisely when several tools are used in parallel.
8. Production quality may only become apparent after a longer period.
9. The report is not a recommendation for or against a particular product.
10. New studies and updated versions must be reviewed again before publication.

---

20. Permissible and impermissible statements

Permissible:

“The evidence available up to July 2026 shows strongly context-dependent effects; controlled tasks, real maintainer work, and enterprise telemetry produce different results.”

Impermissible:

“AI demonstrably always makes developers faster.”

Permissible:

“One enterprise case reached 2.09x PR throughput per developer and roughly double the review load.”

Impermissible:

“AI doubles software productivity.”

Permissible:

“METR found a 19 percent slowdown in its early-2025 setting.”

Impermissible:

“AI slows software development by 19 percent in general.”

---

21. Version history

• 1.0 · July 2026 · Ten existing evidence items plus a longitudinal 2026 enterprise case, review capacity model, 17 controls, and 90-day experiment design consolidated.
• Live version open · Update preprints, DORA, METR, and new controlled studies before publication.

22. Sources

1. Peng, S. et al., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, 2023, https://www.microsoft.com/en-us/research/publication/the-impact-of-ai-on-developer-productivity-evidence-from-github-copilot/.
2. Becker, J. et al. / METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 10 July 2025, https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/.
3. Becker, J. et al. / METR, “We are Changing our Developer Productivity Experiment Design”, 24 February 2026, https://metr.org/blog/2026-02-24-uplift-update/.
4. DORA, “The Impact of Generative AI in Software Development”, https://dora.dev/ai/gen-ai-report/, accessed in July 2026.
5. Song, F.; Agarwal, A.; Wen, W., “The Impact of Generative AI on Collaborative Open-Source Software Development”, arXiv:2410.02091v3, 14 May 2026, https://arxiv.org/abs/2410.02091.
6. Faros AI, “The AI Engineering Report 2026: The Acceleration Whiplash”, https://www.faros.ai/research/ai-acceleration-whiplash, accessed in July 2026.
7. GitHub, “Does GitHub Copilot improve code quality? Here’s what the data says”, https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says/, accessed in July 2026.
8. Pearce, H. et al., “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions”, IEEE S&P 2022 / arXiv:2108.09293, https://arxiv.org/abs/2108.09293.
9. Fu, Y. et al., “Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study”, arXiv:2310.02059, https://arxiv.org/abs/2310.02059.
10. LinearB, “2026 Software Engineering Benchmarks Report”, https://linearb.io/resources/software-engineering-benchmarks-report, accessed in July 2026.
11. LinearB, “8 million pull requests reveal where engineering productivity breaks down”, https://linearb.io/blog/8-million-prs-engineering-productivity, accessed in July 2026.
12. Google Cloud, “2024 Accelerate State of DevOps Report”, https://dora.dev/research/2024/dora-report/, accessed in July 2026.
13. NIST, “Secure Software Development Framework (SSDF) Version 1.1”, https://csrc.nist.gov/pubs/sp/800/218/final, accessed in July 2026.
14. OWASP, “Top 10 for Large Language Model Applications”, https://genai.owasp.org/llm-top-10/, accessed in July 2026.
15. He, Hao; Agarwal, Shyam; Denisov-Blanch, Yegor; Azaletskiy, Pavel; Koyejo, Sanmi; Vasilescu, Bogdan (2026), “AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate”, arXiv:2607.01904, https://arxiv.org/abs/2607.01904, accessed in July 2026.
16. NIST, “Secure Software Development Practices for Generative AI and Dual-Use Foundation Models”, SP 800-218A, https://csrc.nist.gov/pubs/sp/800/218/a/final, accessed in July 2026.
17. DORA, “State of AI-assisted Software Development 2025”, https://dora.dev/dora-report-2025/, accessed in July 2026.

---

Disclosure and disclaimer

FW Delta develops AI-supported automation and software. The company can benefit commercially from implementation and governance projects. The report therefore contains positive, negative, and uncertain evidence and separates controlled studies from vendor telemetry.

The report does not replace security, data protection, employment law, or product approval. Local data, risk class, and regulatory context must be assessed individually.