One-sentence Summary

By studying 1.02 million pull requests from 207 GitHub projects transitioning from human-centric to LLM-assisted and agentic code review, the authors identify three AI adoption practices and model pull request discussions as reviewer interaction sequences, revealing that agent-involved collaboration patterns accelerate review decisions under Gradual AI Adoption and Rapid AI Agent Adoption, but these efficiency gains do not improve review quality, and human-AI collaboration becomes the strongest explanatory factor for efficiency.

Key Contributions

• A modeling approach represents pull request review discussions as sequences of reviewer interactions, enabling characterization of collaboration patterns among human, LLM, and AI agent reviewers.
• Based on 1.02 million pull requests from 207 GitHub projects, three AI reviewer adoption practices are identified: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption. Agent-involved patterns, including agent-initiated and multi-agent reviews, correlate with faster review decisions under Gradual AI and Rapid AI Agent Adoption but do not improve review quality, while Rapid LLM Adoption is associated with higher review-smell prevalence without efficiency gains.
• Once AI reviewers participate, human-AI collaboration patterns become the strongest explanatory factor for review efficiency, surpassing traditional factors such as pull request type and review activity.

Introduction

Generative AI now drives a massive increase in code output, making code review and integration downstream bottlenecks. As teams move from human-only review to LLM-assisted and agentic review, understanding how this shift affects review practices and quality remains an open challenge. Prior studies have focused on human-centric processes or isolated LLM tools, offering little evidence on how review quality evolves when AI reviewers participate across multiple eras. The authors address this gap by analyzing 1.02 million pull requests from 207 open-source projects as they transition from Pre-LLM to LLM and Agent eras. They characterize three AI reviewer adoption practices, examine how human, LLM, and AI agent reviewers collaborate, and model the combined influence of collaboration patterns and traditional factors on review efficiency and code review smells.

Dataset

The dataset underpinning the study consists of 1.02 million reviewed pull requests from 207 active GitHub projects, collected to trace code review across three eras: pre-LLM, LLM-assisted, and AI-agent review. The authors build the dataset through a multi-stage selection, labeling, and classification pipeline.

Composition and sources

• 2,490 candidate projects were identified via GitHub advanced search, filtered by continuous review activity from May 2022 to February 2026.
• Final 207 projects retain those with over 400 reviewed pull requests in each of the three review eras (pre-LLM, LLM, agent), ensuring sufficient data for within-project comparisons.
• Total pull requests: 1.02 million, each with review conversations, reviewer identities, timestamps, merge/reject decisions, and review comments.

Key details for each subset and filtering

• Project eligibility criteria
• At least 100 stars (community adoption and active maintenance).
• Created before May 2022 (at least six months of pre-ChatGPT history).
• At least one reviewed pull request per month from May 2022 to February 2026 (continuous review activity).
• Review era definition
• For each project, the pre-LLM era ends when the first LLM reviewer participates; the LLM era ends when the first AI agent reviewer participates; the agent era starts thereafter.
• Only projects with >400 reviewed pull requests per era are kept, yielding 207 projects.

Reviewer labeling and metadata construction

• The authors use the GitHub REST API to distinguish human and bot accounts.
• Bot accounts are manually classified by inspecting official documentation into:
• Rule-based bots (e.g., GitHub Actions)
• Traditional machine learning reviewers (e.g., Amazon CodeGuru Reviewer)
• LLM reviewers (e.g., LlamaPReview)
• AI agent reviewers (e.g., Claude Code)
• A manual validation of 384 agent-era pull requests (95% confidence, 5% margin) confirmed agentic behavior (planning, context retrieval, commits) in 360 cases; the remaining 24 still involve accounts documented as AI agents.

Pull request type classification

• Each pull request is assigned one of 11 types (e.g., bug fixing, feature addition) using GPT‑4.1‑mini on titles and descriptions, following the taxonomy of Watanabe et al.
• A manual validation on 384 samples gave a Cohen’s κ of 0.91, indicating near-perfect agreement.

How the paper uses the data

• The dataset is not split into training/validation/test sets; it is used directly for three research questions (RQs) that examine:
• Common AI reviewer adoption patterns by clustering per-project adoption series.
• Changes in review quality and pull request type distribution across eras.
• Human‑AI collaboration patterns, their efficiency, and relationship with review quality.
• Explanatory models are built to assess the link between review quality, collaboration patterns, and traditional pull request characteristics.

Processing and cropping strategy

• No cropping or subsampling is applied to the pull requests beyond the project-level era filters; the full 1.02 million pull requests are used.
• The only content processing is the GPT‑4.1‑mini classification of pull request types, which enriches the dataset with a categorical type label for each pull request.

Method

The authors leverage a multi-stage analytical framework to investigate the adoption and impact of AI reviewers in open-source software projects. The overall pipeline begins with project selection and data collection. Starting from an initial pool of 2,490 candidate GitHub projects with continuous review activity, the authors filter the dataset down to 207 projects that contain sufficient reviewed pull requests spanning the pre-LLM, LLM, and agent eras. During the data collection phase, they gather review conversations, classify the reviewers involved, and identify the specific review eras for each project.

Following the initial data filtering, the authors cluster the generative AI adoption series for each project into common AI reviewer adoption practices. To facilitate a comprehensive analysis across their research questions, they define a set of shared metrics and explanatory factors. Pull request types are assigned using a taxonomy of 11 code-change purposes, classified by analyzing titles and descriptions with GPT-4.1-mini. Review quality is evaluated through two primary measures: the presence of review smells and review efficiency. Review efficiency is calculated as the normalized duration from pull request creation to the final decision: $\text{Review Efficiency} = \frac{T_{\text{decision}} - T_{\text{creation}}}{\# \text{Thousand lines of code}}$Review Efficiency=#Thousand lines of codeTdecision​−Tcreation​​ where $T_{\text{creation}}$Tcreation​ is the timestamp of the creation of the pull request and $T_{\text{decision}}$Tdecision​ is the timestamp of the final decision. Additionally, the authors establish explanatory factors encompassing pull request characteristics, review activity, and participant experience, which also account for the emerging presence of LLM and AI agent reviewers.

With these metrics defined, the framework branches into three distinct analytical approaches. First, to understand AI adoption practices, the authors examine the pull request types that increasingly involve AI reviewers and investigate the corresponding changes in review quality across the three eras. Second, to explore human-AI review collaboration patterns, they identify common interaction patterns among human, LLM, and AI agent reviewers, comparing these patterns based on review efficiency and quality. Finally, to determine the factors influencing review efficiency and quality, the authors build explanatory models that assess the relationship between review outcomes, the identified human-AI collaboration patterns, and traditional review-process factors.

Experiment

The evaluation setup measures review efficiency and six review smells across pre-LLM, LLM, and agent eras using longitudinal data from 2022-2026. Three experiments examine AI adoption practices, human-AI collaboration patterns, and their association with quality alongside traditional review factors. The findings indicate that gradual AI adoption and rapid agent adoption improve review efficiency, while rapid LLM adoption increases review smells; agent-initiated patterns can be faster but AI involvement consistently raises the Review Buddies smell, and traditional factors like author experience still matter.

Traditional review factors such as commit count, inline discussion threads, and pull request type remain important predictors of review smells after generative AI adoption. Agent-era collaboration patterns are associated with fewer Sleeping Reviews under gradual and rapid AI agent adoption practices, but they also correlate with more Review Buddies and larger changesets. In the agent era, agent-involved reviews consistently involve larger code changes than in earlier eras. Under Rapid LLM Adoption, agent patterns like Agent-Init and Agent-ML are associated with a higher probability of Sleeping Review. Agent-involved collaboration patterns in the agent era are positively associated with Large Changeset across gradual and rapid AI adoption practices. Agent-era collaboration patterns reduce the likelihood of Sleeping Review under Gradual AI Adoption and Rapid AI Agent Adoption.

Across all adoption paths, reviews in the LLM and agent eras show a higher proportion of review buddies and an increase in review smells relative to pre-LLM baselines. Efficiency changes are small except for a significant drop under gradual AI agent adoption, and the types of pull requests that grow shift notably, with feature and refactoring work rising in the gradual agent era and documentation, build, and style changes increasing under rapid LLM adoption. Review buddy rates increased in every AI-assisted era, with the largest jump occurring during rapid LLM adoption. Under gradual AI adoption, the agent era is associated with a significant efficiency decline and a rise in feature and refactoring pull requests.

Human-only reviews consistently achieve top efficiency ranks and the lowest review smell prevalence across all eras and adoption practices. In contrast, human-AI collaboration patterns such as Human-Bot, Human-Bot-ML, and LLM-Assist show higher smell rates, and their efficiency ranks vary, with LLM-Assist ranking lowest under Rapid LLM Adoption. These patterns are also associated with a higher frequency of non-feature pull requests, particularly chores and fixes. Human-only reviews are the most efficient pattern in the LLM era (R1 across all adoption contexts) and never rank below R2 in the pre-LLM era, while maintaining the lowest smell percentages (69–76%). All human-AI collaboration patterns exhibit elevated review smell prevalence compared to human-only reviews, with LLM-Assist reaching 87% under Rapid LLM Adoption and Human-Bot-ML reaching 88% in Gradual AI Adoption.

In the agent era, collaboration patterns involving AI agents are linked to both improvements and drawbacks: while certain patterns reduce sleeping reviews under gradual and rapid agent adoption, they are consistently associated with more review buddies and larger changesets. Traditional review factors remain significant even after generative AI adoption, and in the LLM era, agent patterns sharply increase the likelihood of sleeping reviews. Agent-era collaboration patterns reduce Sleeping Review under Gradual AI Adoption and Rapid AI Agent Adoption, but they are also associated with more Review Buddies and larger changesets. In the LLM era, Agent-Init and Agent-ML are associated with a much higher probability of Sleeping Review, with impact scores of +42 and +94.

The study examines code review outcomes across pre-LLM, LLM, and agent eras under different AI adoption practices, focusing on review smells, efficiency, and collaboration patterns. Human-only reviews consistently outperform human-AI collaborations in efficiency and smell prevalence, while agent-era patterns reduce Sleeping Reviews but increase Review Buddies and Large Changesets. Traditional review factors remain important, and the impact of AI assistance depends on adoption strategy and specific collaboration types.