One-sentence Summary

Researchers at the Indian Institute of Science evaluated 50 LLM-generated research documents from autonomous research agents and found that 24% were paraphrased or significantly borrowed from existing work and 32% partially overlapped with prior research, while controlled experiments showed that automated plagiarism detectors are inadequate at catching such plagiarized ideas, underscoring the need for careful assessment of LLM-generated research.

Key Contributions

• The paper presents the first systematic expert-led audit of plagiarism in LLM-generated research documents, asking 13 experts to compare 50 generated documents with existing literature rather than judge novelty in isolation.

• The audit finds that 24% of the evaluated documents are paraphrased with one-to-one methodological mappings or significantly borrowed from existing work, and an additional 32% partially overlap with prior research, with source attribution missing and reported cases cross-verified by authors of the source papers.

• Controlled experiments show that current automated plagiarism detectors, including conventional text-matching tools and retrieval-augmented LLM systems, fail to catch sophisticated idea plagiarism in LLM-generated research documents.

Introduction

The authors address a practical and consequential problem in automating scientific discovery: LLMs are increasingly used to generate research hypotheses, proposals, and papers, making it critical to assess whether their outputs are genuinely novel or simply recycled prior work. Prior studies mainly evaluate LLM-generated ideas by asking experts to rate novelty or by using LLM-based judges, often on relatively small samples. This is challenging because scientific novelty is subjective, and existing automated plagiarism detectors can miss highly paraphrased or structurally similar content, especially when LLM outputs do not explicitly cite sources.

The authors contribute a more adversarial expert-led evaluation in which researchers are asked to actively look for plagiarism in LLM-generated research documents. Across 50 documents, they find that 24.0% show clear signs of plagiarism when authors of referenced papers are reachable, rising to 36.0% when claims involving unreachable authors are included. They also show that current tools such as embedding-based retrieval and commercial detectors are inadequate, suggesting that LLM-generated research content may appear novel on the surface while borrowing heavily from existing work.

Dataset

• Dataset purpose and composition

  • The authors construct a synthetic evaluation dataset for plagiarism detection, consisting of generated plagiarized research proposals paired with their original source papers.
  • The dataset is used as a test set, not as a training corpus or a mixture for model training.

• Source papers and subset size

  • The original source papers are retrieved during the literature review step of the baseline method.
  • The authors select 12 research topics and use 40 papers per topic, producing a test set of 480 source papers.
  • Each source paper is paired with one generated plagiarized proposal, yielding 480 proposal source paper examples.

• Generation process

  • GPT-4o is used to generate the plagiarized proposals by paraphrasing details from the source paper to avoid detection.
  • The authors choose GPT-4o for this task because Claude 3.5 Sonnet refuses requests involving plagiarism.
  • The prompt used for generation is provided separately in the paper.

• Schema and metadata

  • Each dataset example can be understood as containing the research topic, the original source paper, and the generated plagiarized proposal.
  • The paper does not describe cropping, multimodal extraction, or additional filtering beyond topic selection and prompt-based generation.

• How the data is used

  • The full 480 example set is used to evaluate plagiarism detectors across several setups.
  • In the oracle access setup, the model receives both the proposal and its source paper and decides whether plagiarism occurred.
  • In the parametric knowledge setup, the model must retrieve and judge similarity using only its internal knowledge, without external retrieval tools.
  • In the Semantic Scholar augmented retrieval setup, the system first retrieves candidate papers through Semantic Scholar and then determines similarity between the proposal and retrieved papers.

• Reduced evaluation subset for manual tools

  • For OpenScholar and Turnitin, manual submission and review limits evaluation to 3 generated proposals per research topic.
  • This produces a reduced subset of 36 proposal cases across the 12 topics.

• Interpretation of the dataset

  • The dataset is easier than real world plagiarism detection because each proposal is generated to plagiarize from a single known source paper.
  • The authors note that detector performance on this synthetic set may overestimate performance on more subtle plagiarism from systems designed to generate novel research.

Method

The proposal generation pipeline is organized as a multi-stage workflow with Claude 3.5 Sonnet serving as the central language model. For each research topic, the process begins with a retrieval-augmented generation module that queries the Semantic Scholar API, retrieves relevant papers, and ranks them to provide grounding evidence. These retrieved papers are then used to generate initial seed ideas. The seed ideas are deduplicated through text embeddings, preserving only a small fraction, roughly 5 percent, of the original ideas to remove semantic repetition while retaining diversity. The remaining ideas are expanded into full project proposals, each containing a title, problem statement, motivation, proposed method, experiment plan, test cases, and fallback plan. The proposals then enter a Swiss tournament selection module, where they are evaluated through five rounds of pairwise comparisons to identify the strongest candidates.

After proposal generation, the pipeline applies a plagiarism detection module based on Semantic Scholar Augmented Generation. The module iteratively generates search queries for the Semantic Scholar API, using previously retrieved papers as context to refine subsequent queries. The iterative retrieval stops after either 100 papers have been collected or 10 search iterations have been completed. The retrieved papers are then scored by the language model for relevance to the proposal, and the ten most similar papers are retained. Claude 3.5 Sonnet performs pairwise comparisons between the proposal and each of these top papers, deciding whether the proposal is substantially similar to a retrieved paper. If substantial similarity is detected, the proposal is discarded. This filtering stage removes only a small proportion of proposals, around 1 percent, indicating that the detection module acts as a conservative final safeguard.

To optimize computational cost, the pipeline adjusts the initial idea-generation volume. Rather than producing a very large number of ideas per topic, it generates 500 ideas per topic. This yields approximately 138 unique ideas per topic after deduplication, compared with around 200 in a larger-scale setting. The reduction in unique ideas is much smaller than the reduction in raw generated ideas, reflecting diminishing returns: as the number of generated ideas increases, overlap grows and the share of newly unique ideas decreases. The ranking, plagiarism detection, and remaining generation steps follow the same overall design as the original pipeline, while the reduced sampling effort provides a cost-efficient version that still preserves a substantial fraction of unique proposals. The methodology is selected because it reflects common prompt-engineering patterns in research-idea generation, supports a broad evaluation setting, and uses comparatively simple prompts, allowing the raw capabilities of the language model to be assessed with less confounding from elaborate prompt construction.

Experiment

The experiments combine an expert evaluation of LLM-generated research proposals, author verification, and a synthetic test set for automated plagiarism detectors. The expert evaluation validates that domain experts can identify serious methodological overlap and uncredited borrowing, while the detector study validates retrieval and similarity judgment across LLM-based and commercial tools. Overall, LLM-generated proposals show substantial content misappropriation and limited diversity, and current automated plagiarism detection methods are inadequate for reliable identification.

The rubric provides five levels for judging how closely an LLM-generated research proposal matches prior work. Higher scores indicate stronger methodological overlap, with the top level reserved for direct copying and the lowest level reserved for completely original proposals. The middle levels distinguish partial or slight similarity without exact correspondence to a small set of prior papers. Top scores identify proposals with one-to-one mapping to existing methods from closely related prior papers. A score just below the highest indicates significant uncredited borrowing from multiple prior works. Mid-range scores describe decent similarity without exact correspondence to limited papers. Lower scores indicate minor resemblance and mostly novel proposals.

Expert evaluation indicates that LLM-generated research proposals can contain plagiarized content, with confirmed cases occurring at high similarity levels. Because verification was limited to proposals with higher initial scores, the reported plagiarism rate may understate the true extent, and including unverified high-score claims increases the suspected share. Verified plagiarism occurs in proposals with the highest similarity scores. Including unverified high-score claims raises the estimated plagiarism rate. Existing automated tools did not consistently identify source papers for verified highly similar documents.

Automated plagiarism detectors perform poorly in realistic settings even when proposals are deliberately plagiarized. Providing both the source paper and suspected proposal gives the strongest detection accuracy, but this oracle setting is rarely available because the source is unknown. Retrieval-based methods and parametric knowledge are much weaker, with OpenScholar failing to detect plagiarism. Oracle access yields the best results, showing that similarity assessment is easier when the source is known. SSAG and parametric performance remain limited, indicating that retrieving the correct source paper is a major bottleneck.

The table suggests that signs of plagiarism identified from peer reviews of human-written papers are much less common than the overall rate reported for LLM-generated proposals. The estimated rates vary by venue, with one venue showing the strongest signal and several venues showing only small review-driven concerns. The pattern supports a qualitative comparison rather than a definitive measurement of actual plagiarism. Automated peer-review analysis detects only limited signs of plagiarism in human-written papers. Human-paper plagiarism signals are substantially lower than those found in LLM-generated proposals. Detected concerns differ by conference, and higher-score signals are rare.

The experiments use a five-level similarity rubric and expert verification to assess whether LLM-generated research proposals reproduce prior work, with confirmed plagiarism concentrated in the highest-similarity proposals. Automated detection experiments show that tools are strongest when given the source paper, while retrieval and parametric methods are weak, indicating that identifying the correct prior work is a major bottleneck. A related review-based comparison finds substantially fewer plagiarism signals in human-written papers than in LLM-generated proposals, though this analysis may underestimate actual cases.