One-sentence Summary

Researchers from the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences propose $\text{RACE}$RACE, a fine-grained detection method that uses Rhetorical Structure Theory to model creator and editor roles through discourse-level logic graphs and EDU-level features, outperforming $12$12 baselines in four-class LLM-generated text detection with low false alarms for policy-aligned regulation.

Key Contributions

• The paper introduces a four-class fine-grained detection setting that distinguishes human-written text, LLM-generated text, LLM-polished human text, and humanized LLM text, moving beyond binary and ternary classification setups that cannot meet nuanced policy regulation needs.
• The paper proposes RACE, a detector that separately models the roles of creator and editor by building a Rhetorical Structure Theory (RST) logic graph to capture the creator's logical organization and extracting Elementary Discourse Unit (EDU)-level features to characterize the editor's style, then performing rhetoric-guided message passing on the resulting logic-aware graph.
• Results on the HART benchmark show RACE outperforms 12 baselines across the fine-grained four-class identification task while maintaining a low false alarm rate, providing a policy-aligned solution for LLM-generated text regulation.

Introduction

The explosive growth of Large Language Models (LLMs) has transformed content creation, but it has also eroded trust by enabling large-scale misinformation, unauthorized academic assistance, and identity fraud. Detecting LLM-generated text is therefore critical for preserving information credibility. However, early detectors framed this as a binary classification task, either human or LLM, which fails to capture real-world collaborative scenarios. People routinely ask LLMs to polish their drafts or, conversely, humanize LLM outputs to evade detection, producing hybrid texts that blur conventional boundaries. Prior work moved to fine-grained settings, often adding a third mixed category, but even this ternary split is insufficient for nuanced regulatory policies, particularly in academic writing where text polishing is viewed as legitimate assistance while humanizing to bypass detectors is prohibited.

In this context, the authors study a practically important four-class detection setting that separates the mixed category into LLM-Polished Human Text and Humanized LLM Text. They analyze these classes through the dual lenses of creator and editor: the creator establishes the underlying logical structure and argumentative flow, while the editor shapes the linguistic expression and surface style. For pure human or pure LLM text, these roles are indistinguishable, but the two mixed classes exhibit divergent creator-editor collaboration modes, making it essential to model the roles separately. To tackle this, the authors propose RACE (Rhetorical Analysis for Creator-Editor Modeling), which segments text into Elementary Discourse Units (EDUs) to capture the editor's surface-level refinements, and leverages Rhetorical Structure Theory (RST) to build an EDU-based logical relation graph that models the creator's foundational organization. A rhetoric-guided message passing mechanism then propagates information across this graph to capture complex rhetorical dependencies. The authors contribute a task-specific framework for the four-class setting, a creator-editor modeling method grounded in RST, and extensive experiments showing that RACE outperforms baselines with low false alarms.

Dataset

Dataset Overview

• Source and composition: The authors use the HART benchmark as the evaluation dataset, chosen for its coverage of the desired fine-grained categories (News, Writing, ArXiv, and Essay). The official release only includes validation and test partitions, so the authors reorganize the data distribution to enable supervised learning.

• Train/validation/test split: The reorganized data is split at a 70:20:10 ratio using stratified sampling across diverse domains. This ensures distribution consistency across all partitions.

• Group-aware split: In addition to the standard split, the authors report a stricter group-aware split in Appendix B.2, where all variants derived from the same base text are assigned to the same partition. This prevents data leakage and tests generalization to unseen base texts.

• Task setup: The data supports a 4-class fine-grained detection task. Because no task-specific method exists for this setting, the authors adapt 12 learning-based or metric-based methods originally designed for two- or three-class settings.

• Learning-based baselines: Includes RoBERTa, RoBERTa-DANN, CoCo, DeTeCtive, and LF-Motifs. The classification head is extended from 2 to 4 output entries. CoCo and LF-Motifs incorporate discourse information, while DeTeCtive uses contrastive learning similar to the authors' approach.

• Metric-based baselines: Includes Binoculars, Fast-DetectGPT, and TDT. These methods produce scalar predictions, so the authors extract last-layer representations and append a learnable MLP to enable four-class prediction.

• Evaluation metrics: The authors prioritize metrics that assess global ranking capability rather than hard predictions. They use Macro-Averaged AUROC, which measures the probability that a randomly selected positive instance from any class is ranked higher than a randomly selected negative instance, and TPR@1% FPR, which requires precise judgments while avoiding false alarms.

• Implementation details for RACE: The model uses RoBERTa-base as the backbone, fine-tuning only the last layer while keeping preceding layers frozen. Extracted features are projected to a dimension of 128 to initialize node features. The graph component is an RGCN with 2 layers, a hidden dimension of 512, and 10 bases for parameter regularization. Temperature for supervised contrastive loss is set to 0.07. The best validation checkpoint is selected for testing, and all experiments run on a single NVIDIA RTX 4090 GPU.

Method

The authors propose a logical-structure-aware detection framework designed to capture the dual traces of creators and editors. Preliminary analysis indicates that rhetorical structures differ significantly between sources; as shown in the distribution chart, human creators exhibit over-expression in relations like Attribution and Temporal, while LLMs rely more heavily on Elaboration and Cause. These structural signatures persist even after editing, motivating the use of rhetorical graphs for detection.

The overall architecture, illustrated in the framework diagram, consists of four key components: Dual Trace Extraction, Logic-Aware Graph Initialization, Rhetoric-Guided Message Passing, and Graph Readout and Classification.

First, the Dual Trace Extraction module transforms unstructured text into a hierarchical representation using an end-to-end RST parser. This process generates a binary constituency tree where leaf nodes correspond to Elementary Discourse Units (EDUs) and internal nodes represent logical relations (e.g., Contrast, Cause). This tree serves as the skeleton for the subsequent graph construction.

Next, the Logic-Aware Graph Initialization module formalizes the tree as a multi-relational graph $\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathcal{R})$G=(V,E,R). Node features are initialized using a hybrid strategy. Initially, "Descendant Span Pooling" computes content representations: EDU nodes are initialized by mean-pooling token embeddings, while relation nodes are initialized by averaging the embeddings of their descendant nodes. Subsequently, an "Information Bottleneck Projection" filters surface-level noise by projecting these embeddings into a compact structural space via a linear layer, augmented with learnable type embeddings.

Then, the Rhetoric-Guided Message Passing module employs an L-layer Relational Graph Convolutional Network (RGCN). Unlike vanilla GCNs, this module assigns relation-specific transformation matrices, allowing the model to learn distinct propagation rules for different rhetorical logics. The node representation is updated by aggregating neighbor information based on the specific relation type. To prevent parameter explosion and overfitting, the authors apply Basis Decomposition, which constrains the weight space by expressing relation-specific matrices as linear combinations of shared basis matrices and learnable scalar coefficients.

Finally, the Graph Readout and Classification module extracts the global text representation by pooling the hidden state of the root node. This vector is passed through a classification head comprising dropout, linear layers, and a softmax function. The model is optimized using a joint loss function that combines supervised contrastive loss and cross-entropy loss, encouraging the learning of a compact and discriminative representation space.

Experiment

The experiments were conducted on a reorganized HART benchmark with a 70:20:10 train/val/test split (including a stricter group-aware split), evaluating performance via macro AUROC and TPR@1%FPR against adapted baselines. The main results show that RACE outperforms all baselines on TPR@1%FPR, particularly beating the best discourse-aware method CoCo by 3.36% absolute, and that learning-based methods generally surpass metric-based ones. Ablations confirm the necessity of relational graph modeling, contrastive learning, basis decomposition, and information bottleneck projection, with the largest drops on LLM-generated and polished classes, while further analysis demonstrates RACE's superior feature discriminability, better handling of short texts, and stronger out-of-distribution robustness.

Sample-level cosine similarities between rhetorical-relation frequency vectors show that texts retain high similarity to their source regardless of subsequent editing, with mean values around 0.84 to 0.95 and small standard deviations. This indicates that the underlying rhetorical structure of the original creator is preserved through polishing or humanization, supporting the notion of distinct creator and editor roles. Texts produced by the same creator remain rhetorically close after editing, with mean similarities above 0.84 for all pairings. The highest similarity occurs between LLM-generated and humanized texts, at 0.95, while human-written and LLM-generated pairs are least similar, at 0.84. Small standard deviations across all pairs indicate consistent preservation of rhetorical signatures through editing operations.

RACE achieves the highest average TPR@1%FPR among compared methods under the 4-class setting, outperforming the best baseline CoCo by 3.36% absolute. Learning-based methods generally surpass metric-based ones, with some metric-based detectors showing near-zero TPR at low false alarm rates. RACE also demonstrates strong performance across all four text categories, particularly on the Humanized class. RACE leads the average TPR@1%FPR with a margin of 3.36% over the best baseline, CoCo. Learning-based detectors outperform metric-based ones, as metric-based methods often collapse feature space needed for multi-class tasks. RACE achieves the best average AUROC (98.20) and highest TPR@1%FPR on Humanized texts, indicating robust fine-grained detection. CoCo and LF-Motifs, both discourse-aware, fall behind RACE, which uses relational graph message passing to capture structural anomalies.

The ablation study reveals that every component of RACE contributes to fine-grained detection performance, with the most significant degradation occurring for LLM-generated and LLM-polished classes when components are removed. The relation modeling and contrastive learning components are crucial for distinguishing creator-editor patterns, while the bottleneck and basis decomposition provide targeted benefits for specific classes and stability. Removing any component reduces average detection performance, with the largest drops for LLM-generated and LLM-polished classes. The w/o Relation variant shows severe performance degradation and high variance, indicating that logical relations are essential for capturing creator-editor roles. Removing the Information Bottleneck specifically hurts LLM-Polished detection, suggesting it helps avoid overfitting to superficial editor patterns. Removing Basis Decomposition increases performance variance, highlighting its role as a regularizer through weight sharing.

The parser-robustness analysis shows that RACE maintains consistent performance when the underlying RST parser is replaced, with both DMRST and IsaNLP yielding nearly identical AUROC and average TPR values. This stability indicates that the model's gains stem from its creator-editor modeling rather than from parser-specific artifacts. RACE with DMRST achieves an AUROC of 97.64 and an average TPR@1%FPR of 82.93, while RACE with IsaNLP reaches 97.99 and 83.06, respectively. The minimal performance difference across parsers confirms that RACE's robustness is independent of the chosen RST parser, reinforcing the importance of its core modeling approach.

RACE achieves better feature discriminability than CoCo, as shown by a lower Davies-Bouldin Index and a substantially higher Calinski-Harabasz Index. The improved embedding space likely stems from explicitly modeling creator and editor logic, leading to more compact and distinct clusters. RACE also demonstrates stronger performance on shorter texts, where its graph-based approach captures nuanced differences more effectively. RACE lowers the Davies-Bouldin Index from 0.9286 to 0.8042, indicating better intra-cluster compactness relative to inter-cluster separation. The Calinski-Harabasz Index for RACE is nearly double that of CoCo, reflecting a significantly more distinct embedding space. For texts between 200 and 600 tokens, RACE outperforms CoCo, suggesting its graph-based modeling is more efficient with limited information.

Experiments reveal that rhetorical structures remain highly similar to their source across editing, supporting the notion of distinct creator and editor roles, with RACE outperforming baselines in fine-grained detection through relational graph message passing. Ablation and parser robustness tests show that each component contributes, especially relation modeling and contrastive learning, while performance stays stable across different RST parsers. RACE also yields more discriminative embeddings than CoCo and is notably stronger on shorter texts, indicating effective capture of structural anomalies.