One-sentence Summary

This study proposes a model-agnostic framework to measure and mitigate privacy risks from Chain-of-Thought leakage by analyzing risk-weighted token-level events across 11 PII types and varying reasoning budgets, ultimately benchmarking inference-time gatekeepers including a rule-based detector, a TF-IDF and logistic regression classifier, a GLiNER-based NER model, and an LLM-as-judge to advocate for hybrid, style-adaptive policies.

Key Contributions

• The paper introduces a model-agnostic framework that defines PII leakage as risk-weighted, token-level events across eleven distinct PII types to analyze how reasoning traces resurface sensitive information.
• This work establishes a reproducible protocol to trace leakage curves as a function of the allowed Chain-of-Thought budget, revealing that reasoning processes systematically increase leakage in a manner that is highly dependent on the specific model family and budget size.
• The study benchmarks several lightweight inference-time gatekeepers, including rule-based detectors, lexical classifiers, GLiNER-based NER models, and LLM-as-judge variants, demonstrating that no single method provides universal protection across all models and reasoning budgets.

Introduction

While Chain-of-Thought (CoT) prompting significantly enhances the reasoning capabilities of large language models, it introduces a critical privacy vulnerability by potentially resurfacing personally identifiable information (PII) from the prompt into intermediate reasoning traces and final outputs. Prior research has largely focused on PII extraction from training data or general contextual privacy, often overlooking the specific risk of direct, inference-time leakage where models violate "do not restate PII" policies during step-by-step reasoning. The authors leverage a model-agnostic framework to quantify this risk, defining leakage as risk-weighted, token-level events across eleven PII categories. They demonstrate that CoT consistently increases leakage and that these patterns vary significantly based on the model family and the reasoning budget allowed. To address this, the authors benchmark several lightweight inference-time gatekeepers, including rule-based detectors, lexical classifiers, NER models, and LLM-as-a-judge approaches, ultimately arguing that effective privacy protection requires hybrid, style-adaptive strategies rather than a single monolithic solution.

Dataset

• Dataset Composition and Sources: The authors utilize a subset of the PII Masking 200k dataset from AI4Privacy/Hugging Face Team. This dataset consists of 209,000 synthetic, human-validated texts containing 54 types of PII across various languages and use cases. The authors chose this over other sources like BigCode PII due to its higher data volume and better alignment with required PII labels.

• Subset Details and Risk Categorization: The researchers focused on a specific subset of eleven PII labels, which they organized into three risk-based tiers:

  • Group A (Mild): Includes name, sex, job title, and company name.
  • Group B (Medium): Includes date of birth, IP address, MAC address, phone number, and personal email addresses.
  • Group C (High Risk): Includes credit card numbers and social security numbers.

• Data Processing and Usage: The dataset is used to conduct leakage experiments without involving real personal information. Because the dataset provides both unmasked and masked pairs along with fine-grained labels, the authors use it to compare model outputs against specific PII annotations. This allows for the evaluation of PII leakage specifically within Chain of Thought (CoT) reasoning traces.

• Metadata and Evaluation Strategy: The authors use the synthetic nature of the data to perform aggregate analysis of token-level leakage metrics. This approach focuses on statistical trends rather than extracting specific sensitive strings, ensuring the study remains centered on privacy measurement rather than individual identification.

Method

The authors propose a methodology designed to evaluate and mitigate the leakage of Personally Identifiable Information (PII) through Chain of Thought (CoT) hijacking. The overall process is structured into three primary stages: injection, retrieval, and gatekeeping.

As shown in the framework diagram below:

In the injection phase, the researchers embed various types of PII into the context of a prompt. These PII types are categorized into three distinct risk levels: Group A (mild), Group B (medium), and Group C (high risk). This categorization allows for a nuanced assessment of how different levels of sensitivity impact privacy.

During the retrieval phase, the model under test is queried to extract the PII present in the context. The authors compare two distinct prompting strategies. The plain baseline involves direct queries, such as asking for specific email addresses. To elicit CoT, the authors employ a hijacking prompt that explicitly requests step by step reasoning and a structured JSON output containing a steps array and a final answer field. The goal is to observe if the reasoning process increases the likelihood of token level leakage.

To intercept potential CoT hijacking, the authors develop three distinct gatekeeper approaches:

1. Rule-based Gatekeeper: This approach relies on pattern matching for specific PII types. For instance, it identifies email addresses by searching for the "@" symbol, social security numbers by looking for hyphens, and credit card numbers by detecting sequences of 12 to 19 digits.
2. Lexical ML Classifier: The authors train a binary logistic regression model using TF-IDF features. This detector is trained on a balanced dataset of 2,220 samples where positive instances contain the original PII embedded text and negative instances have the PII removed. This allows the system to identify responses likely to contain PII without relying on specific patterns.
3. NER-based and LLM-as-a-Judge Gatekeepers: A more sophisticated option involves using GLiNER2, a 205M parameter generalist information extraction model. For each PII category, semantically related labels are provided to classify whether a matching span is detected above a confidence threshold. Additionally, an LLM-as-a-Judge approach utilizes a separate model, specifically GPT-o4-mini, to assess whether the primary model leaks PII during reasoning. To prevent the judge from echoing audit instructions or becoming overly verbose, the authors use a strictly delimited user message with explicit prohibitions against repetition.

The effectiveness of these methods is evaluated using several metrics. Recall is defined as the fraction of sensitive tokens from the prompt that are leaked in the output: $\mathrm { R e c a l l } = \frac { \mathrm { \# L e a k e d ~ s e n s i t i v e ~ t o k e n s } } { \mathrm { \# S e n s i t i v e ~ t o k e n s ~ p r e s e n t ~ i n ~ p r o m p t } }$Recall=#Sensitive tokens present in prompt#Leaked sensitive tokens​

To account for the varying severity of different PII types, the authors introduce the risk-adjusted F1 score, which weights the macro F1 score by the risk associated with each PII group: $\begin{array} { r l } & { F 1 _ { \mathrm { r i s k } } = \frac { \sum _ { k = 1 } ^ { n } w _ { k } F 1 _ { k } } { \sum _ { k = 1 } ^ { n } w _ { k } } , } \\ & { \quad \quad w _ { k } \in \left\{ w _ { A } , w _ { B } , w _ { C } \right\} , \quad w _ { C } > w _ { B } > w _ { A } . } \end{array}$​F1risk​=∑k=1n​wk​∑k=1n​wk​F1k​​,wk​∈{wA​,wB​,wC​},wC​>wB​>wA​.​ The weights $w_k$wk​ follow a geometric progression where $w_A = 1$wA​=1, $w_B = 3$wB​=3, and $w_C = 9$wC​=9, ensuring that high risk leaks are penalized more heavily. Finally, the Sensitive Privacy Violation ($S_{\text{Priv}}$SPriv​) score is used to measure the density of leakage within the generated text of length $|G|$∣G∣: $S _ { \mathrm { P r i v } } = \frac { 1 } { | G | } \sum _ { i = 1 } ^ { | G | } m _ { i }$SPriv​=∣G∣1​∑i=1∣G∣​mi​ where $m_i$mi​ is a binary indicator that equals 1 if the $i$i-th token is an unmasked sensitive entity.

Experiment

This study evaluates PII leakage across several closed and open-source model families by injecting sensitive information into prompts and testing both plain and chain-of-thought (CoT) prompting styles. The experiments reveal that CoT reasoning significantly amplifies privacy risks, with most models reaching high leakage plateaus once a reasoning budget is established. While GPT-o3 demonstrates superior privacy preservation compared to its peers, specialized NER-based gatekeepers like GLiNER2 provide more robust protection for high-risk data than LLM-as-a-judge approaches, which struggle with the semantically transformed leakage found in extended reasoning chains.

The authors evaluate the accuracy of different entity types in a detection task. The results demonstrate a clear variation in performance depending on the specific category of information being identified. Detection accuracy is highest for demographic information such as sex and job type. Structured identifiers like IP and MAC addresses show high levels of accuracy. Highly sensitive identifiers, including SSN and credit card numbers, exhibit the lowest accuracy levels.

The authors evaluate the performance of various gatekeeper mechanisms across different model families to prevent PII leakage. Results show that while different gatekeepers excel depending on the target model, specialized NER models provide more consistent risk-adjusted protection compared to LLM-based judges. Specialized NER models like GLiNER2 achieve better risk-weighted performance and lower leakage density for high-risk categories. LLM-as-a-Judge approaches show high performance on standard models but experience significant degradation on reasoning-intensive models. Gatekeeper effectiveness is highly dependent on the specific model being protected, with no single method dominating all scenarios.

The authors evaluate the performance of different gatekeeper mechanisms across six model families to prevent PII leakage. Results show that while different methods excel depending on the target model, specialized NER models and LLM-based judges provide varying levels of protection and recall. LLM-as-a-Judge Opus achieves the highest overall recall and Macro-F1 scores across the evaluated models. GLiNER2 provides superior risk-weighted F1 scores and lower leakage density by focusing on high-risk categories. Gatekeeper effectiveness is highly dependent on the target model, with reasoning-heavy models posing a greater challenge to detection.

The authors evaluate the performance of various gatekeeper mechanisms across different model families to prevent PII leakage. Results show that the most effective gatekeeper is highly dependent on the target model being protected. LLM-Opus achieves near-perfect performance for Llama and Mixtral models GLiNER2 serves as the best performing gatekeeper for both O3 and Opus models Rule-based methods provide the highest score for DeepSeek-R1, which appears to be the most challenging model for all gatekeepers

The authors evaluate the performance of different gatekeeper mechanisms across six model families using various PII types. Results show that while some methods achieve high raw recall, their effectiveness varies significantly depending on the target model's architecture and reasoning capabilities. The performance of gatekeeper mechanisms is highly dependent on the specific model being protected. Specialized NER models demonstrate better risk-adjusted performance compared to LLM-as-a-Judge approaches. Reasoning-intensive models present a greater challenge for standard detection paradigms.

The authors evaluate entity detection accuracy and the effectiveness of various gatekeeper mechanisms across different model families to prevent PII leakage. While detection performance varies by entity type, the choice of gatekeeper is highly dependent on the target model's architecture and reasoning capabilities. Specialized NER models offer more consistent risk-adjusted protection for sensitive categories, whereas LLM-based judges excel in certain scenarios but struggle with reasoning-intensive models.