One-sentence Summary

Researchers from Stanford, CMU, UT Austin, and other institutions propose AISPA, a user-centric framework for systematically auditing system prompts in commercial AI products across eight dimensions such as transparency, safety, and fairness, and their analysis of 3,249 instructions from 88 products reveals that while protective instructions are widespread, 40% of products still contain at least one problematic instruction, highlighting the need for greater transparency, standardization, and independent oversight.

Key Contributions

• AISPA, a user-centric auditing framework, provides an eight-dimension taxonomy and a human-in-the-loop workflow for systematically evaluating system prompts.
• The framework is applied to audit 3,249 instructions from 88 commercial AI system prompts, classifying each instruction as protective or problematic.
• The audit reveals that 98.9% of products contain at least one protective instruction, yet only 24% cover all eight dimensions, and approximately 40% include instructions that work against user interests.

Introduction

System prompts are the hidden, developer-authored instructions that configure general-purpose LLMs into specific products, defining their persona, boundaries, and safety behaviors. Despite their power to shape interactions for billions of users, they are rarely disclosed and lack any systematic review, creating a governance gap: existing safety work treats system prompts as trusted artifacts to be defended against attacks, not as objects of independent scrutiny. The authors propose AISPA, a user-centric auditing framework built on eight dimensions derived from human rights principles, and a human-LLM collaboration pipeline. Applying this framework to audit 88 commercial system prompts, they find that while protective instructions have grown, roughly 40% of products still contain directives that harm user interests, comprehensive coverage is rare, and a recurring class of “gray area” instructions resists simple safety classification, underscoring the need for third-party prompt auditing.

Dataset

The authors construct a dataset of system prompts from 88 commercial AI products, sourced from six open-source GitHub repositories that contain leaked or publicly disclosed prompts. The corpus covers general-purpose chatbots, coding assistants, autonomous agents, search and research tools, and other specialized applications.

• Collection and validation: Prompts are gathered from the six repositories. Authenticity is verified by contacting repository maintainers about their curation procedures and by cross-referencing prompts for the same product across independent sources to compute content overlap.

• Annotation workflow:

  • Round 1: Claude-4.6-Opus acts as a pre-annotator to generate candidate spans.
  • Round 2: Six trained annotators independently screen the candidates. They first complete a calibration exercise on 20 spans, achieving a pairwise inter-annotator agreement of 0.933.
  • Round 3: Three experts collectively review the remaining cases and adjudicate final labels. The process yields 2,420 entries from 1,818 unique text spans.

• Label distribution: 2,346 entries are labeled as protective instructions (+1), 74 as problematic instructions (−1), and 44 entries across 29 spans are flagged as “gray area” cases for separate analysis.

• Usage in the paper: The dataset is the audit corpus. The authors do not split it into training and test sets; instead, they apply the three‑round annotation process directly to all collected prompts. The resulting labeled spans are used to study the prevalence and nature of protective versus problematic instructions in real‑world system prompts, with the gray‑area cases examined in a dedicated analysis section.

Method

The authors introduce Artificial Intelligence System Prompt Assurance (AISPA), a user-centric framework designed to evaluate system prompts in Large Language Model applications. Unlike conventional safety approaches that focus on defending against external adversarial attacks, AISPA aims to protect users from harms originating from the AI system itself. The taxonomy is anchored in the Universal Declaration of Human Rights, ensuring that auditing dimensions correspond to fundamental user rights.

As shown in the figure below, the taxonomy comprises eight distinct dimensions. Each dimension functions as a single axis to evaluate prompt spans as either protective or problematic, eliminating the need for separate taxonomies for safeguards and harmful behaviors.

The dimensions include Identity Transparency, Truthfulness and Information Integrity, Privacy and Data Protection, Tool and Action Safety, User Agency and Manipulation Prevention, Unsafe Request Handling, Harm Prevention and User Safety, and Fairness, Inclusion and Neutrality. For instance, Identity Transparency assesses whether the system discloses its AI nature (protective) or conceals it (problematic).

To operationalize this taxonomy, the authors define specific auditing guidelines centered on the prompt span as the basic unit of analysis. A span is typically a single sentence conveying a self-contained instruction. The framework distinguishes between core logic spans, which define essential product functionality and are excluded from auditing, and non-core logic spans, which impose supplementary behavioral directives with ethical or safety implications. Supplementary clauses attached to core logic spans are also auditable.

For each auditable span, the framework assigns a polarity under the relevant dimensions: $+1$+1 for Protective (promoting transparency, safety, etc.) and $-1$−1 for Problematic (encouraging deception, unsafe behavior, etc.).

Refer to the framework diagram for an illustration of this span-level auditing process.

In the example shown, a protective span regarding data privacy is labeled $+1$+1 under Privacy and Data Protection, while a problematic span instructing the AI to conceal its nature is labeled $-1$−1 under Identity Transparency.

The authors formalize a practical auditing workflow that combines the scalability of LLM-based analysis with the reliability of human judgment. This is achieved through a three-round collaborative audit protocol.

As illustrated in the figure below, the pipeline begins with Round 1, where an LLM acts as an expert pre-annotator. The model decomposes system prompts into candidate spans, identifies auditable scopes, and proposes provisional dimension and polarity assignments with rationales. This stage maximizes recall.

In Round 2, trained human annotators screen the LLM proposals to enforce precision. After a calibration phase, annotators independently review the candidates, rejecting unsupported labels and identifying missed spans. Finally, Round 3 involves a review by three domain experts who verify correctness and resolve disagreements. A strict asymmetric threshold is applied here: problematic labels ($-1$−1) are retained only if all three experts agree unanimously, reflecting the high stakes of false positives in reputation damage. This division of labor ensures comprehensive coverage, high precision, and normative consistency.

Experiment

A systematic audit of system prompts in commercial AI applications reveals that while protective instructions have become near-universal and are growing in length and coverage over time, problematic instructions persist in roughly a third of products, and comprehensive protection across all eight safety dimensions remains rare. Organization-level analysis shows leading providers like Anthropic and OpenAI have steadily increased safeguards while minimizing harmful directives, whereas others, such as Venice, exhibit more problematic than protective content, and coding assistants like GitHub and Cursor display a tension between autonomous execution and user agency. The audit also identifies a significant gray area of borderline cases—including human mimicry, parasocial dependency cues, user-enabled safety overrides, and politically charged content policies—that highlight design trade-offs between usability and user protection, demonstrating that the mere presence of a protective instruction does not guarantee adherence.

The AISPA taxonomy organizes system prompt instructions into eight dimensions, each spanning protective and problematic examples. Protective instructions emphasize transparency, truthfulness, safety, and user agency, while problematic ones conceal AI identity, suppress information, manipulate conversations, or weaken safety defaults. The patterns observed show that many problematic instructions emerge from design trade-offs such as fostering parasocial bonds or allowing user overrides, blurring the line between user experience and user protection. Identity Transparency includes both explicit AI disclosure and explicit concealment, illustrating the dimension's bidirectional nature. Truthfulness and Info Integrity is undermined by instructions that hide from users when a triggered action is not executed. Privacy and Data Protection contrasts notifying users about shared data with silently incorporating personal information without explanation. User Agency and Manipulation ranges from doing only what is asked to scripts that steer conversations and prevent closure without user consent. Unsafe Request Handling can be protective by prioritizing safety rules over user requests, or problematic by removing content restrictions entirely. Harm Prevention and User Safety shows a split between addressing emotional distress signals and outright refusing to discuss any sensitive topics.

The AISPA taxonomy categorizes system prompt instructions into eight protective and problematic dimensions, revealing that many harmful instructions arise from design trade-offs between user experience and safety, such as fostering parasocial bonds or allowing user overrides. Key patterns include bidirectional identity transparency (explicit AI disclosure vs. concealment), truthfulness undermined by silent failures, privacy breaches through undisclosed data use, and manipulation via scripted conversation steering. The taxonomy also highlights divergent approaches to unsafe request handling and harm prevention, ranging from safety rule prioritization to outright removal of restrictions, illustrating the blurred line between protection and problematic design.