One-sentence Summary

Researchers at NVIDIA introduce AEGIS, a no-regret online adaptation framework with theoretical guarantees that deploys an ensemble of LLM safety experts (AEGISSAFETYEXPERTS) for content moderation, trained on AEGISSAFETYDATASET, a new $26{,}000$26,000 human-annotated dataset covering $13$13 and $9$9 risk categories, achieving performance that surpasses or matches state-of-the-art safety models, demonstrating robustness across jailbreak attack categories, and preserving alignment quality as measured by MT Bench scores.

Key Contributions

• This paper introduces AEGISSAFETYDATASET, a dataset of approximately 26,000 human-annotated human-LLM interaction instances supported by a content safety taxonomy that spans 13 critical and 9 sparse risk categories, filling a gap in comprehensive safety benchmarks.
• The paper presents AEGISSAFETYEXPERTS, instruction-tuned LLM safety models that surpass or match state-of-the-art safety models and general-purpose LLMs, and evaluations show robustness across jailbreak attack categories including TAP and GCG while alignment with the dataset does not degrade MT Bench performance.
• The paper proposes AEGIS, a novel application of a no-regret online adaptation framework with theoretical guarantees that performs content moderation through an ensemble of LLM safety experts, enabling the system to adapt to real-time feedback from safety compliance teams or third party evaluation systems for dynamic robustness to shifting data distributions and safety policies.

Introduction

Large language models are being rapidly adopted in real world products, which makes robust content safety moderation a critical part of deployment. Existing approaches fall into two broad categories: alignment-based methods, which fine-tune the model itself through techniques like RLHF or constitutional AI, and moderation-based systems, which rely on classifiers with fixed category heads or closed-source APIs. Both have significant drawbacks. Alignment methods are resource intensive, require harmful content to be pre-specified, and still leave models vulnerable to breaches. Moderation classifiers such as OpenAI Content Moderation and Perspective API struggle to generalize to emerging risks like self-harm or illegal activity, and recent LLM-based efforts like Llama Guard face limitations in taxonomy granularity, such as failing to distinguish between sexual content and sexual abuse of minors or between violence and hate, which are critical safety distinctions. Adversarial robustness also remains under-addressed, with existing methods often testing against a narrow set of optimized attacks rather than transferable ones.

To address these gaps, the authors introduce a multi-phased content safety system centered on a model called AEGIS. They first construct a detailed safety risk taxonomy spanning 13 major categories and 9 sub-categories, with an associated policy that defines violation boundaries for each. They then curate AEGISSAFETYDATASET, a human-annotated dataset of LLM interactions, comprising about 26,000 annotations at the time of writing, and instruction-tune a suite of LLM content safety experts on this data, benchmarking them against multiple in-domain and out-of-domain safety datasets plus adversarial jailbreak attacks such as TAP and GCG. Finally, they propose a no-regret online adaptation meta-algorithm that dynamically weights the predictions of these specialist experts based on context, allowing the moderation system to adapt over time to shifting data distributions, evolving safety policies, and novel attacks, while also learning from human compliance team feedback. The authors plan to release the dataset, taxonomy, and models to the research community.

Dataset

AEGISSAFETYDATASET Overview

The authors construct AEGISSAFETYDATASET to support the safety risk taxonomy used throughout their content moderation work. The dataset description breaks down as follows:

Taxonomy and sources

• The safety taxonomy is influenced by the OpenAI Content Moderation API, Perspective API, and Llama Guard. The authors select categories with direct overlap across these taxonomies, as overlapping categories represent the most relevant risk areas and simplify evaluation.
• They also identify differences between taxonomies. Llama Guard does not distinguish between Sexual and Sexual Minor categories, nor between Violence and Hate. Since these finer distinctions are valuable for customer needs and informed moderation, the authors keep them separate.
• Profanity and Threat categories are adopted from the Perspective API taxonomy.
• A dedicated Needs Caution category is introduced so ambiguous instances are not automatically labeled unsafe. This allows the system to choose between a defensive response or a permissive, helpful response depending on interpretation.

Dataset creation and composition

• The authors use the Hugging Face version of human preference data about harmlessness. They extract only the prompts and elicit responses from the Mistral LLM, which excels at instruction following and generates high quality responses for the moderation categories.
• To ensure diversity, system prompt examples instruct Mistral not to generate similar responses.
• The dataset includes four formats: user prompt only, system prompt with user prompt, single turn user prompt with Mistral response, and multi-turn user prompt with Mistral responses.
• A total of 26,541 examples of human-LLM interaction data are collected for annotation, with positive and negative categories balanced.

Annotation details

• Annotation is performed by a team of 12 annotators and two data quality assurance persons.
• Each instance receives at least three annotations, yielding 86,736 total annotations.
• For quality assurance, the dataset is split into 11 chunks of 1,000 to 3,000 samples each, with a 10 to 15 percent audit by the quality team. The research and engineering team also runs automatic testing every few days.
• Annotations are performed at the dialog level rather than separately for prompts and responses. Inter-annotator agreement is approximately 74 percent.

How the data is used

• The authors present the annotation distribution in Table 2, showing how examples are spread across the safety categories.
• The paper includes a sample Mistral-generated prompt and response with corresponding annotations, such as Guns/Illegal Weapons and Criminal Planning/Confessions, demonstrating how the data maps to the taxonomy.
• The dataset serves as the foundation for training and evaluating the safety classification model, with the taxonomy guiding category labels and the Needs Caution category enabling flexible moderation behavior.

Method

The authors construct a diverse ensemble of Large Language Model (LLM) based safety experts, denoted as AEGIS SAFETY EXPERTS, to handle content moderation. This ensemble prioritizes diversity in pre-training data, architecture, and safety taxonomy interpretation. The base models include Llama Guard (a 7B parameter model) and Nemotron 43B. To create variation in safety policy enforcement, the Llama Guard base model is instruction-tuned in two distinct configurations: a defensive variant (LLAMAGUARDDEFENSIVE) that maps the "Needs Caution" category to "Unsafe", and a permissive variant (LLAMAGUARDPERMISSIVE) that maps "Needs Caution" to "Safe". Additionally, a defensive variant of the Nemotron model (NEMO43B-DEFENSIVE) is included.

The training of these experts utilizes Low-Rank Adapters (LoRA) to fine-tune the models on the curated dataset. For the Llama Guard base model, the authors employ LoRA with a rank $r = 16$r=16, a context length of 4096, and a learning rate of $10^{-6}$10−6 over 3 epochs. The Nemotron 43B model is adapted using the NGC NeMo LLM Service Customization framework with a higher rank $r = 32$r=32, a learning rate of $5 \times 10^{-6}$5×10−6, and 10 epochs. This fine-tuning aligns the experts with the specific safety taxonomy and policy guidelines.

The core of the system is a meta-algorithm named AEGIS, which applies a no-regret online learning framework to dynamically select the most effective expert from the ensemble. This process is modeled as a game between the learning algorithm and the environment. At each round $t$t, the learner selects an expert $I_t$It​ from the set $E$E of experts. The environment then reveals a loss function, and the learner incurs the loss $l_t(I_t)$lt​(It​) associated with the chosen expert. The objective is to minimize the expected regret $R_T$RT​ over $T$T rounds, defined as:

The authors implement the Exponential Weights (EW) algorithm, also known as Hedge, to manage this selection process. Initially, each expert is assigned a weight of 1. In every round, the algorithm selects an expert randomly from a distribution proportional to their current weights. Upon receiving feedback from an oracle (such as a human moderator or an LLM judge), the weights are updated to decrease the influence of experts that incur higher losses.

To balance computational feasibility with adaptive performance, the system operates in alternating phases. During adaptation phases of length $m$m, the algorithm collects feedback to update expert weights and identify the best performer. This is followed by compliance phases of length $p$p, where the currently best expert is deployed for moderation while its performance is monitored. This cycle allows the system to dynamically adapt to real-time data and policy feedback. The effectiveness of this online adaptation is illustrated in the performance curves, where the algorithm converges toward the best expert's performance over time.

To enhance robustness against adversarial attacks and to prevent the algorithm from getting stuck with a suboptimal expert if performance shifts over time, the authors introduce a perturbation mechanism to the weight updates. This involves sampling a value from a Gumbel distribution and adding it to the weights during the update step, formulated as:

This perturbation introduces controlled randomness, allowing the learner to explore other experts more frequently. The stability and performance of this perturbed version are demonstrated across multiple trials, showing consistent error reduction over the time horizon.

Experiment

The evaluation compares AEGISSAFETYEXPERTS against baselines such as Llama Guard and OpenAI's moderation API across three safety benchmarks plus the SimpleSafetyTests suite. The experts, trained on only half of the AEGISSAFETYDATASET, outperform all baselines in AUPRC or F1 across all datasets while baselines struggle with out-of-distribution data. Beyond static evaluation, a no-regret online learning framework with exponential weights is applied to ensemble the experts, with an oracle providing feedback, and this adaptation algorithm asymptotically converges to the best performing expert, with a perturbed weight version showing better robustness when expert performance shifts over time.

The AEGIS safety dataset defines a content safety risk taxonomy that merges overlapping categories from existing industry taxonomies while adding finer-grained distinctions, such as separating Sexual from Sexual Minor and Violence from Hate. It also incorporates categories like Profanity and Threat from Perspective's taxonomy. The design prioritizes both coverage and granularity to support nuanced content moderation. The taxonomy is built from overlapping categories across OpenAI, Perspective, and Llama Guard to focus on the most relevant risk areas. Unlike Llama Guard, it distinguishes between Sexual and Sexual Minor as well as Violence and Hate for more detailed moderation. Profanity and Threat categories are adopted from Perspective's taxonomy.

The AEGIS Safety Dataset includes 26,541 human-LLM interaction examples annotated across seven risk categories, with a highly imbalanced distribution. Hate/Identity Hate and Violence are the most prominent categories, while Threat and Sexual Minor are the least represented. Hate/Identity Hate receives the highest number of annotations, followed by Violence. Threat and Sexual Minor have the fewest annotations, each under 100 examples. Suicide and Self Harm and Guns/Illegal Weapons appear at intermediate frequencies, with counts in the hundreds.

The AEGIS safety experts, when instruction-tuned on only half of the AEGIS safety dataset, outperform base models on both off-policy and on-policy benchmarks in either AUPRC or F1 scores. They also show stronger generalization to out-of-distribution data compared to existing APIs and baselines, which struggle on novel or near-distribution safety risks. AEGIS defensive and permissive variants achieve higher AUPRC than the base Llama Guard on the on-policy test set, with defensive reaching 0.941 AUPRC. On the ToxicChat off-policy benchmark, both AEGIS variants improve AUPRC over both Llama Guard base and the OpenAI Mod API. Existing APIs like Perspective and OpenAI Mod show much lower F1 scores on the on-policy dataset (0.24 and 0.34) compared to AEGIS variants (0.76 and 0.85).

The table compares accuracy on SimpleSafetyTests for critical safety risks, showing that the proposed defensive and permissive models achieve the highest scores, including perfect accuracy for the defensive LlamaGuard variant. All in-house models outperform external baselines such as Perspective API, LlamaGuard base, and GPT-4. The defensive NEMO model also performs competitively despite restrictions on safety instruction length during training. The proposed LlamaGuard defensive variant achieves perfect accuracy, the highest among all evaluated models. In-house models consistently outperform external baselines, with the smallest gap being over 4 percentage points. The defensive NEMO model remains competitive even with constrained safety instruction length in its system prompt.

The experiments validate the AEGIS safety taxonomy and dataset, which integrates and refines categories from existing industry taxonomies, and show that instruction-tuned models using this data significantly outperform base models and external APIs on both on-policy and off-policy benchmarks. The AEGIS variants demonstrate stronger generalization to out-of-distribution risks and achieve higher scores on critical safety tests, with the defensive LlamaGuard variant reaching perfect accuracy, while existing APIs lag notably on the imbalanced dataset. Overall, the findings confirm that the tailored taxonomy and training approach provide a more effective and robust content moderation solution.