One-sentence Summary

FAIR (Meta) and Univ. Grenoble Alpes/Inria propose adding dedicated register tokens to Vision Transformer input sequences to absorb high-norm artifacts that degrade feature maps, completely eliminating these artifacts in both supervised and self-supervised ViTs, setting a new state of the art on dense prediction tasks, and enabling smoother attention maps for downstream visual processing.

Key Contributions

• The paper identifies and characterizes artifacts in Vision Transformer feature maps, showing that high-norm outlier tokens in low-informative background areas are internally repurposed during inference; it also describes a simple detection method based on outlier token norms.

• This work introduces a register-token mechanism for Vision Transformers, appending input tokens that are not used as outputs and add no information, allowing the model to store and retrieve internal computations instead of hijacking patch tokens; this produces smoother feature and attention maps.

• Results show that the register-token approach removes the artifacts in both self-supervised and supervised ViT models, improves dense visual prediction and object discovery performance, sets a new state of the art for self-supervised visual models on dense prediction tasks, and generalizes to models such as DINOv2, DeiT-III, and OpenCLIP.

Introduction

Embedding images into reusable visual features is a core capability in computer vision, because annotated data is often scarce or expensive for specialized domains. Modern approaches typically pretrain strong models, such as supervised or self-supervised Transformers, and use them as frozen feature extractors for downstream tasks. However, the authors found that strong modern vision Transformers, including DINOv2, DeiT-III, and OpenCLIP, can produce artifact tokens with unusually high norms, especially in larger and longer trained models. These artifacts concentrate global image information in patches with little local content, which can harm dense prediction and object discovery. The authors address this by adding extra register tokens to the sequence, giving the model dedicated places to store global information and removing the artifacts from the patch features.

Dataset

• The authors build the dataset around three training settings used to evaluate their method: label supervised, text supervised, and self supervised learning.

• For label supervised training, they use ImageNet-22k with the DeiT-III recipe and the ViT-B configuration. This dataset provides labeled images for standard classification training.

• For text supervised training, they use a text-image aligned corpus based on Shutterstock. This subset includes only licensed image-text pairs and is used to train or modify OpenCLIP-style models with a ViT-B/16 image encoder.

• For self supervised training, they again use ImageNet-22k, but with the ViT-L configuration and the DINOv2 setup. Here, the images are used to learn visual features from unlabeled data.

• Overall, the paper uses three data sources or variants: ImageNet-22k for supervised and self supervised experiments, and a Shutterstock licensed image-text corpus for text supervised training. The reported data usage focuses on matching each dataset to a corresponding training method and model scale, without additional cropping or metadata construction details in the provided text.

Method

The authors address a diagnostic issue in modern vision transformers: as models become sufficiently large and are trained for long enough, some patch tokens develop unusually large embedding norms and behave like attention outliers. These tokens often appear in visually redundant or homogeneous regions, suggesting that the network is not merely failing at local feature extraction, but is repurposing low-information patches as informal storage slots for global information. This redistribution of roles inside the patch sequence can degrade local features and harm dense prediction tasks. As shown in the figure below:

The training dynamics further clarify the origin of the problem. High-norm outlier tokens do not appear immediately after initialization. They typically emerge in intermediate encoder layers, after a substantial portion of pretraining has occurred, and are especially pronounced in larger model architectures. This indicates that the behavior is a learned optimization outcome rather than an inherent property of the vision transformer architecture itself. The pretraining objective influences the onset of the phenomenon, but model capacity and training duration are also decisive factors. Refer to the analysis below:

Based on this diagnosis, the authors propose a simple architectural remediation: make the implicit register mechanism explicit. Instead of allowing arbitrary patch tokens to absorb global state, the model is given dedicated register tokens. After the patch embedding layer, the input sequence is augmented with learnable tokens that participate in the transformer encoder exactly like other tokens, but are not intended to be used as image outputs.

Let the patch embeddings of an image be

and let a global classification token be $x_{\text{cls}} \in \mathbb{R}^d$xcls​∈Rd. The authors add $R$R register tokens

whose initial values are learned parameters, analogous in spirit to the classification token. The encoder input becomes

These register tokens are passed through the same transformer blocks as the rest of the sequence. Through self attention, they can aggregate global context, interact with patch tokens, and broadcast information back to the sequence. Because they are not tied to a specific spatial patch, they provide neutral locations for the model to perform global storage and retrieval without corrupting patch features.

At the output of the vision transformer, the register tokens are discarded. The final representation consists only of the updated classification token and the patch tokens:

During training, the register embeddings are learned end to end with the rest of the model. At inference, they can be removed with negligible architectural cost. This design preserves the standard vision transformer workflow while preventing the model from converting redundant patch tokens into de facto memory registers.

Experiment

The authors validate register tokens by integrating them into supervised DeiT-III, text-supervised OpenCLIP, and self-supervised DINOv2 training, combining qualitative visualizations with feature norm and downstream evaluations. They find that adding registers largely removes high norm artifacts and improves local feature maps without hurting representation quality, with one register already providing much of the benefit and several registers offering gains on some tasks. Further tests show registers enhance unsupervised object discovery for DINOv2 and DeiT-III, with mixed results for OpenCLIP, and attention maps reveal that registers can naturally specialize on different objects.

Linear probing of patch tokens shows that normal tokens retain more local information than outlier tokens. Outlier tokens are harder to use for position prediction and input patch reconstruction, while their predicted features are less faithful to the original patches. This suggests that artifact-like tokens store less local patch information. Outlier patch tokens show lower accuracy for predicting position and reconstructing input patches. Outlier patch tokens have larger reconstruction distance and higher L2 error than normal patch tokens.

Linear probing on patch tokens shows that high-norm outlier patches classify images far more accurately than regular patches across the listed benchmarks. Their performance is much closer to the class token than normal patch tokens are, suggesting that these artifacts store substantial global image information. This indicates that the strongest class-level cues are carried by a small subset of unusual local features rather than by most patch representations. Outlier patch tokens achieve higher image classification accuracy than normal patch tokens on every benchmark shown. The class token still leads on many datasets, but outlier tokens often come close to its performance. Normal patch tokens carry considerably weaker class information, indicating that useful global image details may concentrate in outlier tokens.

Adding registers to the trained models largely preserves downstream performance across classification and dense prediction tasks. The largest gains appear for DINOv2 on semantic segmentation and depth estimation, while OpenCLIP also improves on depth prediction. For DeiT-III, registers keep classification performance unchanged and give slight segmentation gains. Registers do not broadly harm downstream accuracy and often improve dense task metrics. DINOv2 benefits most from registers, especially on segmentation and depth evaluation. OpenCLIP shows slightly better depth performance with registers despite similar classification.

The results compare unsupervised object discovery with LOST on feature backbones trained with and without register tokens across PASCAL VOC and COCO. Register tokens markedly improve discovery performance for DeiT-III and DINOv2, while producing a slight decline for OpenCLIP. Overall, registers often make backbones more viable for object discovery, though the effect is not uniformly positive. Registers produce the largest gains for DINOv2 across all datasets. DeiT-III performance increases substantially when registers are added. OpenCLIP shows slightly lower performance with registers.

The experiments evaluate how patch tokens encode information through linear probing and assess the effect of adding register tokens on downstream vision tasks and unsupervised object discovery. The probing results show that normal patch tokens preserve more local patch information, while outlier or high-norm tokens encode weaker local details but stronger global class cues, often approaching class token performance. Adding registers generally preserves classification accuracy and often improves dense prediction, especially for DINOv2, and it makes feature backbones more useful for object discovery for DINOv2 and DeiT, with only a slight decline for OpenCLIP.