One-sentence Summary

DORA is a self-supervised image pretraining method that discovers and tracks objects over time in continuous unlabelled videos via transformer cross-attention, deriving multiple views from the tracks for use in a self-supervised distillation loss, enabling a single Walking Tours video to become a strong competitor to ImageNet on several image and video downstream tasks.

Key Contributions

• A Walking Tours dataset of 10 unlabeled, first-person, high-resolution, hours-long, uninterrupted city-tour videos is introduced, including single-video and mixed-video splits, with the mixed split equal in size to ImageNet.

• DORA is an end-to-end self-supervised visual pretraining method adapted from DINO that uses transformer cross-attention to implicitly discover and track multiple objects across frames. It diversifies tracked object instances with Sinkhorn-Knopp clustering of patch embeddings and trains them through a classical multi-view self-supervised distillation loss.

• Pretraining on a single Walking Tours video with DORA produces image encoders that outperform ImageNet-pretrained DINO on ADE20k semantic segmentation and MS COCO object detection, despite using far less pretraining data.

Introduction

Self-supervised learning has driven major progress in vision by letting neural networks pretrain on large unlabelled image datasets, but scaling images alone raises an efficiency question because humans learn visually from continuous first person experience rather than static image galleries. Earlier video based pretraining often underperformed image pretrained models because it usually relied on object centric internet videos, treated temporal frames as simple augmentations, or still depended on large ImageNet style datasets and pretrained backbones.

The authors contribute both data and method. They introduce Walking Tours, a set of 10 long, open source first person walking tour videos that are dense, natural, and transparent to inspect. They also propose DORA, a DINO inspired self supervised pretraining method that implicitly discovers and tracks objects across frames, using Sinkhorn-Knopp clustering to encourage diverse instance signals for recognition. This approach can produce strong segmentation and detection representations from a single walking tour video, suggesting a more efficient alternative to blind dataset scaling.

Dataset

• The authors use Walking Tours, or WTours, a new egocentric video dataset collected from YouTube. It consists of 10 long walking videos filmed in Amsterdam, Bangkok, Chiang Mai, Istanbul, Kuala Lampur, Singapore, Stockholm, Venice, and Zurich, plus one Wildlife safari video.

• The source is publicly available YouTube footage released under the Creative Commons CC-BY license. The dataset is collected in a data-first way, without manual curation, class-balance control, search-term annotation, or required human labeling.

• The videos are high-resolution and long, captured at 4K resolution, specifically 3840 × 2160 pixels, and 60 frames per second. Durations range from 59 minutes for the Wildlife safari video to 2 hours 55 minutes for Bangkok, with an average duration of 1 hour 38 minutes.

• The footage is mostly continuous, with only one or two shots per video on average. The authors analyze this lack of cuts as one reason WTours is useful for learning long-range visual context compared with other video datasets.

• The dataset is visually dense. Object-detector analysis shows that WTours contains many objects per frame and an average number of object classes close to ImageNet. Compared with Epic-Kitchens and AVA, WTours has more frequent unique classes and more unique objects per frame.

• The videos show natural variation rather than curated distribution. Examples include gradual lighting changes from bright to dim to dark, transitions across diverse locations, and continuous changes in object pose, viewpoint, perspective, occlusion, relative position, and background clutter.

• The paper uses WTours as unlabeled data for unsupervised or self-supervised visual pretraining. The authors evaluate representations learned from these videos on downstream tasks, using the dataset’s long, continuous, and open-ended structure to improve visual representation learning.

• The provided text does not describe cropping, metadata construction, training split details, or mixture ratios. It focuses on the dataset’s collection, properties, and use as self-supervised pretraining data.

Method

The authors build DORA as a DINO-style self-supervised video framework that leverages temporal video structure to discover and track multiple objects without using an external object detector, optical flow network, or pre-trained tracker. The pipeline treats attention maps from the vision transformer itself as the source of object evidence. Starting from one anchor frame, the method identifies candidate object regions, refines their spatial boundaries, propagates them across subsequent frames, converts them into arbitrary-shape object crops, and uses those crops in a teacher-student distillation loss.

The input is a video clip with $T$T frames

Each frame is divided into $N = HW/p^2$N=HW/p2 non-overlapping patches of size $p \times p$p×p. The patches are linearly projected into token embeddings of dimension $d$d, and a [CLS] token embedding is prepended. The resulting sequence is processed by a vision transformer encoder, producing

The output is split into the [CLS] embedding and patch embeddings:

where $Z_t^{[\mathrm{CLS}]} \in \mathbb{R}^{1 \times d}$Zt[CLS]​∈R1×d and $\tilde{Z}_t \in \mathbb{R}^{N \times d}$Z~t​∈RN×d. Following DINO, there is a student network with parameters $\theta$θ and a teacher network with identical architecture and parameters $\theta'$θ′ updated as an exponential moving average:

The full mapping including tokenizer, encoder, and prediction head is denoted by $f_{\theta}$fθ​. The head contains an MLP and a scaled softmax, so token outputs can be interpreted as probability distributions.

Object discovery starts from an initial frame $\mathbf{X}_{t_0}$Xt0​​. In the last transformer layer of the teacher network, query and key embeddings are computed:

For a multi-head attention block with $m$m heads, these are partitioned into

with $Q^{(r)},K^{(r)} \in \mathbb{R}^{(N+1)\times d/m}$Q(r),K(r)∈R(N+1)×d/m. Each head computes self-attention:

The [CLS] attention vector of head $r$r is the row of $A^{(r)}$A(r) that measures attention from [CLS] to patch tokens:

The authors randomly select a subset $\mathcal{I} = \{i_1,\ldots,i_k\}$I={i1​,…,ik​} of $k < m$k<m heads and stack their [CLS] attention vectors:

Intuitively, each row of $A^{\mathcal{I}}$AI corresponds to an attention map that can capture a different object or salient region. To obtain object representations in the embedding space, these attention maps are used to combine patch query embeddings:

The matrix $P$P acts as a set of $k$k object prototypes derived from the initial frame.

Given these prototypes, a first attempt at temporal tracking uses cross-attention against patch keys from another frame $t$t:

Each row of $T_t$Tt​ is an attention map intended to localize one object in frame $t$t. However, the raw attention maps often overlap spatially, meaning a single map may attend to multiple objects or background regions.

To improve object disentanglement, the authors establish explicit correspondences between object prototypes and patch tokens using optimal transport. Let the teacher patch embeddings at the anchor frame be $\tilde{Z}_{t_0} \in \mathbb{R}^{N\times d}$Z~t0​​∈RN×d. The goal is to find a transport plan

that aligns the $k$k prototypes $P$P with the $N$N patches $\tilde{Z}_{t_0}$Z~t0​​. The pairwise similarity cost is taken as

The transport plan is obtained with Sinkhorn-Knopp normalization:

Here $\epsilon$ϵ is an entropic regularization coefficient. The matrix $M^*$M∗ is constrained to be nonnegative with uniform marginals: rows sum to $1/k$1/k and columns sum to $1/N$1/N. This balanced assignment encourages each prototype to be associated with a distinct set of patches while preventing all prototypes from collapsing onto the same object.

The refined object prototypes are then formed as weighted combinations of patch embeddings:

These refined prototypes are used to compute improved cross-attention maps on frame $t$t:

The refinement reduces spatial overlap among tracked objects and produces cleaner per-object attention maps. The comparison between raw and refined attention is shown below.

The refined cross-attention maps are then converted into masks for the student branch. For each object $i \in \{1,\ldots,k\}$i∈{1,…,k}, the $i$i-th row of $T_t'$Tt′​ is reshaped from $N$N attention values to a $H/p \times W/p$H/p×W/p spatial map and upsampled to $H \times W$H×W to match the original frame resolution. It is tiled along the channel dimension to form

The object-masked frame is obtained by the Hadamard product:

This operation creates an arbitrary-shape object crop whose shape is driven by attention rather than by a rectangular box. Because the prototypes are tracked over time through cross-attention, these crops correspond to temporally consistent object views.

For training, the method follows DINO’s global view scheme. For each frame $\mathbf{X}_t$Xt​, two standard globally augmented views are generated:

In addition, each global view can be transformed into masked object views $\mathbf{X}_t^{v,o_i}$Xtv,oi​​ for $v \in \{a,b\}$v∈{a,b} and $i \in \{1,\ldots,k\}$i∈{1,…,k}. The multi-object distillation loss encourages the teacher’s global representation of one view to predict the student’s representation of an object-masked version of the other view. For frame $t$t, the multi-object loss is written as

where $\operatorname{CE}$CE denotes the cross-entropy between the teacher’s [CLS] output and the student’s [CLS] output. The masked student input plays a role analogous to a local crop in DINO, but it can follow a tracked object over multiple frames and has a non-rectangular shape determined by attention.

In addition to this multi-object loss, the authors apply a local loss following the multi-crop self-supervised training design. The final objective is the sum of the multi-object global loss and the local loss, averaged over all frames in the clip:

This creates a training signal in which the model must learn representations that are invariant across augmented global views while remaining accurate for object-centered, temporally consistent masked views generated directly by the network’s internal attention.

Experiment

The experiments evaluate DORA using self-supervised pretraining on WTours videos, testing learned representations through frozen classification, unsupervised object discovery, video object segmentation, and fine-tuned dense prediction and tracking. Ablations show that single WTours videos are more effective than matched-duration object-centric or egocentric datasets, three tracked objects work best, and multi-object masking is critical for strong object-patch correspondence. Comparisons with state-of-the-art methods show that DORA generalizes well across dense and video tasks, with non-contrastive pretraining especially strong on a single WT video and additional WTours data offering only limited gains.

Walking Tours is compared with prior video datasets as an uncurated source of continuous, longer, higher-resolution online videos. It relies on data-first collection rather than search terms or manual curation, making it scalable and suited to self-supervised representation learning. Downstream results show that pretraining on Walking Tours can give strong performance on dense vision tasks and frozen-feature tasks. Walking Tours videos are longer, continuous, and higher resolution than earlier egocentric datasets. The dataset is uncurated and does not require search terms or human labeling effort. Models pretrained on Walking Tours improve dense prediction and feature-learning tasks over comparable baselines.

Dense perception results show that walking tour video pretraining improves a ViT-S/16 model across semantic segmentation, object detection, and instance segmentation. DORA pretrained on all WTours videos outperforms DINO pretrained on ImageNet-1k on these tasks, and even a single walking tour video gives competitive segmentation performance. This suggests that walking tour pretraining provides strong, data-efficient representations for dense prediction. Walking tour video pretraining improves dense scene understanding over ImageNet-1k pretraining for the same ViT-S/16 backbone. DORA trained on all WTours videos exceeds DINO trained on ImageNet-1k on semantic segmentation, object detection, and instance segmentation. A single walking tour video can produce competitive segmentation, indicating strong data efficiency for dense prediction.

Video object segmentation with frozen features shows that DORA pretrained on a single Walking Tours video improves both region similarity and contour accuracy over DINO under the same pretraining setting. It also reaches segmentation performance close to ImageNet pretrained DINO despite using much less pretraining data. Fine tuning for multi-object tracking gives DORA consistent gains over DINO across mean overlap and success rates. DORA outperforms DINO on video object segmentation when both use a single Walking Tours video for pretraining. Using only one Walking Tours video, DORA gets segmentation scores close to ImageNet pretrained DINO. DORA improves multi-object tracking over DINO by a few points on overlap and success metrics.

With features frozen after pretraining on walking tour videos, non-contrastive methods perform better than contrastive methods on both image classification and unsupervised object discovery. DORA shows a clear advantage over DINO on these frozen feature evaluations, and moving from a single walking tour video to the full walking tour dataset gives DORA only a small further gain. Non-contrastive pretraining gives stronger frozen feature classification and object discovery than contrastive pretraining on a single walking tour video. DORA outperforms DINO on both classification and unsupervised object discovery when trained on a single walking tour video. The full walking tour dataset provides only limited additional benefit over a single video for DORA.

Experiments evaluate Walking Tours as a scalable, uncurated source of continuous, longer, higher-resolution egocentric videos for self-supervised pretraining. Dense vision benchmarks show that pretraining on Walking Tours, especially with a non-contrastive approach, improves semantic segmentation, detection, instance segmentation, and frozen-feature video object segmentation and tracking relative to comparable baselines, even when using limited pretraining data. Frozen-feature evaluations further indicate strong representation quality for image classification and unsupervised object discovery, with little additional benefit from larger sets beyond a single video.