One-sentence Summary

Researchers from SUSTech, Tencent Robotics X, Peking University, and Futian Laboratory propose ReferTrack, a referring-then-tracking paradigm that first selects the target from indexed bounding boxes using a single forward-facing camera, then decodes tracking waypoints while preserving motion cues via a sliding-window queue of previous boxes injected through temporal-viewpoint-bbox indicator (TVBI) tokens and co-training on a custom Refer-QA dataset, achieving state-of-the-art single-view performance on EVT-Bench and robust sim-to-real transfer.

Key Contributions

• ReferTrack proposes a referring-then-tracking paradigm that grounds target identification in the selection of an indexed image-space bounding box, avoiding abstract latent reasoning and aligning decisions with explicit visual evidence.
• The method introduces a sliding-window queue of previously selected bounding boxes injected into the visual stream via temporal-viewpoint-bbox indicator (TVBI) tokens, preserving target motion cues and transforming spatiotemporal indexing into target-conditioned memory.
• Co-training on a custom Refer-QA dataset using the same indexed catalog strengthens catalog-based referring, enabling state-of-the-art single-view performance on EVT-Bench (89.4%, 73.3%, 74.1% on single-target, distracted, and ambiguity splits) and successful sim-to-real transfer to legged and humanoid robots.

Introduction

Embodied visual tracking enables mobile robots to continuously follow a person described in natural language, relying only on onboard vision in crowded, dynamic settings. Success requires tightly coupling target identification with collision-free trajectory planning. Earlier pipelines treat these as separate stages, using vision foundation models for identification and learned planners for navigation, but errors from recognition can propagate downstream. Recent vision-language-action (VLA) models unify identification and planning via next-token prediction; large-scale co-training with other navigation tasks can build general priors, while specialized trackers like TrackVLA++ introduce chain-of-thought reasoning over abstract spatial codes. However, reasoning in a latent spatial space often fails to ground the target precisely in visual scenes, especially under clutter and ambiguity. The authors propose a referring-then-tracking paradigm that casts target identification as selecting one indexed bounding box from a set of onboard detections, effectively turning the problem into a constrained multiple-choice task. Their model, ReferTrack, organizes current detections into an indexed catalog, emits a single Refer-CoT token to choose the target, and propagates selected boxes through a sliding-window queue to provide motion cues before trajectory prediction. Co-training on a custom Refer-QA dataset built from person re-identification data reinforces this catalog-based referring prior. The result is an end-to-end policy that grounds reasoning in image-space evidence, achieving state-of-the-art single-view tracking performance and strong sim-to-real transfer on legged and humanoid robots.

Dataset

The authors construct the training data for ReferTrack from two complementary sources, both organized around a shared indexed-bbox catalog formulation.

• Navigation data (1.3M trajectories)

  • Source: Expert tracking trajectories curated from the EVT-Bench training split inside the Habitat 3.0 simulator.
  • Each sample contains forward-view visual observations, a natural language target description, a detection-based candidate catalog, the ground‑truth Refer-CoT step index, and the expert waypoints.

• Refer-QA data (1.3M samples)

  • Source: Custom-built referring expression samples synthesized from the SYNTH-PEDES dataset, the same base dataset used by earlier Embodied Visual Tracking methods.
  • Purpose: Explicitly strengthen visual grounding skills through static referring questions.

Both subsets are used for co‑training at a 1:1 mixture ratio within a two‑stage Supervised Fine‑Tuning pipeline. Stage 1 aligns the vision projectors with general multimodal QA data; Stage 2 performs full fine‑tuning jointly on the navigation and Refer‑QA tasks. Because the language instruction, the discrete catalog, and the visual tokens are sequenced identically in both tasks, the referring capabilities learned from the static QA supervision transfer directly to the dynamic online tracking setting.

Method

The authors propose ReferTrack, a Vision-Language-Action (VLA) framework designed for Embodied Visual Tracking (EVT). The model extends recent VLA architectures into a dual-branch system capable of simultaneous navigation and question-answering. Given a natural language instruction $L$L and a stream of forward-view RGB observations $\mathcal{O}_{1:T}$O1:T​, the agent predicts a continuous trajectory $\mathcal{W}_T$WT​ of egocentric displacements and heading changes.

As shown in the figure below:

The architecture processes visual inputs through a dual-encoder setup utilizing SigLIP and DI-NOv2. A grid pooling strategy extracts fine tokens $V^{\text{fine}} \in \mathbb{R}^{64 \times C}$Vfine∈R64×C for current observation details and coarse tokens $V^{\text{coarse}} \in \mathbb{R}^{4 \times C}$Vcoarse∈R4×C for broader historical context. To balance long-range context with inference latency, the system maintains a sliding window of the latest $H$H frames, organized as $\mathcal{V}_T = \{V_{T-H}^{\text{coarse}}, \dots, V_{T-1}^{\text{coarse}}, V_T^{\text{fine}}\}$VT​={VT−Hcoarse​,…,VT−1coarse​,VTfine​}. A two-layer MLP projector $\mathcal{P}_{\text{vision}}(\cdot)$Pvision​(⋅) maps this stream into the LLM latent space. To explicitly inject target geometry into the visual history, the authors interleave temporal-viewpoint-bbox indicator (TVBI) tokens between frame visual token groups. For a valid normalized box $b_t$bt​, it is embedded via a shared two-layer MLP $\mathcal{P}_{\text{bbox}}(\cdot)$Pbbox​(⋅) and added to the TVI token:

If the target is unobserved, a deterministic absence sentinel is used. Crucially, current-frame fine tokens rely strictly on TVI-only indicators, forcing the model to ground the target using historical TVBI cues and raw visual features.

For image-space referring, the system constructs a discrete candidate catalog $\mathcal{C}_T = \{\langle ped_1 \rangle, \dots, \langle ped_K \rangle, \langle \text{NO\_EXIST} \rangle\}$CT​={⟨ped1​⟩,…,⟨pedK​⟩,⟨NO_EXIST⟩} using an off-the-shelf real-time object detector. Each candidate entry is represented by a dedicated identifier token followed by a bbox token computed as $E_{\text{bbox}} = \mathcal{P}_{\text{bbox}}(b_T^{(k)})$Ebbox​=Pbbox​(bT(k)​). A fixed virtual index $\langle \text{NO\_EXIST} \rangle$⟨NO_EXIST⟩ handles cases where the target is entirely absent from the surroundings.

The core reasoning and planning occur in two sequential LLM stages. In the first pass, the model generates a single Refer-CoT token $E_T^{\text{refer}}$ETrefer​ via classification over the discrete vocabulary of registered special tokens. This strictly one-token decision selects the indexed bbox that best aligns with the instruction $L$L. In the second pass, the LLM utilizes $E_T^{\text{refer}}$ETrefer​ as a conditioning prefix to generate an action token $E_T^A$ETA​, which a dedicated MLP head decodes into $M$M waypoints $\mathcal{W}_T$WT​.

To maintain temporal consistency, the authors implement a Referred-Target Bbox Queue. Once $E_T^{\text{refer}}$ETrefer​ is resolved, the corresponding bbox is pushed into a first-in-first-out queue of capacity $H-1$H−1. This queue conditions the historical TVBI stream for subsequent timesteps, closing the reasoning loop by propagating the tracking decision into the geometric history.

The framework is optimized using a weighted sum of three loss functions:

where $\alpha$α is set to 10. The trajectory loss $\mathcal{L}_{\text{traj}}$Ltraj​ minimizes the Mean Squared Error between predicted and expert waypoints. The referring loss $\mathcal{L}_{\text{refer}}$Lrefer​ supervises the Refer-CoT step via cross-entropy over the ground-truth target index. The text loss $\mathcal{L}_{\text{text}}$Ltext​ computes standard cross-entropy over text tokens for Refer-QA samples, bypassing the action head to focus entirely on visual grounding and language alignment.

Training leverages two complementary datasets sharing the unified indexed-bbox catalog formulation: 1.3M expert tracking trajectories for navigation and 1.3M custom-built referring samples for Refer-QA. The authors employ a two-stage Supervised Fine-Tuning recipe. Stage 1 aligns vision projectors using general multimodal QA datasets. Stage 2 jointly performs full fine-tuning on both navigation and Refer-QA tasks, ensuring that robust referring skills transfer seamlessly to dynamic online tracking.

Experiment

ReferTrack is evaluated on the EVT-Bench suite within Habitat 3.0 using a single forward-facing camera, comparing against classical trackers, offline and RL-based methods, and VLA policies on tracking tasks with distractors and ambiguity. Quantitative results demonstrate that a compact, SFT-only policy equipped with explicit image-space referring outperforms all single-view baselines and matches multi-camera methods, highlighting that referring is more effective than wider camera coverage when target disambiguation is the main challenge. Ablation studies confirm that target identification is the primary bottleneck in distracted scenarios, and that the referring-then-tracking design (Refer-CoT and TVBI) provides robust target selection and temporal stabilization. Real-world deployments on a quadruped and a humanoid robot further validate the approach, showing reliable target following under partial views and multi-person interference.

A compact 4B-parameter model trained with only supervised fine-tuning on a modest dataset achieves the best single-view tracking performance on EVT-Bench. It substantially outperforms the strongest single-view baseline on distracted and alternating tracking, and matches or exceeds multi-camera references on identification-heavy splits. The referring interface does not compromise standard tracking stability, demonstrating that explicit referring capabilities can bridge gaps typically addressed by larger models, extra cameras, or reinforcement learning. ReferTrack improves over the strongest single-view baseline, TrackVLA++, by +6.8 SR and +13.0 TR on DT, and +22.9 SR and +22.3 TR on AT. On the identification-heavy DT and AT splits, ReferTrack matches or surpasses reported multi-camera baselines while using only a single forward camera.

Ablation on the Distracted Tracking split reveals that target identification is the primary bottleneck. An oracle variant with perfect identification reaches 81.5% success rate, close to the expert policy's 85.1%, while the full model achieves 73.3%. Removing the explicit referring mechanism causes a sharp drop to 55.7%, confirming that image-space target selection is crucial for robustness under distractors. An oracle using ground-truth bounding boxes without Refer-CoT achieves 81.5% success rate, far above the full model's 73.3%, indicating that target identification, not motion planning, is the main bottleneck. Removing both Refer-CoT and TVBI drops success rate from 73.3% to 55.7%, showing that explicit image-space target selection is the primary source of distractor robustness.

A compact 4B-parameter model trained only with supervised fine-tuning achieves state-of-the-art single-view tracking on EVT-Bench, substantially outperforming strong baselines on distracted and alternating tracking while matching multi-camera systems using only a forward camera. Ablations reveal that target identification is the primary bottleneck, and the explicit image-space referring mechanism is essential for distractor robustness, with its removal causing a sharp performance drop. The approach shows that explicit referring capabilities can bridge gaps typically requiring larger models, extra cameras, or reinforcement learning.