One-sentence Summary

Researchers at Zhejiang University and Zhejiang Humanoid Robot Innovation Center Co., Ltd. propose TAMP-Nav, a unified embodied navigation framework that combines pixel-to-3D action formulation, selective reasoning with anchor-trajectory memory, and GRPO-based two-level alignment, achieving a $66.2\%$66.2% success rate on R2R-CE while requiring only $90\text{k}$90k training trajectories.

Key Contributions

• TAMP-Nav introduces a Pixel-to-3D action formulation that reformulates navigation as 2D pixel selection, where selected pixels are projected into 3D coordinates for a low-level SLAM controller, aligning the action space with the VLM’s 2D visual pretraining.
• An integrated selective reasoning and anchor-trajectory memory mechanism dynamically triggers Chain-of-Thought at critical nodes and compresses redundant trajectories into lightweight Space-Time Indicators, preserving critical historical information and strengthening spatio-temporal perception.
• A two-level GRPO alignment paradigm combines global outcome rewards with fine-grained process rewards to provide dense supervision; experiments report state-of-the-art performance such as 66.2% SR on R2R-CE and high sample efficiency with only 90k training trajectories.

Introduction

Embodied navigation requires agents to follow natural language instructions through complex 3D environments. Prior vision-language model based navigators often force the model to predict low-level actions or 3D coordinates, which can cause spatial hallucinations and poor sample efficiency. Dense chain-of-thought reasoning improves planning but adds substantial inference latency, while long-horizon memory systems either overflow with redundant visual features or discard critical spatio-temporal information. The authors propose TAMP-Nav, which introduces a Pixel-to-3D action space so the model can indicate 2D pixel locations projected into 3D via SLAM, a selective reasoning and anchor-trajectory memory that triggers deep reasoning only at key nodes, and a two-level GRPO alignment framework combining global outcome rewards with fine-grained process rewards. This design aims to bridge the gap between visual-language reasoning and physical execution while improving inference efficiency, memory control, and sample-efficient navigation performance.

Dataset

• Sources and composition: MultiNav-CoT is built from VLN-CE and contains 90k trajectories. The authors filter samples to remove rendering artifacts such as black images and discard low-quality instructions identified by Gemini.
• Selective reasoning annotations: The dataset includes selective Chain-of-Thought annotations. These are designed to supervise sparse reasoning and help solve the RL cold-start problem for TAMP-Nav.
• Key node mining: For each trajectory, the authors compute an importance score $S(t) = S_{sem}(t) + S_{vis}(t)$S(t)=Ssem​(t)+Svis​(t), where $S_{sem}$Ssem​ measures CLIP similarity between the current visual observation and the instruction, and $S_{vis}$Svis​ captures scene transitions through visual feature differences. They apply distance-based greedy filtering with a minimum spatial distance $D_{min}$Dmin​ and temporal padding with a maximum gap $D_{max}$Dmax​. This yields sparse reasoning anchors covering roughly 30% of the trajectory.
• CoT generation: The authors use Gemini 2.5 Flash in a multi-stage prompt pipeline. Instead of generating a full reasoning chain in one pass, they independently generate Task Phase Localization, Current Observation Analysis, and Future Action Reasoning, then fuse them into a coherent narrative. This reduces hallucination and stabilizes annotation quality.
• Usage: MultiNav-CoT is used to provide TAMP-Nav with high-level logical reasoning and spatial understanding. Its selective CoT annotations supervise sparse reasoning. The provided section does not specify exact training, validation, or test splits, mixture ratios, or cropping-specific processing.

Method

To seamlessly bridge high-level reasoning with low-level execution while achieving high sample and inference efficiency, the authors propose TAMP-Nav.

The framework operates through a continuous interaction loop. At each navigation step, the agent first integrates current visual observations with an Anchor-Trajectory Memory to maintain long-horizon spatial awareness. Then, it autonomously decides whether to trigger Selective Reasoning to analyze complex scenarios. Following this, the model employs a Pixel-to-3D Action Formulation to directly predict a pixel waypoint, which is then converted into 3D spatial coordinates and passed to the low-level planner for execution. The entire decision-making process is optimized via Two-Level GRPO, aligning the agent to maximize navigation success while minimizing computational overhead.

In the Pixel-to-3D Action Formulation, TAMP-Nav acts as a visual pointer. At step $t$t, the agent receives four egocentric observations $V_t = \{v_{t,1}, v_{t,2}, v_{t,3}, v_{t,4}\}$Vt​={vt,1​,vt,2​,vt,3​,vt,4​} to cover a 360-degree field of view. The VLM first selects the optimal view $v_{t,i}$vt,i​ and subsequently outputs a 2D pixel $a_t = (u, v)$at​=(u,v) pointing to the target waypoint. This pixel is projected into a local 3D point $P_t$Pt​:

where $D_{t,i}$Dt,i​ denotes the depth map and $K$K represents the camera intrinsic matrix. This 3D coordinate is then transformed into the world coordinate system and dispatched to a low-level SLAM controller, which handles local movement to reach $P_t$Pt​. This formulation relieves the VLM from learning complex geometric transformations, allowing it to focus entirely on visual-semantic grounding.

To address the dilemma between full storage leading to context overflow and sparse sampling leading to information loss, the authors construct the Anchor-Trajectory Memory via two components: Explicit Anchors and Space-Time Indicators (STI). Designed to inject precise geometric positions $(x, y, yaw)$(x,y,yaw) and temporal $(t)$(t) causality into the context stream, the STI utilizes a feature fusion approach. Specifically, spatial coordinates, temporal indices, and angular information are separately encoded using 2D and 1D Rotary Position Embeddings (RoPE). These embeddings are concatenated and projected through a Multi-Layer Perceptron (MLP) to form a unified STI token:

where $[;]$[;] denotes the concatenation operation. Instead of directly encoding the scalar yaw, it is mapped to its continuous trigonometric representation $(\sin(yaw), \cos(yaw))$(sin(yaw),cos(yaw)) to prevent discontinuity at the $0^\circ/360^\circ$0∘/360∘ boundary. When the selective reasoning mechanism triggers a Chain-of-Thought (CoT) at a critical topological node $t_k$tk​, the state is stored as a triplet:

Here, $M_{STI}^{(k)} = E_{STI}(t_k, x_k, y_k, yaw_k)$MSTI(k)​=ESTI​(tk​,xk​,yk​,yawk​) provides the explicit spatiotemporal coordinate; $M_{vis}^{(k)} = v_{t_k}$Mvis(k)​=vtk​​ retains high-fidelity raw visual features to support pixel-level loop closure detection; and $M_{state}^{(k)}$Mstate(k)​ acts as a Planning Beacon by storing CoT to represent the task phase localization. For the redundant path interval $\tau$τ between two anchors, the model discards redundant visual features and retains only a pure spatiotemporal stream:

At each decision step, the dynamic input context $C_t$Ct​ concatenates a Working Memory with the Long-term Topological Memory.

To ensure the model possesses robust priors before the RL phase, the authors first perform a brief Supervised Fine-Tuning (SFT) cold start on the MultiNav-CoT dataset. After this cold start, they employ Group Relative Policy Optimization (GRPO) for further fine-tuning, designing a Dual Reward System combined with group-based relative optimization.

Local step rewards $R_{local}^{(t)}$Rlocal(t)​ guide the agent to make logical, safe, and timely reasoning decisions at each step. Five explicit components are defined: Target Approach Reward $(r_{app}^{(t)})$(rapp(t)​) encourages continuous movement toward the target; Collision Avoidance Reward $(r_{coll}^{(t)})$(rcoll(t)​) ensures physical traversability; Stop Action Reward $(r_{stop}^{(t)})$(rstop(t)​) provides supervision for termination; Reasoning Value Reward $(r_{rea}^{(t)})$(rrea(t)​) evaluates the actual utility of reasoning by correlating CoT indicators with approach scores; and Format Adherence Reward $(r_{fmt}^{(t)})$(rfmt(t)​) ensures valid spatial constraints and JSON formatting. The Total Local Reward is defined as:

The global reward evaluates the holistic quality of the completed trajectory, aggregating three metrics: Task Success Reward $(r_{suc})$(rsuc​) is the primary driver relaxing strict success to oracle boundaries; Trajectory Efficiency Reward $(r_{spl})$(rspl​) optimizes navigation efficiency via SPL; and Reasoning Density Reward $(r_{den})$(rden​) provides a non-negative efficiency bonus that discourages excessive CoT generation. The Total Global Reward is defined as:

This reward design encourages the agent to minimize cognitive overhead while ensuring task success, naturally guiding it to identify critical decision nodes and eliminate redundant reasoning.

Unlike previous methods that rely on step-by-step expert trajectories, the GRPO paradigm encourages autonomous exploration by superimposing advantages from two hierarchical levels of rollouts. For a given instruction $q$q, a group of $G = 8$G=8 complete navigation trajectories is generated. To construct each trajectory, at each decision step $t$t, the policy generates a candidate set $\mathcal{A}_t = \{a_1, a_2, ..., a_M\}$At​={a1​,a2​,...,aM​} containing $M = 4$M=4 actions via temperature-based sampling based on the current context $C_t$Ct​. These candidates are evaluated using the local step reward $R_{local}^{(t)}$Rlocal(t)​. To ensure a healthy variance of trajectory qualities for effective gradient updates, an annealed guided sampling strategy is introduced during early RL stages. The probability of actually selecting candidate $a_i \in \mathcal{A}_t$ai​∈At​ to execute and continue the trajectory is formulated as:

where $\beta_k$βk​ is a guidance coefficient at the global RL training iteration $k$k, explicitly defined as an exponential decay schedule $\beta_k = \beta_0 \cdot \alpha^k$βk​=β0​⋅αk with initial value $\beta_0 > 0$β0​>0 and decay rate $\alpha \in (0, 1)$α∈(0,1). During early stages $(\beta_k > 0)$(βk​>0) this explicitly weights the probability of selecting a candidate by its local reward. As training progresses and $\beta_k \to 0$βk​→0, $P_{select}(a_i)$Pselect​(ai​) approaches a uniform distribution $1/M$1/M. Once the $G$G trajectories are generated, $A_{global}$Aglobal​ and $A_{local}$Alocal​ are computed via separate standardizations. Specifically, $A_{global}$Aglobal​ is obtained by Z-score standardizing the global trajectory rewards across the $G$G rollouts. Concurrently, at each decision step, the local advantage $A_{local}^{(t)}$Alocal(t)​ for the actually executed action $a_i$ai​ is derived by Z-score standardizing its local reward $R_{local}^{(t)}(a_i)$Rlocal(t)​(ai​) against the $M$M candidates in $\mathcal{A}_t$At​. By independently normalizing both advantages to a standard normal distribution $(\mathcal{N}(0, 1))$(N(0,1)), their statistical scales are aligned to facilitate a more balanced integration into the superimposed total advantage $A_S^{(t)}$AS(t)​:

The policy is updated via the standard GRPO loss formulas.

Experiment

The evaluation covers standard VLN-CE benchmarks, long-horizon navigation subsets, depth robustness tests, component ablations, and real-world robot deployment. TAMP-Nav achieves state-of-the-art navigation results with strong sample and inference efficiency, while its learned reasoning concentrates on decision-critical locations such as crossroads, doorways, and target objects rather than trivial corridors. Ablations confirm the contributions of the Pixel-to-3D action space, two-level GRPO, anchor-trajectory memory with space-time indicators, and spatially consistent chain-of-thought supervision, and robustness tests plus real-world deployment indicate reliable generalization without real robot fine-tuning.

TAMP-Nav outperforms prior methods on the R2R-CE and RxR-CE validation unseen splits, achieving the highest reported success rates on both benchmarks. Among listed baselines, ETPNav posts the strongest R2R-CE success and the best RxR-CE metrics, while several earlier methods report only R2R-CE results. TAMP-Nav also uses fewer training trajectories and interaction steps than comparable systems, indicating improved sample and computational efficiency. TAMP-Nav achieves success rates above 65% on both benchmarks, improving over the strongest listed baselines by roughly 9 points on R2R-CE and 11 points on RxR-CE. Among baselines, ETPNav has the lowest R2R-CE navigation error and the highest success rate, and ScaleVLN has the highest SPL on R2R-CE. Dense reward RL lifts TAMP-Nav success relative to supervised-only training, while using fewer training trajectories and interaction steps than DualVLN and StreamVLN.

An automatic reasoning trigger strategy achieves navigation success nearly equal to a dense reasoning baseline while using only about a quarter of the reasoning budget. Fixed-interval triggering reduces reasoning density but also lowers success relative to both dense and automatic strategies. The results indicate that selective reasoning at critical decision points preserves navigation performance while avoiding redundant computation. Automatic triggering nearly matches dense reasoning success with a much lower CoT ratio. Fixed-interval reasoning underperforms both dense and automatic triggering on navigation success. The learned policy shifts reasoning away from straight corridors toward critical nodes such as crossroads and doorways.

Success rate declines as multiplicative depth noise increases. The degradation is modest at lower noise levels and becomes more pronounced at the highest tested noise level. Higher multiplicative depth noise is associated with lower success rates. The largest performance drop occurs at the highest noise level, while lower noise levels cause only minor degradations.

On long-horizon validation unseen routes exceeding 12.5 meters, TAMP-Nav with anchor-trajectory memory achieves the strongest success rate among the compared navigation systems. The ablation comparison indicates that this memory design outperforms uniform sampling and full history retention, and space-time indicators contribute a meaningful gain. TAMP-Nav with anchor-trajectory memory records a 49.8% success rate, leading the evaluated models on long-horizon tasks. Conventional memory strategies based on uniform sampling or full history retention trail the anchor-trajectory design, with full history only slightly ahead of uniform sampling. Removing space-time indicators lowers the success rate by more than four percentage points, though the ablated version still outperforms the compared baselines.

Controlled ablations under matched budgets show that replacing the SFT-only waypoint action with pixel-to-3D yields large gains, and adding GRPO with global trajectory advantage improves results further. The full Two-Level GRPO with annealed guided sampling produces the best overall navigation quality on both benchmarks. Full history memory also provides a small but consistent improvement over uniform sampling. The full Two-Level GRPO variant achieves the lowest navigation error and the highest success and path fidelity metrics across both benchmarks. Pixel-to-3D contributes substantial gains over the SFT-only baseline, and annealed guided sampling adds further improvement over Two-Level GRPO without guidance.

The experiments evaluate TAMP-Nav on R2R-CE and RxR-CE benchmarks and validate its sample efficiency, reasoning trigger, robustness to depth noise, long-horizon memory, and training ablations. TAMP-Nav sets new state-of-the-art success rates while using fewer training trajectories and interactions, with dense reward RL improving over supervised-only training. An automatic reasoning trigger preserves performance close to dense reasoning at about a quarter of the reasoning budget by concentrating reasoning at critical decision points, whereas fixed-interval triggering degrades success. Long-horizon results show anchor-trajectory memory with space-time indicators is most effective, and controlled ablations confirm that pixel-to-3D actions, global trajectory advantage, and annealed guided sampling provide meaningful navigation gains.