One-sentence Summary

MIT researchers introduce UQ-DAAAM, an embodied spatio-temporal memory system that quantifies object-level semantic uncertainty across multi-view VLM captions, actively refines uncertain objects under a fixed query budget by selecting high-quality views and fusing the resulting captions into a single object description, and provides probabilistic guarantees that higher-quality selected views are more likely to reduce uncertainty, improving reliability and spatio-temporal question answering on the OC-NaVQA benchmark.

Key Contributions

• An object-level semantic uncertainty score for multi-view VLM captions combines cross-view semantic disagreement with per-view incoherence, enabling the memory system to flag semantically unresolved objects instead of treating stored descriptions as reliable.
• A budgeted active refinement and fusion strategy in UQ-DAAAM allocates additional view queries to uncertain objects, selects high-quality viewpoints, and consolidates the resulting multi-view captions into stable object descriptions.
• A probabilistic guarantee links DAAAM's view-quality heuristic to one-step uncertainty reduction with a closed-form update, and UQ-DAAAM achieves state-of-the-art results on OC-NaVQA with larger uncertainty reduction and better spatio-temporal question answering than baselines.

Introduction

The authors address long-horizon robot memory, where persistent semantic representations of objects, agents, and regions support downstream spatio-temporal reasoning and decision-making. Recent systems build on 3D or 4D scene graphs and use vision-language models to store open-vocabulary captions, but they typically treat VLM annotations as oracle outputs even though captions can be noisy and inconsistent across viewpoints. The key limitation is that an unreliable stored description can silently corrupt memory when the selected representative views are poor. To close this gap, the authors introduce UQ-DAAAM, an uncertainty-aware extension of DAAAM that assigns each object a semantic uncertainty score based on cross-view caption embedding scatter, then actively refines uncertain objects under a fixed budget of additional VLM queries. They also provide a probabilistic justification for view selection and a closed-form uncertainty update, reporting improved uncertainty reduction and state-of-the-art performance on OC-NaVQA.

Method

The authors build on DAAAM, a real-time 4D scene graph construction pipeline that selects a single representative frame for each tracked object fragment $o_j$oj​ from a rolling window of RGB-D frames $f_i$fi​. In the original formulation, a binary linear program assigns each fragment to one selected frame by maximizing a view-quality score $q_{ij}$qij​, subject to visibility and frame-budget constraints. The selected crops are then captioned by a localized vision-language model, and the resulting text is attached to the corresponding object node in the scene graph.

The proposed method extends this assignment from single-view to multi-view evidence. Instead of requiring exactly one selected frame per object, the authors introduce a fixed per-object budget $r$r and enforce

where $y_{ij} \in \{0,1\}$yij​∈{0,1} indicates whether frame $f_i$fi​ is a representative view for object $o_j$oj​. The visibility and frame-budget constraints are retained, while the original objective $\sum_{i,j} q_{ij} y_{ij}$∑i,j​qij​yij​ remains unchanged. Thus, each object is associated with a selected view set $\mathcal{V}_j$Vj​ of size $r$r, and the vision-language model produces one caption $c_{ij}$cij​ for each selected view.

For uncertainty quantification, the authors map each caption $c_{ij}$cij​ into a normalized semantic embedding

The normalized embeddings for all selected views of object $o_j$oj​ are stacked into a semantic evidence matrix

Because semantic spread alone can be misleading when individual captions are not reliable, the authors also assign each caption an incoherence weight

where $p_{ij}$pij​ is the average per-token probability of the caption and $\alpha \ge 0$α≥0 is a scaling hyperparameter. The weighted evidence matrix is

with $\boldsymbol{\Gamma}_j = \mathrm{diag}(\gamma_{1j}, \dots, \gamma_{rj})$Γj​=diag(γ1j​,…,γrj​). The object-level semantic scatter matrix is then

where $\lambda > 0$λ>0 is a regularization constant. The uncertainty score is defined as

The determinant measures the squared volume spanned by the weighted caption embeddings. It increases when captions from different views point in different semantic directions or when highly incoherent captions receive larger weights.

If an object has uncertainty above a threshold $\tau$τ, the agent actively acquires additional views. For object $o_j$oj​ with $t$t currently selected views, the current evidence is denoted $\mathcal{D}_j^{(t)}$Dj(t)​, and the candidate set is

The next view is chosen according to Bayesian optimal experimental design principles. The exact one-step utility of a candidate view $i$i is

To make this tractable, the authors derive a closed-form update for the augmented scatter matrix after adding view $i$i. Given the similarity vector $\boldsymbol{b}_{ij}^{(t)} = (\boldsymbol{Z}_j^{(t)})^\top \boldsymbol{z}_{ij}$bij(t)​=(Zj(t)​)⊤zij​, the augmented matrix is

They further define the redundancy score

and the Schur score

The one-step uncertainty update is then

Because evaluating the exact expected utility requires querying the vision-language model on every candidate view, the authors use the precomputed view-quality score $q_{ij}$qij​ as a cheap proxy. Under a first-order stochastic dominance assumption, a higher-quality view yields a stochastically smaller Schur score. This implies that maximizing $q_{ij}$qij​ over the candidate set is equivalent to maximizing the exact one-step BOED utility

For each uncertain object, the method uses a refinement budget $B$B. Candidate views are ranked by $q_{ij}$qij​, and the top views are queried sequentially. After each additional caption, the uncertainty score is recomputed. Refinement stops when the uncertainty drops below $\tau$τ, the budget is exhausted, or no feasible candidate views remain.

After view refinement, the authors aggregate the multi-view caption set into a single object description using a constrained LLM-based fusion step. Captions are ordered by acquisition stage, with refinement views appearing after initial views. The LLM is instructed to prefer later refinement captions when resolving conflicts. If disagreement exists only among initial views, the aggregated description preserves the alternatives with cautious language instead of collapsing ambiguity. The LLM is also constrained to use only information present in at least one source caption, which reduces the risk of hallucination in the final scene graph description.

Experiment

The experiments evaluate UQ-DAAAM on object-centric spatio-temporal question answering using the CODa and OC-NaVQA benchmarks, with a fixed budget for post hoc view refinement. Budgeted refinement results show that uncertainty-guided selection outperforms no refinement, random allocation, quality-based allocation, and alternative uncertainty signals. Uncertainty-reduction and ablation studies confirm that refinement substantially resolves initially uncertain objects, that most benefit is achieved with a small extra-view budget, and that constrained stage-aware fusion plus quality-weighted view selection contribute to the gains. Runtime and memory tests further validate that the system operates faster than the sensor rate with negligible overhead.

On OC-NaVQA, DAAAM-QR delivers the largest overall accuracy improvement and the lowest overall positional error among the compared methods, while DAAAM-Ent achieves the lowest overall temporal error. Random refinement and the other compared variants remain close to the baseline across accuracy and error metrics. Positional gains are concentrated in short and medium horizons, while long-horizon positional error stays higher than the baseline. DAAAM-QR improves descriptive question accuracy across all horizons, with the largest gain on long questions. DAAAM-QR and DAAAM-Ent reduce positional error in short and medium settings but do not improve long-horizon positional error. DAAAM-Ent achieves the lowest overall temporal error, slightly ahead of DAAAM-QR. DAAAM-RR, DAAAM-NC, and DAAAM-LN show little to no improvement over DAAAM.

At a fixed refinement budget, UQ-DAAAM reduces uncertainty substantially more than the DAAAM variants. Its uncertainty reduction rate is the highest and its mean uncertainty decrease is the largest, while DAAAM-RR leaves uncertainty nearly unchanged and DAAAM-QR falls in between. UQ-DAAAM achieves a 92.3% uncertainty reduction rate at a fixed budget B=2, compared with 34.2% for DAAAM-QR and 5.6% for DAAAM-RR. Mean uncertainty reduction follows the same ordering: UQ-DAAAM shows the largest decrease, DAAAM-QR shows a moderate decrease, and DAAAM-RR shows only a slight decrease.

The ablation study isolates the contributions of refinement budget, object selection, view selection, caption fusion, and refinement threshold. Full uncertainty-based object selection with incoherence weighting, quality-weighted view selection, constrained stage-aware fusion, and moderate threshold choices each improve accuracy while reducing position and temporal error. Most refinement benefit appears by a budget of two additional views, with diminishing returns afterward. Increasing the refinement budget improves accuracy and lowers position and temporal error, with most gains achieved by B=2 and smaller additional benefit from B=3. Full uncertainty selection performs best among object selection rules, with uncertainty without incoherence weighting close behind and caption dispersion trailing. Quality-weighted view selection gives the strongest view refinement results, while random view selection noticeably degrades performance and the direction-based heuristic closely tracks the quality-weighted policy. Constrained stage-aware fusion achieves the best accuracy and error metrics compared with no fusion, initial-only fusion, refined-only fusion, and unconstrained LLM fusion. Refinement performance improves as the threshold moves from the 10th to the 30th quantile, while the 40th quantile yields only marginal additional gain, indicating diminishing returns beyond the 30th percentile.

On the OC-NaVQA benchmark, DAAAM-QR provides the largest overall accuracy gain and lowest positional error, while DAAAM-Ent achieves the lowest temporal error, with improvements concentrated in short and medium horizons and limited long-horizon positional gains. Under a fixed refinement budget, UQ-DAAAM reduces uncertainty much more effectively than the DAAAM variants, with the largest mean uncertainty decrease. Ablation results show that uncertainty-based object selection, quality-weighted view selection, constrained stage-aware fusion, and moderate thresholds each improve accuracy and reduce positional and temporal error, with most refinement benefit realized by two additional views and diminishing returns afterward.