How to Make Robot Memory More Reliable Under a Limited Query Budget? MIT Proposes UQ-DAAAM to Introduce Uncertainty Assessment for Spatiotemporal Memory

As robots gradually move from short-horizon, single-task settings toward long-horizon autonomous manipulation in open environments, they need to build long-term memory that preserves objects and their spatial relationships across time and viewpoints, in order to support tasks such as navigation, object finding, and spatio-temporal question answering.

In recent years, scene descriptions based on vision-language models (VLMs) have provided a new path for robots to build semantic memory. However, VLM outputs are often noisy and may be semantically inconsistent across different observation viewpoints—when objects are far away, occluded from a viewpoint, or in complex lighting conditions, a VLM may generate text that appears highly confident but incorrectly describes object attributes. In memory systems such as DAAAM, if the selected viewpoint happens to be of poor quality, the resulting memory entry may contain incorrect information, and the system cannot detect it.

To address this challenge, a research team from MIT introduces uncertainty scoring into an advanced spatial semantic memory system and names it UQ-DAAAM. Under a fixed query budget, UQ-DAAAM uses this score to proactively optimize objects with high uncertainty: it filters high-quality viewpoints to obtain new descriptions and then fuses the descriptions generated from multiple viewpoints into a unified object description. Experimental results show that uncertainty quantification improves the reliability and effectiveness of 4D memory systems for embodied AI. On the OC-NaVQA benchmark, compared with baseline methods, UQ-DAAAM significantly reduces uncertainty and achieves better spatio-temporal question-answering performance.

The related research results, titled "Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees", will be published at NeurIPS.

Research highlights:

• Propose an object-level uncertainty scoring method: for multi-view image descriptions, establish an object-level semantic uncertainty metric
• Propose a budget-constrained active optimization and fusion strategy: when the additional VLM query budget is limited, prioritize optimizing high-uncertainty objects and fuse multi-view descriptions.
• Provide probabilistic guarantees for the view selection strategy: theoretically establish the connection between DAAAM's view quality evaluation method and single-step uncertainty reduction, and derive the corresponding closed-form update formula

Paper link:

https://arxiv.org/pdf/2606.08277

Data Preprocessing, Scene Reconstruction, Segmentation, and Tracking

Dataset and Preprocessing

The researchers evaluate UQ-DAAAM's performance on spatio-temporal question answering using the CODa outdoor driving dataset. This dataset provides 30 Hz stereo images, depth maps generated by LiDAR projection, and globally consistent pose information.

All experiments use images captured by the front camera (cam0), with center cropping and resizing to a final resolution of 480×640 pixels. Depth data is derived from raw 3D LiDAR projection estimates and is filtered to retain only data with depths in the range [0.05, 20.0] meters. Video frames are processed at 10 Hz, consistent with the sensors' data publishing frequency.

Scene Reconstruction

The researchers use Khronos for real-time 3D scene graph construction. The TSDF (truncated signed distance field) voxel size is set to 0.2 meters, and the truncation distance is set to 0.6 meters.

The extraction parameters for Object Fragments are set as follows: minimum volume 0.005 cubic meters, maximum volume 50 cubic meters; each object fragment requires at least 8 observations, and the assignment confidence must not be lower than 0.5.

The width of Traversability Places is set between 0.4 and 2.0 meters, while label projection onto the ground plane is enabled.

Segmentation and Tracking

The researchers use FastSAM for open-vocabulary instance segmentation and compile it into a TensorRT engine. The input image resolution is 480×640 pixels, and the minimum segmentation mask area is set to 400 pixels.

Object tracking uses ByteTrack, and appearance feature matching is implemented via a CLIP-based appearance re-identification model (ReID), which is also deployed on a TensorRT engine. For tracking parameters, the confidence threshold for a new track is set to 0.6, and the track buffer length is set to 30 frames. An object trajectory must accumulate at least 8 observations before entering the semantic annotation stage.

UQ-DAAAM Framework: From Uncertainty Identification to Active Observation and Semantic Fusion

This work builds on DAAAM, which can construct a 4D scene graph in real time from an RGB-D data stream. The figure below shows the overall UQ-DAAAM framework; the construction process includes three main stages: uncertainty quantification, active view selection, and multi-view description aggregation:

① Uncertainty Quantification

To quantify object-centric semantic uncertainty, researchers introduced the Volumetric Semantic Measurement method into DAAAM's multi-view robot memory construction scenario.

In the setting studied in this paper, each object is observed from multiple selected viewpoints, and each viewpoint is accompanied by a natural-language description generated by a VLM. When descriptions generated from different viewpoints are semantically inconsistent, or when a single description itself has poor internal coherence, the system's uncertainty should increase accordingly. Therefore, the uncertainty computation method used in this study considers two aspects: Cross-view Semantic Scatter and Per-view Incoherence.

The larger the semantic directional difference among descriptions from different viewpoints, or the less coherent a single description is, the higher the computed uncertainty usually becomes. As a result, the system no longer relies solely on the confidence of one description, but instead jointly examines the consistency among multi-view information and the reliability of each individual description.

① Active View Selection

After completing the initial uncertainty assessment, the system compares object-level uncertainty with a preset threshold. When an object's uncertainty exceeds the threshold, it means that the semantic interpretation in the current memory still contains significant ambiguity and further observational evidence is needed. In this case, the system does not distribute the additional query budget evenly across all objects, but instead prioritizes objects with higher uncertainty.

Theoretically, selecting additional viewpoints can be modeled as a Bayesian Optimal Experimental Design (BOED) problem: each candidate viewpoint corresponds to an experiment, the object's latent semantic state is the quantity to be inferred, and the description generated by the visual language model under the new viewpoint is the experimental outcome. Ideally, the system should select the viewpoint that can maximally reduce expected uncertainty.

② Multi-view Descriptions Aggregation

After uncertainty-based viewpoint selection, each object is associated with a set of text descriptions from different viewpoints. Researchers fuse these descriptions into a single unified object description through a constrained, large language model (LLM)-based aggregation step. Specifically, the system orders the descriptions according to their acquisition stages: descriptions generated during the initial viewpoints are placed first, while descriptions acquired during the active optimization stage are placed later. When resolving conflicts among different descriptions, the LLM is instructed to prioritize later-acquired descriptions, i.e., those generated during the optimization stage.

If disagreement exists only among descriptions from the initial viewpoints, the fused description uses cautious wording to preserve multiple possibilities rather than prematurely eliminating ambiguity. In addition, the study imposes further constraints on the LLM: the fused result may only use information already contained in at least one source description, in order to prevent the model from generating hallucinated information.

UQ-DAAAM Achieves Better Spatio-Temporal Question-Answering Performance

The study evaluates UQ-DAAAM from multiple aspects, including downstream spatio-temporal question-answering performance, uncertainty reduction effectiveness, module ablations, and runtime efficiency.

③ OC-NaVQA Task Evaluation

In the OC-NaVQA task, the study compares UQ-DAAAM with the original DAAAM and several active optimization and uncertainty quantification baselines. Among them, DAAAM-RR allocates the additional viewpoint query budget randomly; DAAAM-QR allocates the budget to objects with lower quality based on the original viewpoint quality scores; in addition, methods constructed from different uncertainty signals, such as neighborhood consistency, next-token log-likelihood, entropy, and evidence scores, are also compared.

The results in the table below show that UQ-DAAAM outperforms all other baseline methods on all three evaluation metrics, with the most significant improvements observed in question accuracy and localization error. This indicates that uncertainty-aware active optimization can allocate a fixed semantic query budget more effectively, thereby improving memory quality.

④ Uncertainty Reduction Evaluation

In addition to downstream task performance, the study also directly evaluates whether active optimization achieves its direct objective, namely reducing object-level semantic uncertainty. The main evaluation metric used is the Uncertainty Reduction Rate (URR), defined as the proportion of objects that were judged uncertain in the initial stage and whose uncertainty dropped below the threshold after active optimization, relative to the total number of initially uncertain objects.

The uncertainty reduction results in the figure below show that UQ-DAAAM has a clear advantage over two active optimization baselines that do not rely on uncertainty. With the active optimization budget fixed at (B=2), UQ-DAAAM achieves a URR of 92.3%; the viewpoint-quality-based optimization method (DAAAM-QR) achieves 34.2%, while the random optimization method (DAAAM-RR) achieves only 5.6%. Moreover, UQ-DAAAM shifts the post-optimization uncertainty distribution noticeably to the left, indicating that this method can achieve larger and more stable uncertainty reductions across more objects.

These results show that the proposed uncertainty-aware active optimization method has significantly higher optimization efficiency.

③ Ablations

The study also used several ablation experiments to further decompose the sources of UQ-DAAAM’s performance: First, as the additional optimization budget for a single object increased from (B=1) to (B=3), the uncertainty-reduction effect continued to improve, but the marginal gains gradually diminished. As shown below, most of the gains were already achieved at (B=2); downstream OC-NaVQA performance showed the same trend, indicating that a small number of additional views are usually sufficient to resolve semantic ambiguity for most objects.

Second, for the selection of objects to optimize, the complete uncertainty metric outperformed approaches that used only cross-view description dispersion or ignored the single-view incoherence weight, indicating that considering both cross-view semantic disagreement and the quality of individual descriptions provides more effective uncertainty judgments. For view selection during optimization, random selection significantly degraded system performance; the strategy based on object observation directions performed similarly to the view-quality scoring strategy, while quality-weighted scoring could further filter candidate views.

The multi-view description fusion experiments showed that the fusion strategy also affects final memory quality. Compared with using only the single best description, fusing only initial descriptions or optimization-stage descriptions, and plain LLM fusion without conflict-handling rules, the paper’s constrained, stage-aware fusion method achieved the best performance. This means that actively acquiring more observations is not enough; how the system distinguishes information from different stages and handles conflicting descriptions also determines whether newly added evidence can be effectively converted into reliable memory.

Regarding the active optimization threshold, the experiments compared settings corresponding to different percentiles of the object-level uncertainty distribution. The results showed that performance improved markedly when the threshold was adjusted from the 10th percentile to the 20th percentile; further adjustment to the 30th percentile still yielded gains, whereas the additional benefit from the 40th percentile was limited. The researchers therefore concluded that the default threshold achieves a good balance between semantic-query cost and downstream-task gains.

Conclusion

UQ-DAAAM offers an uncertainty-aware approach to long-term spatiotemporal memory for robots: the system no longer treats descriptions generated by vision-language models directly as reliable facts; instead, it identifies potential ambiguity through cross-view semantic differences and single-view description quality, then uses active observation to supplement evidence, and forms more stable object memories through constrained description fusion. In the future, the researchers hope to use UQ-DAAAM to guide different types of robots, such as mobile robot arms, and to apply their uncertainty quantization techniques to other domains.

References:

https://arxiv.org/pdf/2606.08277 https://news.mit.edu/index%2ephp/2026/clearing-up-robot-fuzzy-memory-1105