One-sentence Summary

MIT CSAIL proposes Generalized Grounding Graphs ($G^3$G3), a framework that, unlike prior fixed-structure models, dynamically instantiates a probabilistic graphical model according to the hierarchical and compositional semantic structure of a natural language command, enabling autonomous robots to follow grounded instructions in semi-structured environments.

Key Contributions

• Generalized Grounding Graphs is introduced as a probabilistic graphical model for natural language navigation and mobile manipulation that dynamically instantiates a grounding graph from a command's hierarchical and compositional semantic structure, rather than assuming a fixed sequential plan structure.
• A learning procedure pairs crowdsourced natural language commands with correct robot actions to train the model, enabling it to learn meanings of complex manipulation verbs such as “put” and “take” and to infer plans and object groundings.
• Experiments in a realistic robot simulator with user evaluations show that the model can infer and execute plans for many commands from a corpus collected from untrained users, including “Put the tire pallet on the truck.”

Introduction

Natural language instructions are useful because they let people ask robots to perform mobile manipulation, such as telling an autonomous forklift to load a pallet or asking a wheelchair robot arm to fetch a book. Earlier approaches often assumed fixed, flat command structures, generated plans statically, or focused on route instructions with narrow action spaces, which limited their ability to handle varied human phrasing composed of events, objects, and places. The authors leverage the structure of the command to automatically build a probabilistic graphical model that represents uncertain groundings and candidate plans together. They also treat the grounding graph as a structured conditional random field, allowing the system to learn meanings from paired language and robot actions while inferring the most likely sequence of actions.

Dataset

• The authors use a robot-command corpus built to pair natural language instructions with grounded warehouse actions for a simulated robotic forklift.

• Source and scale:

  • The authors posted 22 videos of simulated forklift actions to Amazon Mechanical Turk.
  • The videos showed actions such as moving objects, picking up pallets, and driving to locations.
  • They collected language from 45 subjects, with multiple commands recorded for each video.
  • The corpus averages about 13 commands per video, producing a few hundred language examples overall.

• Composition and metadata:

  • Each video is paired with a full log of the environment state and the robot’s actions.
  • Each example combines a typed natural language command with the corresponding simulated action sequence.
  • Example commands include phrases such as “Pick up the pallet of boxes in the middle and place them on the trailer to the left.”

• Collection and filtering rules:

  • Subjects saw no text describing the actions or objects in the videos.
  • This led to unconstrained natural language, including misspellings and nonsensical commands.
  • The paper does not describe heavy filtering beyond collecting the typed commands.

• Annotation and processing:

  • The authors manually annotated Spatial Description Clauses, or SDCs, in each command.
  • OBJECT and PLACE SDCs were manually associated with grounded objects or locations in the environment.
  • PATH and EVENT SDCs were automatically linked to action or agent paths from the video log.
  • This automatic path and event grounding is efficient but can be ambiguous for compound commands that describe multiple stages of an action.

• Training data construction:

  • The annotated SDCs and their associated groundings provide positive examples for learning grounded language meanings.
  • The authors also create negative examples by assigning random groundings to SDCs.
  • Some random object groundings remain plausible in context, so ambiguous negative examples are re-annotated as positive.

• How the data is used:

  • The corpus is used to train a grounding model on paired command and environment information.
  • The data supports learning how linguistic constituents map to objects, places, paths, and events.
  • It also provides supervised grounding targets for both positive and constructed negative examples.

Method

The authors leverage a grounded language architecture that converts a natural language command into a robot plan by learning and then reasoning about mappings between linguistic constituents and possible world states. The environment is represented as a semantic map containing candidate objects, places, paths, and event entities. The system first parses the command into a hierarchy of Spatial Description Clauses (SDCs). Each SDC has a type, chosen from OBJECT, PLACE, PATH, and EVENT, and contains a figure field, a relation field, and up to two landmark fields. Leaf clauses typically correspond to noun-like phrases, while internal clauses combine a relation with child clauses. This decomposition preserves the argument structure of verbs and the modification structure of spatial phrases, which is critical for learning meanings from unconstrained commands.

To represent uncertainty over possible mappings, the authors introduce Generalized Grounding Graphs. Let $\Gamma$Γ be a set of candidate groundings and let each $\phi_i$ϕi​ be a binary variable indicating whether $\gamma_i$γi​ correctly corresponds to part of the command. The inference objective can be written as

$$\operatorname*{argmax}_{\Gamma} p(\Phi=\mathrm{True}\mid \text{command},\Gamma).$$Γargmax​p(Φ=True∣command,Γ).

The SDC hierarchy induces a bipartite factor graph. The variables include the correspondence variables $\phi_i$ϕi​, linguistic field variables such as $\lambda_i^f$λif​, $\lambda_i^r$λir​, $\lambda_i^{l1}$λil1​, and $\lambda_i^{l2}$λil2​, and grounding variables such as $\gamma_i^f$γif​, $\gamma_i^{l1}$γil1​, and $\gamma_i^{l2}$γil2​. Each factor connects the correspondence variable for one SDC to the words and world groundings associated with that SDC. Leaf SDCs generate factors involving their figure text and candidate grounding, while internal SDCs generate factors involving their relation words and the groundings of child clauses. The framework diagram shows this construction for a two argument command, where an event grounding connects an object grounding and a place grounding.

The induced distribution factorizes as

$$p(\Phi\mid \text{command},\Gamma) = p(\Phi\mid \mathrm{SDCs},\Gamma) = \frac{1}{Z}\prod_i \Psi_i(\phi_i,\mathrm{SDC}_i,\Gamma),$$p(Φ∣command,Γ)=p(Φ∣SDCs,Γ)=Z1​i∏​Ψi​(ϕi​,SDCi​,Γ),

and each factor is defined as a log linear potential:

$$\Psi_i(\phi_i,\mathrm{SDC}_i,\Gamma) = \exp\left(\sum_k \mu_k s_k(\phi_i,\mathrm{SDC}_i,\Gamma)\right),$$Ψi​(ϕi​,SDCi​,Γ)=exp(k∑​μk​sk​(ϕi​,SDCi​,Γ)),

where $s_k$sk​ are binary feature functions and $\mu_k$μk​ are learned weights. The graph topology is not a simple surface syntax tree, because the edges reflect semantic dependency relations rather than only linear word order. For example, in “Go to the pallet on the truck”, the phrase “on the truck” modifies “the pallet”, so the corresponding grounding is connected through the object clause rather than directly to the motion clause. A related command produces a different graph, as shown in the figure below.

The authors learn the weights from commands paired with groundings. For every factor, the model constructs features that measure whether a candidate grounding matches the linguistic fields of that factor. For a relation such as “on”, the feature representation combines a geometric relation between two groundings with the presence of the relation word. For example,

$$\mathrm{supports}(\gamma_i^f,\gamma_i^l)\land(\text{“on”}\in\lambda_i^r)$$supports(γif​,γil​)∧(“on”∈λir​)

ties the word “on” to a support relation between the figure and landmark groundings. Base features encode relations between three dimensional bounding boxes, including support, normalized distance, and Attention Vector Sum for directional relations. Continuous features are normalized and discretized into uniform bins. The model then forms binary features by taking the Cartesian product of discretized base features with word presence features in the relevant SDC fields. This allows the model to learn word specific spatial meanings while retaining geometric grounding.

For EVENT and PATH clauses, the groundings correspond to trajectories of the robot and any manipulated objects. Features are computed over complete motion trajectories, including displacement toward or away from landmark objects, average distance from a landmark, and object or place features evaluated at the beginning, end, and average states of the trajectory. For noun labeling, the model also uses word label similarity derived from lexical resources and external image tag statistics, allowing unknown object words to be associated with known visual concepts. Training maximizes the likelihood of the supervised groundings by computing gradients of the log likelihood and optimizing the weights with gradient descent and L-BFGS.

During inference, the system parses a new command, fixes the SDC structure, and searches for the grounding assignment that makes the correspondence variables most probable. Because the search space includes object assignments and feasible action sequences, the algorithm uses a beam search with a fixed beam width of twenty. It also divides inference into two passes. The first pass finds and scores candidate OBJECT and PLACE groundings. The second pass uses those candidates to search over robot action sequences corresponding to EVENT and PATH clauses. This avoids recomputing object and place candidates for every possible event. The result is a set of object and place groundings together with an executable action sequence. The inferred grounding and plan for a typical forklift command are shown in the figure below.

Experiment

Experiments were conducted using a corpus of natural language commands collected for simulated forklift actions, paired with robot state and action logs, to train and evaluate a grounding-based command understanding model. Evaluations of the cost function show that the model robustly learns correspondences between semantic discourse components and grounded world entities, though it struggles with frame-of-reference spatial phrases, ambiguous paths, and rare command terms. End-to-end evaluation, in which generated plans were simulated and judged by new annotators, shows that the system significantly outperforms a random cost baseline and produces qualitatively plausible robot behavior, with confidence estimates that help identify the most reliable command interpretations.

The learned model predicts whether spatial descriptions correspond to visual groundings with high overall reliability. OBJECT descriptions are classified most accurately, while PLACE descriptions are noticeably more difficult, especially when they involve frame-of-reference or multiple objects. PATH and EVENT descriptions show intermediate performance. OBJECT correspondence prediction is the strongest category, with near-perfect precision, recall, and F-score. PLACE correspondence prediction is the weakest category, reflecting challenges in spatial and frame-of-reference reasoning.

Annotators judged commands paired with their original videos as correct far more often than commands paired with random videos. This large gap shows that the corpus language is understandable to new evaluators and that the evaluation procedure separates valid command-video matches from mismatched ones. High agreement for known correct command-video pairs Very low agreement for known incorrect command-video pairs Confidence intervals indicate a clear distinction between correct and mismatched pairs

In human end-to-end evaluations, learned-cost search with reference semantic commands achieves the highest agreement that the robot followed the command, while constrained search with a random cost is much weaker. Automatically extracted semantic commands still perform substantially better than the random-cost baseline, though below the reference-command setting. The stronger results on the most confident plans suggest that the system's probability estimate is useful for identifying successful executions. Learned-cost search performs clearly better than constrained search with a random cost function. Automatic command extraction reduces performance relative to reference commands but remains substantially above the random-cost baseline. The highest-confidence plan subset performs better than the full test set, indicating the confidence measure is informative.

The experiments evaluate learned prediction of description-video correspondence and robot plan selection. Learned models reliably predict whether spatial descriptions match visual grounds, performing best on object descriptions, weaker on path and event descriptions, and most difficult on place descriptions involving frame-of-reference or multiple objects. Human evaluations confirm that valid command-video pairs are judged much more often as correct than mismatched pairs, while end-to-end tests show learned-cost search using reference semantic commands achieves the best perceived robot compliance, automatic command extraction remains above the random-cost baseline, and higher-confidence plans correspond to more reliable executions.