One-sentence Summary

Researchers from Northwestern University, Carnegie Mellon University, and other institutions present a unified survey that organizes progress reward modeling for robotic learning into three interdependent steps—model interface (what information the model receives and what form of progress signal it produces), construction methods (the assumptions and mechanisms behind progress estimation and reward generation), and validation benchmarks (how progress supervision is obtained and what evaluations measure)—to provide a comparative framework, identify current limitations, and suggest future research directions.

Key Contributions

• The paper introduces an interface-based framework that organizes progress models by their task-state representation, goal specification, and output form.
• It presents a taxonomy of the main mechanisms used to construct progress rewards and clarifies the assumptions behind each approach.
• It connects data-construction pipelines with evaluation protocols to explain what different benchmarks can and cannot validate about progress reward quality.

Introduction

The authors survey progress reward modeling for robotic learning, where dense feedback on task advancement is needed because sparse terminal success signals provide little guidance for long-horizon, sequential behaviors. Estimating progress is inherently difficult: it requires inferring a latent, history-dependent state that cannot be read directly from a single image, and the same observation can indicate different stages under different tasks. Despite rapid growth in progress-related methods, prior work remains fragmented—solutions differ in their problem formulations, input-output interfaces, and evaluation practices, making direct comparisons and clear conclusions hard to draw. To unify this landscape, the authors introduce an interface-based taxonomy that categorizes progress models by their task-state representation, goal specification, and output type; they then dissect the construction mechanisms that turn progress signals into rewards, and link data-generation pipelines with evaluation protocols to clarify what existing benchmarks can and cannot assess.

Dataset

The dataset described in the paper comprises robot trajectories annotated with progress supervision, constructed across three paradigms that vary in human involvement. Each paradigm produces a distinct type of progress data, from small-scale human-annotated sequences to large-scale automatically labeled collections.

• Human-driven data

  • Source: Teleoperated robot demonstrations or recordings of humans performing tasks.
  • Processing/annotation: Human annotators assign dense progress scores, mark success cutoffs, identify keyframes and subtask boundaries, sketch reward curves, or label undesirable behavior.
  • Scale and characteristics: Small to medium scale (limited by annotation cost); high semantic grounding (captures grasp stability, acceptable contact, implicit preferences), but subjective and hard to scale for long-horizon tasks.

• Human-in-the-loop data

  • Source: A combination of sparse human input and automated interpolation or generation.
  • Processing/annotation: Only success frames, keyframes, or stage boundaries are annotated by humans; dense progress labels are then interpolated between these anchors. Alternatively, models propose labels that humans verify and correct. In some pipelines, humans design prompts, vocabularies, or reward specifications, and LLMs or scripts generate annotations at scale, with subsequent human inspection to revise errors.
  • Scale and characteristics: Medium scale; provides semantic control with affordable annotation effort, though sparse verification may miss systematic labeling mistakes.

• Fully automated data

  • Source: Temporal structure of successful trajectories, simulator states, ground-truth rewards, expert/noisy policies, and foundation models.
  • Processing/annotation:
    • Temporal methods: later frames in success demonstrations are treated as closer to completion, yielding normalized progress, goal-proximity targets, or pairwise rankings.
    • Structured environments: simulator states, object poses, geometric predicates, and native rewards generate success labels, stage indices, continuous progress, or preference pairs. Expert policies supply successful trajectories; noisy policies create failures.
    • Foundation models: LLMs and VLMs produce subgoal decompositions, stage boundaries, progress scores, and pairwise preferences.
    • Behavior coverage augmentation: successful trajectories are truncated, reversed, paired with mismatched instructions, or perturbed with action noise to create partial, regressing, and failed examples. Filtering ensures synthetic failures remain physically plausible to avoid exploitable artifacts.
  • Scale and characteristics: Large scale and low cost; relies on the assumption that time correlates with progress, and inherits biases from the environment or annotating model.

The paper uses this multi-source progress supervision to train progress models. While no explicit mixture ratios are given, the data is likely combined to balance semantic richness and scale. Metadata such as normalized progress scores, stage boundaries, success/failure labels, and pairwise preference rankings are constructed during annotation. Augmentation strategies (truncation, reversal, noise injection) are applied to the automated data to reduce success-only bias and improve robustness, with physically implausible examples filtered out.

Method

The authors organize the interface of progress models around three core questions: how the current task state is represented, how the task goal is specified, and what form the model output takes.

For the input interface, the current task state can be represented in several ways. Single observation inputs provide the lowest latency but may struggle with visually similar states that have different meanings based on execution history. Temporal context addresses this by feeding the model a sequence of observations, such as a recent window for online estimation or a full trajectory for offline scoring. Relational comparison interfaces estimate progress by comparing two or more reference states, such as before and after an action, turning progress into a relational judgment. Finally, state access interfaces utilize environment code, simulator APIs, or proprioceptive features, which are highly effective in instrumented environments but less practical in open world settings.

Task goals are typically specified through language, vision, or structured formats. Language conditioned goals are widely used due to their compatibility with existing models, though they may underspecify physical details. Vision conditioned goals, such as goal images or demonstration trajectories, provide richer physical grounding but require curated data and are sensitive to visual variations. Structured or programmatic goals use predicates and constraints, offering precise reward construction in simulation but relying on explicitly represented state variables.

The output interface determines how the model prediction is interpreted as progress. The most common output is a state wise scalar score representing task progress, success likelihood, or goal similarity. Progress delta outputs predict whether a transition improves the task state, aligning naturally with reinforcement learning but requiring accumulation over time. Ranking outputs define progress through ordering over states or trajectories, which is easier to supervise but must be converted to scalar rewards. Programmatic methods output executable reward functions, providing flexibility and interpretability but depending on the quality of the generated code and available state variables.

The authors categorize the construction of progress rewards into four main paradigms based on how progress signals are obtained and converted into usable rewards.

The first paradigm leverages frozen foundation models as semantic reward scorers. Instead of training a dedicated reward model, these methods extract reward like signals directly from pretrained models. For instance, CLIP style methods compute image text similarity between the current observation and a language goal. Other approaches prompt a frozen vision language model with a task completion question and use the probability of specific tokens as a hidden reward. While offering zero shot usability, these scores often require normalization or prompt engineering to function as calibrated progress rewards.

The second paradigm learns rewards from temporal and relative supervision. Methods in this category use the temporal order within successful demonstrations, assuming later states are closer to completion, to construct rewards. Others rely on preference or ranking information, comparing two states or trajectories to determine which shows more progress. A frozen vision language model can generate pairwise preferences to train a scalar reward model. Relative supervision is easier to collect than absolute scores but requires converting learned comparisons into a usable reward function.

The third paradigm formulates progress estimation as an instruction tuned prediction task. Rather than relying on implicit visual representations, these methods explicitly define progress judgment through task specific prompts and fine tune vision language models to follow these instructions. Training examples pair visual observations with explicit progress related instructions and target responses. Models may be trained to predict absolute progress, task success, relative improvement, or reasoning grounded progress scores, acquiring progress estimation as a dedicated capability.

The fourth paradigm involves programmatic reward construction. Instead of estimating a reward value directly, these methods translate progress estimation tasks into executable programs, predicates, or structured reward logic. Natural language task descriptions are converted into executable reward functions, which can be refined using feedback from policy training. This approach is particularly effective when tasks can be decomposed into explicit subgoals or geometric relations, though it shifts the challenge toward task decomposition and reliable state access.

Experiment

The evaluation setup groups benchmarks into three goals: progress fidelity (calibration, temporal consistency, task grounding, and uncertainty), robustness and generalization (across tasks, viewpoints, embodiments, and non-monotonic executions), and downstream utility (policy learning, relabeling, filtering, and planning). These experiments validate whether progress models faithfully represent task progress, transfer their understanding to new settings without relying on shortcuts, and provide useful signals for robot control even when not perfectly calibrated. The overall conclusion is that progress estimation must be assessed on multiple dimensions, because a model can be useful for decision-making without being strictly calibrated, and high policy performance alone does not prove faithful progress estimation.