Why this matters
Robotic learning commonly suffers from sparse terminal rewards: a success/failure signal does not reveal whether current behavior is improving or regressing. This survey argues that crafting and evaluating progress rewards—signals that indicate incremental task progress—is essential to make learning sample-efficient and reliable in long-horizon, compositional robotic tasks.
Key Findings
-
A unifying interface perspective: progress models differ in what they observe (eg. state, demonstration, goal specification) and what they output (scalar progress, stepwise rewards, or ranking). So what: reconciling interfaces clarifies when a method transfers across task formulations and what additional supervision is required.
-
Diverse modeling families: methods include handcrafted heuristics, supervised regressors trained on demonstrations, contrastive/ranking objectives, and learned progress estimators that leverage representation learning. So what: each family trades off data needs, generalization, and robustness; newer learned estimators scale but demand careful supervision or proxies.
-
Evaluation & data mismatch: benchmarks and supervision sources vary widely (synthetic environments, scripted demonstrations, human labels), which complicates comparisons. So what: the survey emphasizes standardized protocols and metrics to judge whether a progress signal actually speeds up learning versus just correlating with success.
Who It's For and Trade-offs
Great fit if you are a robotics or RL researcher designing reward functions or progress estimators for long-horizon, sparse-reward tasks, or if you need a taxonomy to select supervision sources and evaluation protocols. Look elsewhere if you only need low-level control primitives or purely model-free baselines where terminal rewards suffice.
Trade-offs the survey highlights: learned progress signals can improve exploration but require representative data and careful validation; simple heuristics are cheap but brittle across task variants; evaluation must measure downstream learning improvement, not just prediction accuracy.
Methods and organization
The paper structures the field into three connected steps: (1) interface—what inputs and outputs a progress model exposes, (2) interior—how models are constructed and trained (objectives, architectures, supervision types), and (3) data & benchmarks—how progress supervision is collected and how methods are evaluated. It closes with limitations (data scarcity, transferability, robust evaluation) and concrete research directions for reproducible comparisons and deployment in real robots.