Evaluation metrics for humanoid motion tracking routinely disagree with human perception: frame-wise kinematic errors average pose differences but often miss the physical artifacts viewers notice most, such as foot skating, mistimed touch-downs, and unstable support. This work argues that scalable, perception-aligned evaluation — not just more data or lower per-frame error — is the right lever for diagnosing and improving contact-rich, long-horizon tracking.
Key Findings
- A new benchmark dataset: roughly 153 hours of high-fidelity optical motion trajectories recorded from 24 professional performers and organized into four motion families (Daily, Highly Dynamic, Interaction, Ground) to stress different failure modes.
- Preference-aligned metric: HumanScore, a trajectory reward model trained on 12K human-labeled pairwise comparisons (24K trajectories), predicts human preferences better than standard kinematic metrics and highlights contact/stability artifacts.
- Reveals real failure modes: across representative state-of-the-art trackers, HumanScore exposes contact timing errors, unstable support, and foot skating that per-frame joint-error metrics often miss.
- Practical dataset packaging: motions retargeted to a benchmark humanoid and released with robot-space .npz references and JSON manifests to enable scalable evaluation and per-family diagnosis.
What Sets It Apart
- Perception-first evaluation: prioritizes human preference signals via a learned reward rather than relying solely on geometric errors, so it better matches what observers consider realistic or acceptable.
- Contact- and stability-focused organization: the four motion families are chosen to elicit specific failure regimes (steady-state drift, impacts and fast phase errors, hand-body coordination, low-posture multi-contact transitions), enabling targeted diagnosis.
- Scale and diversity: substantially larger and more contact-rich than many prior test suites, making it useful for long-horizon controller and teleoperation benchmarks.
Who It's For and Tradeoffs
Great fit if you develop whole-body humanoid trackers, teleoperation controllers, or imitation systems and need evaluation that correlates with human judgments of realism and stability. Look elsewhere if your goal is low-level per-frame pose refinement only or if you cannot perform retargeting to the provided humanoid — the benchmark assumes retargeted references and evaluation in robot-space. HumanScore requires human-labeled comparisons for alignment, so reproducing or extending the metric needs additional annotation effort.
Brief Method Notes
HumanScore is trained from synchronized tracking rollouts where annotators choose which of two tracker outputs better matches the reference. The benchmark emphasizes per-family metrics to pinpoint failure modes and includes prepared clips and manifests to ease benchmarking across trackers.