A Minecraft agent can execute individual actions well and still fail at a long task: the hard part is knowing when a subtask is actually done, what to do next, and when to recover. This work’s key insight is that long-horizon autonomy depends not only on a stronger action policy, but also on a reliable handoff loop that keeps execution grounded in observed progress.
Key Findings
- A capability-aware harness uses structured memory and a vision-language planner to choose subtasks the action model can perform, reducing the gap between abstract plans and available controls.
- A completion detector conditioned on the VLA policy context flags candidate handoffs; the harness then checks visual evidence before advancing, recovering, or replanning. With the VLA held fixed, the full harness more than doubles goal success compared with fixed-horizon handoff.
- One VLA handles both GUI and non-GUI actions through regime-specific normalization branches and separate mouse-movement embeddings. The system reports 85.0% average success on MineStudio Bench and 37.4% on OpenHA, and demonstrates crafting an iron pickaxe from scratch using human-interface actions.
- Training combines injected world knowledge, imitation learning, and reinforcement learning with environment feedback, targeting recovery when actions diverge from demonstrations.
Who It’s For
Great fit for researchers studying embodied agents, visual control, and long-horizon planning under human-interface constraints. Look elsewhere if you need evidence of general-purpose performance beyond Minecraft: the results are benchmark-specific, and multi-step GUI manipulation remains a notable weakness.