Driving systems that learn from human logs inherit human limitations and miss long-tail interactive scenarios. DriveZero answers this by separating "seeing" and "acting": perception is pretrained from massive, label-free visual models, while action is learned via closed-loop reinforcement learning in interactive worlds derived from real logs. The core insight is that perception should be trained to form general, reusable visual representations, whereas action requires closed-loop feedback and interaction to exceed demonstrated behavior.
Key Findings
-
DriveRL (action): converts driving logs into mixed-agent interactive worlds and trains a privileged 5.7M-parameter policy with PPO through closed-loop rollouts; the teacher can be queried under augmented goals to produce supervision not present in the original logs. This yields a planner that surpasses log-replay experts on nuPlan closed-loop splits.
-
DriveVFM (perception): consolidates multiple frozen vision foundation models (e.g., DINOv3, SigLIP2, SAM, Depth Anything V2) into a single image backbone without task-specific labels, producing representations suitable for downstream camera-only planning.
-
DriveZero (unification): distills the DriveRL teacher rollouts into a camera-only planner that learns from privileged, goal-conditioned rollouts rather than human trajectories. The resulting system achieves state-of-the-art performance on NAVSIMv1, NAVSIMv2 and the closed-loop HUGSIM benchmark and outperforms human log-replay on multiple nuPlan settings.
Who it's for and tradeoffs
Great fit if you need a research or engineering recipe to move beyond imitation learning for driving — especially when human logs are sparse in interactive, safety-critical, or long-tail scenarios. The approach reduces dependence on curated human demonstrations and leverages large frozen vision models to avoid per-task labeling.
Look elsewhere if you require end-to-end training purely from raw human logs with minimal simulation or if you cannot afford the infrastructure for mixed-agent closed-loop rollouts and RL training; DriveZero’s gains depend on building interactive rollouts and training a privileged teacher, which requires simulation/rollout tooling and RL compute.
Where It Fits
DriveZero sits between imitation-driven end-to-end pipelines and pure self-play/vectorized RL methods: it reuses real logs to construct interactive scenes (preserving realism) while using closed-loop RL and goal augmentation to cover behaviors absent from the logs. For teams with existing log data plus rollout/simulation tooling, it offers a pragmatic path to closed-loop robustness without manual labeling.
Method snapshot
- Perception: distill multiple frozen foundation models into a driving backbone from raw images (no task labels).
- Action: create mixed-agent interactive worlds from logs and train a privileged teacher policy with PPO; include value-guided test-time action search for improved closed-loop performance.
- Distillation: train a camera-only planner to imitate teacher rollouts, including rollouts under augmented goals to broaden behavioral coverage.
This design emphasizes task-appropriate pretraining regimes (massive visual data for perception; closed-loop interaction for action) and uses teacher-generated rollouts to escape imitation limits while retaining camera-only deployability.
