Why this matters Most test-time training (TTT) methods for reasoning rely on majority-vote pseudo-labels but fail catastrophically when votes are wrong because every token of a mistaken teacher is reinforced. TTPO observes an asymmetry: rollouts that disagree with the pseudo-label are usually wrong, independent of whether the vote itself is correct. Exploiting that asymmetry lets models adapt at inference time without access to ground truth while avoiding the typical collapse modes of naive self-distillation or RL.
Key Findings
- Asymmetric objective: TTPO splits updates into two branches — On-Policy Self-Distillation (OPSD) that distills agreeing rollouts, and Grouped RL that penalizes disagreeing rollouts. Token-level selection further focuses distillation on unconverged positions and restricts RL penalties to confident errors.
- Robust under noisy pseudo-labels: Both branches remain well-grounded even when majority votes are frequently incorrect, because agreeing rollouts are used as positive teachers while disagreeing rollouts act as negative signals rather than corrupting the whole teacher signal.
- Empirical gains: Without labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, lifts Qwen3-1.7B from 38.0% to 45.2% under TTT, and produces large accuracy gains (+25.2% to +36.4%) in the "without thinking" setting. The method also shows strong cross-task generalization.
- Practical mechanics: Majority-vote routing tightens self-supervision as the policy improves; token-level selection and grouped RL make the updates stable and targeted rather than globally destructive.
Who it's for and trade-offs
Great fit if you need to improve LLM reasoning performance at inference time without any labeled data, especially for deterministic/verifiable tasks like math and science reasoning where pseudo-label consensus is meaningful. It is appropriate when you can afford some extra inference rollouts and in-place parameter updates. Look elsewhere if your deployment forbids any model updates at test time, if tasks are highly open-ended (no shared canonical answers), or if tight latency budgets disallow the additional rollouts TTPO needs for reliable consensus.