Live e-commerce agents must answer product questions, follow shifting marketing and compliance rules, and respond in real time — a setting where runtime behavior is updated frequently but tight latency rules rule out large, slow generalist models. The paper's core insight is to make the Harness (Skills, Hooks, prompts, tool schemas) an explicit part of the training distribution so a compact model learns to condition on whatever Harness it is given instead of memorizing a single production configuration.
Key Findings
- Harness-Aware Training (HAT) combines Harness-State Augmentation (HSA), supervised fine-tuning from strong-model trajectories, general-domain on-policy distillation, and agentic RL in augmented environments. So what: this pipeline trains a compact model that both understands evolving runtime configurations and retains general instruction-following ability.
- Empirical results: a HAT-trained 35B agent scores 94.8 on Live-Stream QA (base model 80.3; strongest general LLM 93.0), 94.6 on Harness-Variant QA (base 75.4), and preserves 83.5 on IFEval where fixed-harness SFT causes a 7.7-point drop. So what: HAT avoids the overfitting/regression pitfall of fixed-harness SFT while matching or exceeding strong baselines on production-style benchmarks.
- Deployment metrics and impact: in a controlled replay on one NVIDIA H20 GPU with optimizations, the complete agent achieves P50=3.407s and P95=8.114s, and an online Taobao Live A/B test showed uplifts (UV-normalized) in confirmed-receipt GMV (+4.33%) and item-page views (+0.91%). So what: the approach meets real-time latency constraints and yields measurable product metrics in production.
Who it's for and tradeoffs
Great fit if you build low-latency conversational agents that need frequent runtime behavior updates (e.g., live-stream commerce avatars) and can invest in teacher-model trajectory generation, environment simulators, and RL infrastructure. Look elsewhere if you rely on massive zero-shot models and cannot afford the engineering cost of HSA data generation, on-policy distillation, and simulator-based RL. The method reduces the need for frequent model retraining but shifts effort into diversified harness augmentation, quality trajectory labeling, and production-aware simulator design.
Where it fits
HAT sits between two extremes: it gives small/medium models the robustness to evolve with runtime harnesses (like large general models do) while preserving the low latency required for interactive deployment. It complements engineering efforts in tool-driven agent architectures and MLOps pipelines for continuous behavior rollout.