The deployment gap for embodied agents is practical: models combine perception, multi-rate prediction, and action control but existing runtimes assume request–response LLM serving and fail to meet closed-loop latency and interface needs. This work reframes deployment as a runtime contract problem — the system must support components running at different refresh rates, deterministic low-latency batch-1 inference on heterogeneous edge hardware, and extensible multimodal I/O beyond token streams.
Key Findings
- Shared execution path and five-layer architecture: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters — this modularization turns model-specific glue code into pluggable runtime pieces. So what: new embodied variants can be adopted without rewriting the whole stack.
- Latency-first fused inference and multi-rate scheduling target real-time control: evaluated VLA deployments achieved closed-loop success (HY-VLA 100.0%, pi0.5 91.0%) while respecting per-step timing constraints. So what: the runtime preserves task behavior while optimizing for low-latency control rather than throughput.
- Quantized C++ blocks (GGUF Q4_K) substantially reduce memory for WAM components: a LingBot-VA Transformer block dropped resident weight memory from 312.2 MiB to 88.1 MiB with MAE < 3.3×10^-2 and cosine similarity > 0.9997. So what: constrained edge devices can host larger embodied modules with little functional drift.
Who it's for and trade-offs
Great fit if you need to run multimodal embodied models (VLA/WAM) on robots or edge GPUs/NPUs and care about predictable, low-latency closed-loop control and modular extensibility. Look elsewhere if your primary goal is high-throughput batched serving for text-only LLM workloads or you require full end-to-end support for models not yet integrated (the repo documents supported families but broader model support is evolving). The runtime reduces memory and adapts to heterogeneous backends, but actual latency and peak memory still depend on backbone size, input complexity (e.g., multi-view video), and action-chunking choices.