FreeToken reframes the problem of running frontier MoE models locally: once sparse activation makes per-token compute feasible, the real challenge becomes orchestrating GPU, CPU, host memory and interconnect as a unified, elastic platform. The paper's core insight is that per-machine bandwidth profiles and agentic workload structure are first-order signals that should drive runtime expert placement and execution, not fixed offload strategies.
Key Findings
- Bandwidth-adaptive execution: FreeToken measures pinned PCIe transfer bandwidth and host-side processing bandwidth, and dynamically splits each step's cache misses between PCIe transfers (GPU execution) and in-place CPU execution. This converts residual host bandwidth into useful token progress without stalling cache updates.
- Semantic-aware caching and agentic checkpoints: The system anchors recurrent-state checkpoints at semantic boundaries (e.g., thinking segments, tool calls) so context edits re-prefill only the new suffix, reducing recomputation for agent workloads.
- Elastic expert-memory lifecycle: GPU expert cache is a runtime-adjustable shared LRU across layers; FreeToken can resize and rebuild the cache at scheduler safe points without restarting or reloading the host pool.
- Empirical gains: Across consumer and workstation GPUs (8GB laptop to RTX PRO 6000), FreeToken sustains substantially higher decode throughput and lower tail latency versus state-of-the-art edge serving systems, enabling interactive serving of models from ~35B up to 753B in practical settings.
Who it's for and tradeoffs
Great fit if you need to run MoE-based large models locally (development machines, privacy-sensitive deployments, or offline agents) and your hardware is heterogeneous in CPU/PCIe/VRAM balance. It is aimed at agentic workloads where routing locality and repeated agent turns can be exploited. Look elsewhere if you require minimal engineering integration and expect simpler dense models: FreeToken adds runtime complexity (profiling bandwidths, managing host-resident expert pool, dual-path execution) and is optimized for sparse MoE models rather than dense-only inference.
Where it fits
FreeToken sits between datacenter-focused serving stacks and lightweight local runtimes: it focuses on serving very large sparse models by actively trading off PCIe transfers and CPU execution, complementing systems that assume full-GPU residency or only simple prefetch caches.
How it works (brief)
FreeToken maintains a CPU-resident expert pool as the source of truth and an elastic GPU expert cache with slots for (layer, expert) tensors. During prefill it double-buffers expert streaming behind layer computation; during decode it computes a q* partition of misses between PCIe cache-fills and CPU execution using two profiled bandwidths (B_P and B_H). A shared LRU captures routing locality, and the scheduler can reshape the GPU cache at runtime to adapt to changing memory availability and workload patterns.