Latency and reliability are the bottlenecks when LLMs must emit large, structured outputs (JSON, enums) repeatedly at scale. This project demonstrates a different tradeoff: when each field is constrained to a bounded candidate set (booleans or enums), you can prefill the model prefix once and evaluate all fields in parallel, turning hundreds of sequential forward passes into a single broadcasted evaluation with exact per-field probabilities and guaranteed syntactic validity.
What Sets It Apart
- Single KV-cache prefill + KV-cache broadcasting: the context and schema are encoded once; that shared cache is broadcast across M target fields so the engine performs O(1) sequential forward passes instead of O(K) token steps.
- Sub-vocabulary logit slicing and calibrated softmax: for each field only the candidate token IDs are scored and normalized, yielding exact field-level probabilities rather than heuristic sampling scores.
- Token-tree disambiguation and programmatic assembly: when choices share multi-token prefixes the engine follows continuation slices without reallocating memory, and constructs JSON directly from verified choices—this design guarantees 100% valid JSON syntax with no repair loops.
- Empirical telemetry on Apple Silicon M4 Max: using a 4-bit Qwen2.5-1.5B instruct model, reported scenarios show 5.6x–7.0x latency reductions (e.g., 270 ms vs 1,900 ms for a 28-field schema) and a ~1.1 GB 4-bit memory footprint in the reported configuration.
Who It's For and Trade-offs
Great fit if you need high-throughput, low-latency extraction or categorical routing from documents (fraud routing, ticket triage, tariff classification) and can express outputs as enums/booleans. It is also useful when you need calibrated, auditable field confidences and 100% syntactic correctness.
Look elsewhere if you require unconstrained free-form generation, open-ended key/value discovery, or multi-token natural-language fields not expressible as bounded choices. The engine assumes Apple Silicon (M1–M4 series), macOS 14+, MLX tooling and is demonstrated with mlx-community/Qwen2.5-1.5B-Instruct-4bit; portability to other runtimes is possible but will require compatible MLX-supported decoders and attention to memory/backend specifics.
