Why this matters Real-world LLM deployments are dominated by token-level behavior (time-to-first-token, inter-token latency), bursty arrival patterns, and telemetry-driven SLOs — not just single-request P95 latency. Accurate capacity planning and SLO validation therefore require a tool that can simulate realistic traffic, measure per-token timing, and integrate with observability stacks.
What Sets It Apart
- Token-aware metrics: captures TTFT, inter-token latency, per-user token throughput and related histograms so you can reason about perceived responsiveness and streaming behavior rather than only end-to-end request times.
- Realistic load and timing controls: supports concurrency, request-rate targets, Poisson/constant/gamma arrival patterns, trace replay, and prefills to exercise long-context prefill behavior — useful for both synthetic and production-trace workloads.
- Scalable, observable architecture: multiprocess runner (ZMQ-based services), optional Prometheus/DCGM collection and integrations for telemetry/result sinks (MLflow/OpenTelemetry/Weights & Biases), enabling correlation between inference metrics and server-side GPU/OS metrics.
- Extensibility and ecosystem fit: plugin system for endpoints, datasets, transports and metrics plus out-of-the-box support for OpenAI-compatible APIs, embeddings, audio, image endpoints and adapters for vLLM, Ollama and other runtimes.
Who it's for and trade-offs
Great fit if you are running or evaluating inference infrastructure and need defensible, repeatable performance profiles — SREs, infra engineers, ML engineers validating SLOs, and benchmarking teams. It helps uncover token-level responsiveness issues, resource bottlenecks, and scale limits under realistic arrival patterns.
Look elsewhere if you only need a lightweight single-request microbenchmark or simple latency probes; AIPerf’s feature set and multiprocess architecture add complexity and operational overhead for trivial checks. Very high-concurrency runs may require system tuning (socket/port limits) and familiarity with telemetry tooling to get full value.
Where it fits
Use AIPerf when you want end-to-end benchmarking that ties client-side perceived latency (TTFT/ITL) to server telemetry and can replay real traces or generate realistic traffic mixes. For quick smoke tests, a simpler HTTP load tool is faster; for SLO-driven capacity planning and per-token diagnostics, AIPerf provides the necessary measurement fidelity.
