How we built the most efficient inference engine for Cloudflare’s network

Cloudflare announces Infire, a Rust-based LLM inference engine built to maximize GPU utilization and reduce CPU overhead for edge deployments, outperforming vLLM in benchmarks on H100 NVL while avoiding the security and performance costs of sandboxed Python processes. The system features an OpenAI-compatible server, tokenizer, and a batcher that implements continuous batching with chunked prefill, a paged KV cache, JIT-compiled model-specific kernels, CUDA graphs, and selective use of cuBLASLt, yielding up to 7% faster inference on unloaded machines and significantly lower CPU consumption under real-world load. Infire currently powers Llama 3.1 8B in Workers AI, with a roadmap including Flash Attention 3, kernel fusion, multi-GPU support, quantization, and multi-tenancy.