Making Workers AI faster and more efficient: Performance optimization with KV cache compression and speculative decoding

Cloudflare announces performance upgrades to Workers AI, including newer GPUs supporting larger Llama models, an open-source KV-cache compression method using paged attention to reduce memory usage by up to 8×–64× with minimal quality loss (yielding 3.44×–5.18× throughput gains), and speculative decoding via prompt-lookup to increase generation speed by up to 40% (8B) and 70% (70B), delivering lower-latency, higher-throughput inference for customers.