Building the foundation for running extra-large language models

Cloudflare outlines technical improvements to Workers AI and their Infire inference engine for running extra-large language models, covering prefill/decode disaggregation, token-aware load balancing, shared KV caching (via Mooncake), speculative decoding with draft models, and multi-GPU memory and boot-time optimizations to improve latency, throughput, and cache hit ratios.