Running LLM Inference on Kubernetes: What It Actually Takes

This blog post provides practical guidance for running LLM inference on Kubernetes, detailing the distinct inference pipeline stages (tokenization, pre-fill, decode), cluster prerequisites (GPU device plugin, gang scheduling, networking, and custom scaling metrics), and compares three deployment options—vLLM (Helm), Ollama, and KubeAI—highlighting trade-offs around observability, autoscaling, model caching, and production readiness.