Serving an LLM on Kubernetes: slow starts, heavy weights and the right scaling signal.
Why a model server breaks the defaults you use for web apps, and how to set probes, model storage, autoscaling and routing for AI inference.
A FIELD GUIDE TO BUILDING IN THE CLOUD
Less noise.
More know-how.
Practical guides, lessons learned, and useful resources for your cloud-native journey.
Notes from Ravindra KumarTHE LATEST FIELD NOTE
KubernetesServing an LLM on Kubernetes: slow starts, heavy weights and the right scaling signal.
Why a model server breaks the defaults you use for web apps, and how to set probes, model storage, autoscaling and routing for AI inference.
Read the guidefield-notes / kubernetes
KUBERNETES, WITHOUT THE GUESSWORK
Scale on the queue.
vllm:num_requests_waiting
# Learn something. Try it. Make it yours.
FROM THE NOTEBOOK
Worth your next read.
A GPU is not a bigger CPU. Schedule it like one and your pod waits.
How Kubernetes actually allocates GPUs to AI workloads: whole devices, taints, sharing with isolation in mind, and Dynamic Resource Allocation.
An inference platform on Azure starts with the gateway, not the GPU.
How to choose where models run on Azure, and the shared front door that turns them into a platform: identity, token limits, failover and cost per team.
Treat AI agents as platform tenants, not scripts.
What a platform team should give AI agents before they reach real tools, and three real incidents that show what happens without it.
SMALL STEPS. REAL PROGRESS.