The 3 Levels of AI Inference Engineering

There are three levels of inference engineering, and a lot of people are still stuck at level one.

Here's what each level covers, the free resources I'd use to prepare for it, and the project that gets you to the next one. The projects come from my 5 AI Infrastructure Projects to Build post, so you'll have a step-by-step guide for each.

Save this one.

Level 1: AI Infrastructure Fundamentals

This layer builds on your cloud, DevOps and ML foundations. It can be a little advanced at first, but the role and the pay that come with it make the learning curve worth it.

What you should be familiar with

• Running LLM APIs and application deployments

• GPU-based Kubernetes clusters

• Open-weight vs. foundation models

• ML workloads: pre-training, post-training and inference

Crash courses

• Cloud: pick one to start with, AWS, GCP or Azure

• DevOps: DevOps concepts, Networking and Docker

• Application deployments: CI/CD and GitOps

• LLM APIs: APIs crash course, starting at the chapter on APIs in AI applications

• GPU-based Kubernetes: Kubernetes for DevOps + AI, Part 1

• ML workloads: AI Inference Crash Course: What is AI inference?, where inference sits next to training

Go deeper

• Intro to Large Language Models by Andrej Karpathy: one hour on how LLMs are built, trained and used, including open vs. closed models

• Hugging Face LLM Course: hands-on with open-weight models

• Scheduling GPUs in Kubernetes and the NVIDIA GPU Operator docs: how pods actually get a GPU

• AI Infrastructure & Operations Fundamentals by NVIDIA: the course I mention in the video

What gets you out of this level: build an AI infrastructure project, like deploying a model locally.

→ Build: Project 1, Dockerfile generation with an LLM. Run an open-weight model locally, feed it your project files and let it write and fix a Dockerfile.

• GitHub: ai-assisted-devops (reference project), Ollama (runs the model locally) and Magnitude (finds a model that fits your machine)

→ If your DevOps foundation feels shaky, add Project 3, End-to-end DevOps + AIOps platform. Deploy an app with GitOps, add metrics, logs and alerts, then use an LLM to summarize incidents and suggest root causes. The CI/CD and GitOps crash courses cover its pipeline and Argo CD parts.

• GitHub: devops-ai-playbook

Level 2: Understanding Inference

This is where you go beyond deploying models and understand what's happening under the hood.

What you'll focus on

• Model architecture: the transformer, the prefill and decode phases and, most importantly, KV cache memory management

• Inference engines: vLLM, SGLang and TensorRT-LLM. How they work, how they manage memory and how they reuse the KV cache

• Serving patterns: aggregated vs. disaggregated serving

• Kubernetes: where scaling and scheduling AI workloads come together

Crash courses

• AI Inference Crash Course chapters:

  • Prefill

  • Decode

  • Inference Engine

  • Prefix Caching

  • Paged Attention

  • AI Infrastructure Components

• Kubernetes for AI workloads: Kubernetes for DevOps + AI, Part 2

Go deeper

• The Illustrated Transformer by Jay Alammar: the best visual intro to the architecture

• The PagedAttention paper: the KV cache memory idea behind vLLM

• Engines: vLLM docs, SGLang and TensorRT-LLM

• Disaggregated serving: the DistServe paper, vLLM's disaggregated prefilling guide and NVIDIA Dynamo

• Inference on Kubernetes: vLLM Production Stack (its numbered tutorials are great) and llm-d

• Fast and Efficient LLM Inference with vLLM: a short course by DeepLearning.AI

What gets you out of this level: hands-on practice, and a lot of it. If you have access to a GPU cluster, deploy a model, experiment with different serving patterns and measure how your architecture choices affect performance.

→ Build: Project 2, Production LLM serving stack on Kubernetes. Deploy vLLM with Helm, add replicas behind a router and see KV-cache-aware routing in action.

• GitHub: vLLM Production Stack (follow its numbered tutorials) and LMCache (KV cache reuse across replicas)

Level 3: Optimizing Inference

This is the layer where you make inference faster, more efficient and more cost-effective. The goal is getting your setup to meet SLA targets for latency, throughput and memory consumption.

What you'll focus on

• Observability and benchmarking: time to first token (TTFT), time per output token (TPOT), inter-token latency (ITL), throughput and end-to-end latency

• Request-level optimizations: continuous batching, speculative decoding and chunked prefill

• Model compression: quantization and distillation

• Parallelism strategies: tensor, pipeline, data, sequence and expert

Crash courses

• AI Inference Crash Course chapters:

  • Inference Metrics

  • Batching Techniques (continuous batching and chunked prefill)

  • Speculative Decoding

  • Quantization

  • Parallelism

Go deeper

• LLM Inference Benchmarking: Fundamental Concepts by NVIDIA: what each metric actually measures

• GuideLLM: an open-source tool for load-testing and benchmarking LLM deployments

• Large Transformer Model Inference Optimization by Lilian Weng: quantization, distillation and more in one place

• How continuous batching enables 23x throughput in LLM inference by Anyscale

• Optimizing LLMs for Speed and Memory by Hugging Face

• The Ultra-Scale Playbook by Hugging Face: parallelism across GPUs, explained end to end

• NVIDIA DCGM Exporter: GPU utilization metrics for Prometheus

What makes you successful at this level: build a benchmarking project. Experiment with different configurations, measure TTFT, throughput, GPU utilization and overall latency, then optimize based on what your measurements tell you.

→ Build: a benchmark of your Project 2 stack. Point GuideLLM at it, change one setting at a time (batch size, quantized vs. full model, chunked prefill on or off) and compare the numbers.

• GitHub: GuideLLM

→ Then Project 4, GPU autoscaling inference system. Serve a model with KServe, autoscale on concurrent requests, load-test it and watch how pods and GPU nodes scale.

• GitHub: KServe autoscaling samples, hey (load testing) and Cluster Autoscaler

→ Then Project 5, Scale-to-zero inference with KEDA. The cost side of optimization: requests go into a Redis queue, vLLM workers pull from it, and KEDA scales GPU workers to zero when it's empty.

• GitHub: gpu-autoscale-inference (reference project) and the KEDA autoscaling tutorial in vLLM Production Stack

No GPU? Start anyway

Use a local cluster (minikube, kind or k3d) with small, quantized models. Projects 2–5 still teach you the architecture, just slower.

Final tip

You don't have to master all three levels at once. Build your foundations, understand what inference actually is, then learn how to optimize it. That's how you move from deploying models to actually running inference engineering systems.

📺 Watch the full walkthrough: 5 AI Infrastructure Projects You Must Build