AI Infrastructure Roadmap: From Linux to LLM Inference

Everyone wants to "do AI." Far fewer people can keep a model running at 3 a.m. when the GPU node drops off the cluster. That second group, the engineers who build, ship and scale AI systems, is where the real demand is.

This roadmap covers 12 skills in the order they matter, each with:

• Crash course: get the mental model in a few hourS

• Deep dive: free material to actually get good

• Build it: one small project to prove you learned it

Everything linked here is free.

Phase 1: Foundations

1. Linux Systems

Every model you'll ever serve runs on Linux. Processes, permissions, systemd and logs are non-negotiable.

• The Missing Semester of Your CS Education (MIT): start with the shell, scripting and command-line lectures

• Linux Journey: free, beginner-friendly, end to end

• Linux Upskill Challenge: a month of daily server admin tasks

• OverTheWire: Bandit: learn the terminal as a game

• Spin up a VM, create a user, harden SSH, and run a service with systemd

2. Networking Basics

Half of all "the model is broken" bugs are actually DNS, ports or TLS.

• Cloudflare Learning Center: short, clear explainers on DNS, TCP/IP, HTTP and load balancing

• freeCodeCamp: Computer Networking Course: ~9 hours, Network+ level

• Beej's Guide to Network Programming: sockets, the way they actually work

• Use dig, curl -v, ss and tcpdump to trace one request from your laptop to a website

3. Python Scripting

The glue language of AI. You need it for automation, data wrangling and every ML framework.

• The Official Python Tutorial: a fast, no-fluff tour

• Automate the Boring Stuff with Python: free online book, very practical

• CS50's Introduction to Programming with Python: Harvard, free, with problem sets

• Write a script that checks disk, CPU and memory on a server and posts an alert when a threshold is crossed

Phase 2: Containers & Orchestration

4. Docker Containers

"Works on my machine" doesn't work in production. Containers are how models get shipped.

• Docker: Get Started: the official guide, a couple of hours

• TechWorld with Nana: her Docker crash course is the classic beginner video

• Play with Docker Labs: a free browser-based Docker playground

• Containerize a small FastAPI app, then shrink the image with a multi-stage build

5. Kubernetes Orchestration

The standard way to run many containers (and GPUs) reliably.

• Kubernetes Crash Course for Absolute Beginners (TechWorld with Nana)

• Kubernetes Basics (official tutorial)

• Killercoda: free interactive Kubernetes scenarios in the browser

• Kubernetes the Hard Way: build a cluster by hand once and you'll never fear it again

• Deploy your Dockerized app on a local cluster (kind or minikube) with a Deployment, a Service and a health check

Phase 3: Models & Pipelines

6. LLM Fundamentals

You can't serve or optimize what you don't understand: tokens, attention, context windows, sampling.

• Andrej Karpathy: Intro to Large Language Models (1 hr)

• 3Blue1Brown: Neural Networks series: best visual intuition for transformers

• Karpathy: Neural Networks, Zero to Hero: build GPT from scratch

• Hugging Face LLM Course: transformers, tokenizers, fine-tuning

• The Illustrated Transformer (Jay Alammar)

• Load a small open model, inspect its tokenizer, and generate text with different temperatures

7. ML Pipelines

Models are the easy part. Data, training, versioning and retraining are the real system.

• Google Machine Learning Crash Course

• Made With ML: design, develop, deploy and iterate on production ML

• MLOps Zoomcamp (DataTalksClub): experiment tracking, orchestration, monitoring

• Build a pipeline that ingests data, trains a model, logs metrics, and registers the best version

Phase 4: Hardware

8. GPU Architecture

GPUs are the most expensive line on the bill. Knowing SMs, memory bandwidth and HBM tells you why things are slow.

• Modal GPU Glossary: every GPU term explained in plain language

• GPU MODE Lectures: free community lecture series, from CUDA basics to Flash Attention

• AI Infrastructure & Operations Course by Nvidia

• How to Scale Your Model (Google DeepMind): accelerators, rooflines and parallelism, explained properly

• NVIDIA CUDA C++ Programming Guide

• 🛠️ Run nvidia-smi while a model generates text and explain what each number means

Phase 5: Serving

9. LLM Deployment

Turning weights into a reliable, scalable API.

• Ollama: run an open model locally in under 10 minutes

• vLLM Documentation: the go-to open-source serving engine

• Full Stack Deep Learning: LLM Bootcamp: free recorded lectures on shipping LLM apps

• Serve an open model with vLLM behind an OpenAI-compatible endpoint, on Kubernetes if you're brave

10. Inference Optimization

Same hardware, 2–10x more throughput. This is where AI infra engineers earn their salary.

• Lilian Weng: Large Transformer Model Inference Optimization

• Hugging Face: Optimizing LLMs for Speed and Memory: quantization, Flash Attention, KV cache

• PagedAttention paper (the idea behind vLLM)

• GPU MODE Lectures: see the lectures on quantization and Flash Attention

• TO-DO: Benchmark one model at FP16 vs. 4-bit quantized, and compare latency, throughput and output quality

Phase 6: Production at Scale

11. AI Networking

At cluster scale, the network is the computer. GPUs spend their time waiting on each other unless the fabric (NVLink, InfiniBand, RoCE) and the collective ops are right.

• GPU MODE Lecture 17: GPU Collective Communication (NCCL)

• The Ultra-Scale Playbook (Hugging Face): training LLMs on GPU clusters, including all the communication patterns

• NVIDIA NCCL User Guide

• Run a multi-GPU job (even 2 GPUs on a cloud box) and measure all-reduce bandwidth with nccl-tests

12. AI Observability

If you can't see tokens, latency, cost and GPU health, you're flying blind.

• OpenTelemetry for Generative AI: the short intro

• OpenTelemetry GenAI Semantic Conventions: the emerging standard for LLM telemetry

• Langfuse Docs: open-source LLM tracing and evals

• NVIDIA DCGM Exporter + Prometheus: GPU metrics into dashboards

• Build a Grafana dashboard showing GPU utilization, tokens/sec, p95 latency and cost per 1K tokens

Short on Time? The Crash-Course Sprint

Do just the items, in order, roughly one per evening:

That gives you the vocabulary and mental map in about two weeks. Then come back for the deep dives, one phase at a time.

Final Tip

Don't collect courses; ship projects. The 🛠️ items stack: by the end you'll have a containerized, Kubernetes-hosted, optimized and fully observable LLM service. That one project says more on a résumé than twelve certificates.

Need more project ideas? In this video 5 AI Infrastructure Projects You Must Build, I walk through five projects in the order I'd build them. I also cover the interview topics they bring up, like KV cache, inference metrics, cold starts and cost trade-offs, and show how to build most of them on a laptop, no GPU needed.