5 AI Infrastructure Projects To Build In 2026

As AI models move into production, someone has to deploy them, serve them at scale, manage GPUs, and keep costs in check. The best way to learn that work is to build it. These five projects build on each other, and each one comes with open-source repos and tutorials to follow.

No GPU? Most of these still run on a laptop with a local Kubernetes cluster and small CPU models. See the last section.

1. Dockerfile Generation Using an LLM

What you build: A small Python tool that reads a project's files (requirements.txt, package.json, and so on), sends that context to a local open-weight model through Ollama, and generates a Dockerfile. If docker build fails, the error goes back to the model so it can fix the file.

Before you start: Docker (base images, layers), Python, Linux processes, and basic shell scripting.

Build it

• AI-Assisted DevOps repo (Abhishek Veeramalla). The Dockerfile generator is in Day 3: Running LLMs locally. For a stretch goal, try the Day 10 capstone: build an AI agent for a DevOps use case and deploy it to Kubernetes.

• YouTube: Day 3, Gen-AI Project for DevOps Engineers

Tools & docs

• Ollama: runs open-weight models on your own machine

• Hugging Face Models: where you find open-weight models (filter by "code" and size)

• Magnitude: an open-source local inference engine that tunes itself to your hardware and recommends models that fit your machine. Use it in step 1 to pick a model your RAM/VRAM can run.

• Docker: Multi-stage builds and Docker build best practices: what you need to know to judge whether a generated Dockerfile is any good

• Hugging Face LLM Course (free): background on how these models work

Key takeaways: how model size and quantization affect memory and speed, containerization fundamentals, and why you never trust model output blindly.

2. Production LLM Serving Stack on Kubernetes

What you build: The same idea as Project 1, but served to many users reliably. You deploy vLLM on Kubernetes with Helm, add replicas behind a router, use KV-cache-aware routing, and watch inference metrics in Grafana.

Build it

• vLLM Production Stack (official). Work through the tutorials in order:

  • 00: Install a Kubernetes environment (or the kind version)

  • 01: Minimal Helm installation

  • 17: KV-aware routing and 18: Prefix-aware routing

  • 05: Offload KV cache (LMCache)

• vLLM Router source: the API gateway that routes requests across vLLM replicas

• Observability setup (Prometheus + Grafana)

Tools & docs

• vLLM documentation

• SGLang: the other major open-source inference engine, worth knowing

• Helm docs

• Kubernetes: Schedule GPUs and NVIDIA GPU Operator

• LMCache: KV cache offloading and sharing

• Fast & Efficient LLM Inference with vLLM (DeepLearning.AI, free): covers serving and benchmarking, including TTFT and throughput

• YouTube: vLLM deployment on Kubernetes

• Inference crash course: [ADD LINK]

• Kubernetes series: Part 1 · Part 2 · Part 3

Metrics to watch: time to first token (TTFT), inter-token latency, time per output token, KV cache usage, and request queue length.

Key takeaways: how inference engines work (continuous batching, memory management), why Kubernetes fits stateful AI workloads, and the trade-off between latency and throughput, which comes up often in interviews.

3. End-to-End DevOps + AIOps Platform

What you build: A platform that monitors itself. You deploy a microservices app on Kubernetes with GitOps, add metrics, logs, and alerts, then use an LLM to summarize the alerts into incident reports and suggest root causes. Finally, you automate safe responses (restart a pod, scale a deployment) and keep a human in the loop for anything risky.

Build it

• YouTube playlist: End-to-End DevOps + AIOps Project (complete series)

• GitHub: devops-ai-playbook: 7-service app, Terraform for AWS EKS, GitHub Actions CI, Argo CD GitOps, Prometheus/Grafana, and an AIOps assistant on AWS Bedrock. It also includes intentional bugs for you to troubleshoot.

Tools & docs

• Argo CD docs and GitOps crash course

• Prometheus docs and Grafana docs

• Prometheus Alertmanager: alerts for the things that actually matter

• AIOps Fundamentals (Great Learning, free course; the certificate costs extra)

Exercise: Break things on purpose. Crash a service, fill up a disk, overload an endpoint, and see whether the system catches the problem and suggests the right next step.

Key takeaways: observability as a discipline rather than a toolset, incident-response thinking, and judgment about where AI should help with operations and where it shouldn't act on its own.

4. GPU Autoscaling Inference System

What you build: GPU-backed model serving with KServe. You deploy a model as an InferenceService, configure autoscaling on concurrent requests, then hit it with a load-testing tool and measure two levels of scaling: pods scaling up, and the cluster autoscaler adding a new GPU node when pods are stuck in Pending.

Build it

• KServe autoscaling samples: target concurrency, target QPS, GPU autoscaling, and scale-to-zero (minReplicas: 0). The sample uses an older TensorFlow model, so check the current KServe docs for up-to-date install steps and runtimes.

Tools & docs

• KServe documentation

• hey: the load generator used in the samples

• Kubernetes Cluster Autoscaler: node-level scaling

• Kubernetes: Schedule GPUs and NVIDIA GPU Operator

Key takeaways: how GPUs are exposed, requested, and allocated on Kubernetes; why pod-level and node-level scaling behave so differently for GPU workloads; and real benchmark numbers you can talk through in an interview.

5. Scale-to-Zero Inference with KEDA

What you build: A cost-optimized inference system. Requests go into a Redis queue, vLLM workers on GPU nodes pull from it, and KEDA watches the queue length. When the queue is empty, KEDA scales the workers (and GPU nodes) to zero. When a request arrives, everything scales back up.

Build it

• gpu-autoscale-inference: a full end-to-end build with a FastAPI gateway, Redis, KEDA, and vLLM on GKE with T4 Spot GPUs. It includes a local k3d setup and a Grafana dashboard. It also documents its cold-start optimizations: about 11 minutes cut to about 5.6 by moving model weights to a persistent volume and pre-caching image layers.

Tools & docs

• KEDA docs and KEDA Redis Lists scaler

• Redis docs

• vLLM Production Stack: KEDA autoscaling tutorial

The key lesson: cold start. A cold start includes provisioning a GPU node, pulling a large container image, and loading the model weights into GPU memory. Together that can take several minutes. Experiment with ways to shrink it: smaller images, cached model weights, or a warm replica during busy hours.

Key takeaways: queue-based, event-driven scaling (good for batch and async work); the cost-versus-latency trade-off (scale-to-zero is wrong for real-time chat); and cold-start optimization, which many teams are still working on.

No GPU? Here's How to Still Build These

Most of these projects can run on a laptop. Use a local Kubernetes cluster and small models that run on a CPU. Responses will be slower, but autoscaling and queue-based scaling behave the same way, so you still learn the architecture.

Project 1 already runs on a laptop by design. For the rest, swap in these alternatives:

Can't run anything? Read each project's docs and architecture. Knowing which tool is used where, and why, is the part that carries into real systems.

If you do have GPUs: build them, break them, benchmark them, and document them. That's what gives you real talking points in an interview.

Quick Links

Pick one this weekend and start.

All links are free, open-source repos, official docs, or free courses. Repos and docs change over time, so if a link has moved, search for the title.