The AI Inference Engineering Roadmap

Hi Inner Circle!

Been a while!! But I’m back with a series of technical breakdowns.

These days, I get a very common question: “I’m a Cloud or DevOps Engineer. How do I get into AI?”

The unhelpful answer is: “Learn AI.” :p

A better approach is to build on what you’re already doing and working on.

Why throw away all the fundamentals you’ve already built and completely switch to a new domain when you can use that knowledge to move into AI? Right?

This is where the intersection point ~ AI Infrastructure comes in.

And it has a whole set of components and use cases you can master, from containers and Kubernetes to inference, GPUs, autoscaling, observability, and AIOps.

We start with Inference Engineering today!

In simple english, it’s the process of serving an already-built model. You’re not training it anymore ~ the model is frozen. You’re sending requests to that model, and it generates responses.

Now, you can’t jump into inference engineering just like that. You’ll need to understand a few architectural pieces, including concepts like transformer phases, model weights, and memory management.

So here’s the roadmap, in the order it actually works.

Save this one.

Phase 1 — Foundations you can't skip

1. AI & LLM fundamentals

• A model reads text as tokens and predicts the next one, over and over

• Attention is why long prompts get expensive — every token looks at every token before it

• KV cache stores that work so it isn't redone for each new token; GQA makes the cache smaller

• Training is one big job. Inference is millions of tiny ones. Different problems entirely

Why it's first: this is the vocabulary the rest of the roadmap is written in.

2. AI compute fundamentals

• CPUs do a few things fast, GPUs do thousands of things at once — which is what this math needs

• The number that matters isn't speed, it's memory: what fits on the card, and how fast it moves

• NVLink and NVSwitch are the fast roads between GPUs. PCIe is the slow one

Why it matters: bandwidth between GPUs decides what's possible in step 8.

3. Model architecture

• Transformers come in a few shapes — the ones you'll serve are decoder-only

• Dense models use every parameter for every token

• MoE models use only a few "experts" per token, but all experts still sit in memory

Why it matters: this is how you know how many GPUs you need before touching anything.

4. Inference fundamentals

• Every request has two phases with opposite bottlenecks

• Prefill reads your whole prompt at once ~ heavy on compute

• Decode writes one token at a time ~ heavy on memory bandwidth

TTFT is how long until the first word appears. TPOT is how fast the rest arrive

Phase 2 — Making it fast

Now you know what the workload is. This phase is about making it efficient.

5. Model optimization

• Quantization stores weights in fewer bits — smaller and usually faster

• PagedAttention stops the KV cache wasting memory on fragmentation

• Prefix caching reuses work when requests share an opening — free speed if every request has the same system prompt

• Speculative decoding lets a small model guess ahead, the big model checks in one pass

• FlashAttention speeds up prefill by keeping attention off slow memory

Why it matters: each of these targets prefill or decode. Knowing which tells you if it'll help you at all.

6. Inference runtimes

• vLLM, TensorRT-LLM, SGLang, TGI — engines that implement step 5 so you don't have to

• They differ on hardware support, features, and how much you can tune

• Pin your versions. Performance changes between releases

Don't get attached to an engine. They're all solving the same two-phase problem. What transfers is knowing why a knob exists, not the flag that turns it on.

7. Model serving

• The engine runs the model. The server makes it usable

• An OpenAI-compatible API means every existing client library just works

• Streaming sends tokens as they're made — this is why TTFT is what users actually feel

• Continuous batching lets new requests join mid-flight instead of waiting their turn. Biggest throughput win there is

• Chunked prefill stops one huge prompt from stalling everyone else

Why it matters: this layer is closest to work you've already done.

8. Distributed inference

• When a model won't fit on one GPU, you split it

• Tensor parallelism splits each layer — fast, but needs NVLink-class bandwidth

• Pipeline parallelism splits by layer group — tolerates slower links, adds idle time

• Expert parallelism spreads MoE experts across GPUs

• Prefill and decode can run on separate GPU pools, sized independently

Phase 3 — Running it in production

This is where your existing skills stop being adjacent and start being directly useful.

9. Inference on Kubernetes

• Kubernetes doesn't understand GPUs on its own — the GPU Operator handles drivers and scheduling

• GPUs don't split into fractions, and two GPUs on the same node aren't the same as two across nodes

• Autoscale on KV cache pressure, not CPU. CPU will lie to you

• Dynamo coordinates distributed and split serving on top

Your Kubernetes experience carries. The scheduling assumptions underneath it don't.

10. Inference networking

• Round-robin breaks here — one request is 50 tokens, the next is 50,000

• Routing has to handle long-lived streaming connections, not short ones

• Cache-aware routing sends a request to the replica that already has its prefix

• RDMA and InfiniBand are what make multi-node serving fast enough to bother with

Load balancing you know. What's being balanced is new.

11. Inference observability

• GPU utilization can look busy while the GPU is stalled waiting on memory

• KV cache utilization is the real capacity signal — full cache means queueing, spare compute or not

• Track TTFT, TPOT, tokens/sec, and cost per request

Why it matters: CPU and memory graphs won't tell you why inference is slow. These will.

12. Production AI inference

• Assemble it: engine, server, routing, autoscaling, monitoring

• New failure modes — OOM under load, evicted nodes mid-stream, model loads measured in minutes

• Cost work is mostly keeping expensive GPUs busy

Phase 4 — Proving you can do it

13. Real-world projects and career

• Study a few open-source projects that form the foundation of the inference stack, such as vLLM, SGLang, Dynamo and a few more we discussed above.

• Also, check out this post for projects you can build to get hands-on practice!

Why cloud and DevOps engineers are well-positioned here

The framing that gets this wrong is treating AI inference as a machine learning career you have to restart into. It isn't. It's an infrastructure discipline with unfamiliar vocabulary.

Look at how much of it maps directly onto what you already do:

The mental models transfer. The vocabulary is what you're actually learning.

Final thoughts

That’s it for today! I know this was a long one.. but a required one for my inner circle.

Hope this helps you find a good starting point!

See you in the next edition.

For product teams moving at AI speed.

AI makes it easier to ship anything, even bad ideas. The hard part is knowing which ideas are worth building.

Jira Product Discovery brings your ideas, customer insights, and priorities into one place, so your team can decide what to ship and move forward with confidence.

Capture ideas, prioritize with evidence, and build living roadmaps your team can rally around—all while staying connected to delivery in Jira, so everyone can see what’s being built and why.

Better product decisions in the AI era.

Try Jira Product Discovery