System Design Interview: Design an AI Customer Support Assistant | Part 1

Hi Inner Circle!

Today we're covering the first of three system design questions I think you should prepare for this year.

You've seen the small chat widget on e-commerce sites like amazon. There you type:

"Where is my order?"

And something on the other side understands the question, checks your order, and answers you.

That widget is the still a doable part. Today we're looking into the design of the system behind it.

Before anything, here’s what you SHOULD NOT DO: open your response by naming technologies. Vector database. RAG. LLM. Kubernetes.

A list of components with no flow connecting them.

A much stronger answer follows one request through the system, end to end.

The system, at a glance

• API gateway and load balancer

• Orchestration service

• Data pipeline and vector database

• LLM served on GPUs behind an inference server

• Internal APIs — orders, accounts, refunds

• Guardrails, observability, and evaluation

Now let's go layer by layer, and ask what problem each one actually solves.

Why an API gateway?

When the user hits send, the message travels through an API to our backend.

The gateway handles authentication, rate limiting, and routing. The load balancer spreads requests across our services. Neither of these is AI-specific ~ they're the same layers you'd put in front of any production service, and interviewers notice that system understanding!

Why an orchestration layer?

This is the brain of the application, and it's the piece that separates a chatbot demo from a support system.

The orchestration service reads the incoming message, works out what the customer actually wants, and decides what happens next:

• Search the knowledge base?

• Call an internal API?

• Send it straight to the LLM?

• Escalate to a human?

This can be one service or several, running as containers on Kubernetes. As traffic grows, Kubernetes scales those services. Guardrails sit around this layer too, so requests are checked before anything expensive or unsafe happens downstream.

Notice that the model hasn't appeared yet. Most of the design work in an AI application happens before the model call.

Where does the company knowledge come from?

A support system already sits on a pile of information: product docs, FAQs, past support tickets, refund policies, internal runbooks.

We need a pipeline that keeps that information available to the AI:

Docs → process → chunk → embed → vector database

And when the source changes, we re-index. That last step gets skipped constantly, and it's one of the most common ways these systems fail in production — the architecture diagram looks correct while the assistant confidently quotes a refund policy that was replaced two months ago.

Why an inference server and GPUs?

The model has to run somewhere. You either use a managed model behind an API, or host your own.

If you host it, the model runs on GPU-backed servers behind an inference server such as vLLM. As requests increase, you need more inference capacity — and that's where Kubernetes scheduling, autoscaling, GPU availability, latency targets, and cost per token stop being theory and start being your actual job.

Both options are defensible in an interview. What matters is that you can say why you picked one.

RAG or a live API?

This is the most interesting decision in the whole design, and it's where I'd spend my time if I were answering.

"What's your refund policy?" → the answer lives in a document. Retrieve the relevant chunks from the vector database and pass them to the model as context. RAG is the right tool.

"Where is my order?" → no document contains this. The answer exists only in the orders database, right now. So the orchestration service calls the Order API, gets the current status, and passes that to the model instead.

Same widget, same user, completely different retrieval path. If your answer treats RAG as the solution to everything, you'll retrieve a stale shipping FAQ for a customer asking where their package is.

Plenty of real questions need both — "where's my refund for the order that arrived late?" pulls live order state and the written policy in the same turn.

Let's follow one request

Here's what actually happens when someone types "Where is my order?":

1. The message leaves the widget and hits the API gateway. Authentication, rate limiting, routing. The load balancer picks a backend.

2. Orchestration classifies the intent: order status, needs live data, tied to this specific customer's identity.

3. It calls the Order API with a token scoped to that customer, and gets back the current status and delivery estimate.

4. It assembles the prompt ~ system instructions, the live order data, any retrieved policy text, and the conversation so far.

5. The inference server runs the forward pass on the GPU and streams tokens back as they're generated.

6. A guardrail check runs on the way out, the response streams into the widget, and the whole interaction is traced: latency, token count, cost, whether retrieval fired, and how the customer sounds.

Six hops. One model call. Everything else is orchestration.

Why guardrails and observability aren't the last slide

Throughout that flow, we need to know what's happening — and the questions are different from the ones you'd ask about a normal service:

• Did we read the customer's intent correctly?

• Did we retrieve the right information?

• Was the answer actually correct?

• How long did it take, and what did it cost?

• How does the customer sound?

If someone is clearly frustrated, uses abusive language, or the assistant can't resolve the issue after a couple of turns, guardrails trigger a handoff to a human.

We're not just monitoring whether the servers are healthy. We're monitoring whether the AI is doing its job.

Interview takeaways

Things worth being able to explain, beyond drawing the boxes:

• Streaming changes the connection model. Tokens arrive one at a time, so you’re dealing with an open connection like SSE or WebSockets. That affects load balancing, autoscaling, and what happens to an in-progress response when a pod goes down.

• Session state has to live somewhere. Conversation history becomes part of the prompt. You need to decide where to store it, how long to keep it, and what to do when it gets too large for the context window.

• The assistant should only access what the customer can access. Every internal API call needs the right permissions and scope. Otherwise, you’ve basically created a data breach with a chat interface.

• Cost is per conversation, not just per server. Think tokens, retrieval calls, and GPU time. That’s what determines whether the system is actually affordable to run.

• Human handoff is part of the design. When the AI can’t solve something, the conversation should move to a human with the right context — not start from scratch.

• The pieces change at different times. Your application, prompts, knowledge base, and model can all change independently. Each can affect answer quality, so you need a way to test and monitor those changes

Final Thoughts

This isn’t a fixed architecture. Different companies use different approaches, and the components can be swapped based on the requirements.

For example, the orchestration layer could be an agent harness, or you could use a different approach to define how the model behaves.

The important part is understanding the role each component plays and how the pieces work together — so you can swap components as you see fit.

See you in the next edition!

Agents are already at work. Do you know where?

Cowork, Work, Muse, Grokbot: people are bringing computer-use AI to the office on their own. Harmonic’s Usage Explorer shows who's using what, for how long, and on which tasks, so you can understand the usage first and apply controls that fit.

See Your Workforce’s AI Usage

o