LLM gateway

What is an LLM gateway?

An LLM gateway sits between your applications and the language models they call. Instead of every application holding its own key and calling a model endpoint directly, they all call the gateway, and the gateway decides what happens next.

If that sounds familiar, it should. It is the same idea as an API gateway, applied to a different kind of backend. What makes it worth a separate name is that the things you need to control are different. With a normal API you care about requests per second. With a language model you care about tokens, because tokens are what you are billed for and tokens are what your quota is measured in.

Microsoft describes its version, the AI gateway in Azure API Management, as an extension of the existing API gateway rather than a separate product. That framing is useful: this is not a new category of infrastructure, it is your API layer learning about a new kind of traffic.

What a gateway actually does

Token limits per consumer. This is the reason most teams put one in. Your model deployment has a quota measured in tokens per minute. If three applications share it and one of them goes into a loop, the other two stop working. A gateway lets you set a token limit or a token quota per application, per team or per subscription key, over an hour, a day, a month or a year. It can also estimate the tokens in a prompt before forwarding it, so an oversized request gets rejected at the gateway instead of eating quota at the backend.

Load balancing and circuit breaking. You can put several model endpoints behind one gateway and spread traffic across them, with round robin, weighting or priority. Priority is what you use when you have reserved capacity you want filled first and pay-as-you-go capacity behind it as overflow. A circuit breaker stops sending traffic to a backend that has become unresponsive and comes back when it recovers.

Semantic caching. A normal cache only helps when two requests are byte-identical, which almost never happens with prompts. A semantic cache compares the meaning of the new prompt to prompts it has seen before, and reuses the answer if they are close enough. On a support assistant where people ask the same thing in twenty different wordings, that cuts both cost and waiting time.

Authentication without keys everywhere. The gateway can authenticate to the model using a managed identity, so your applications never hold a model key at all. That removes a whole class of problem: keys in configuration files, keys in source control, keys that nobody dares rotate.

Content checks. You can run prompts through a content safety check at the gateway, which means the rule applies to every application at once instead of being reimplemented in each one.

One place where everything is logged. Prompts, completions and token counts can be logged centrally, with custom dimensions like the calling application or the user, so you can finally answer which team is responsible for which part of the bill.

Why the second application is the trigger

With one application calling one model, a gateway is overhead. You can set a token limit on the deployment itself and be done.

The moment there is a second application, three questions appear at once that you cannot answer without a gateway. Which of the two used up the quota this morning. Which one is responsible for the invoice. And if you have to move to a different model or a different provider next quarter, how many codebases do you have to change.

That last one is the argument that convinces most people. When your applications call the gateway rather than the model, swapping the model behind it is a configuration change. Microsoft has taken this further with a unified model API that exposes several providers through a single endpoint and translates the formats, so you can govern models from different vendors with one set of policies.

Gateways have also grown past models. The same layer now fronts MCP servers and A2A agent APIs, which means the tools your agents use and the agents themselves get the same treatment as your model traffic: authentication, throttling, logging and policy in one place.

What to watch out for with an LLM gateway

You have introduced a single point of failure. Everything AI in your organisation now goes through one component. That is exactly what you wanted for governance, and it means the gateway needs the same availability thinking as the rest of your critical path.

Semantic caching can serve the wrong answer. Two prompts that are close in meaning are not always close enough. "What is the notice period for a permanent contract" and "what is the notice period for a fixed-term contract" sit very near each other in vector space and have different answers. Set your similarity threshold carefully and keep personalised or permission-dependent answers out of the cache entirely.

Central logging is central risk. One place holding every prompt and every answer from the whole organisation is an attractive target and a data protection question. Decide what you keep and who can read it before you switch it on.

A limit that is too tight is invisible. When an application hits its token quota, users usually see a vague error rather than a message saying the budget ran out. Make sure the limit surfaces somewhere a human is watching.

It is a control point, not a policy. A gateway gives you the ability to enforce limits and rules. Which limits and which rules is a decision your organisation still has to make, and buying the component does not make that decision for you.