Give a model a tool it doesn't have by hiding the agent loop in the gateway

A model running behind a gateway I built was asked a question whose answer was three days old. The model’s weights had been frozen for months. It had no native web search, no path to anything that happened after training, and the caller on the other end expected a grounded answer in a single response, the way you expect from any chat endpoint.

The caller got it. One request in, one answer out, sources attached. What the caller never saw was that between its single call and the answer it received, the gateway had run the model through four rounds of search, read, search again: issuing queries, fetching pages, feeding the results back, and letting the model decide when it had enough. The whole agent loop lived server-side, inside one stateless call. That is the claim this essay earns. You can hand a model a tool it does not have by hiding the loop in the gateway, and the client stays a single request and a single response.

Start with how the gateway even notices that a search tool was asked for. The request arrives carrying a tools array, and providers do not agree on what a web search tool looks like inside it. So the gateway checks three ways. A direct name match catches the obvious case. A type-prefix check, startswith('web_search_'), catches most of the rest, because the providers that ship a server-side search tool name its type with that prefix and a version suffix. A combined name-and-type check resolves the entries that set one field and leave the other blank.

The prefix check is the part worth dwelling on. When a provider releases web_search_2026, a version that did not exist the day I wrote the matcher, startswith('web_search_') already matches it. No new branch. No deploy. The routing is correct for a tool that has not shipped yet, which means the gateway does not need me in the loop every time a provider bumps a version number.

Detection only tells you a search tool is present. Normalization makes it usable. Every provider’s native search tool, whatever its field names and nesting, gets converted into one internal shape I call the litellm_web_search format. Downstream, the loop reads that shape and nothing else. Adding a provider means writing one adapter that maps its dialect onto the standard format, and the loop never learns that a new provider exists.

Then the loop itself. Once the gateway sees a normalized search tool in the request, it takes over the conversation. It calls the model through messages.acreate. When the model emits a search call, the gateway runs that query against a real search backend, appends the results to the running message list, and calls the model again. It repeats until the model produces an answer without asking to search. Then it returns that answer, results folded in, as the response to the one call the client made. The client’s SDK sees an ordinary completion. It cannot tell the difference between this and a model that searched on its own, which was the point of building it this way.

Here is the part I would flag in review. The client made one call, so the client has one latency number, covering however many fetches happened inside it. It has one token count, or none, for a span that contained several model turns it never observed. It cannot tell you which four pages the model read, or that it read four and not two. The loop that made the answer trustworthy is the same loop that made the answer hard to audit.

Cost attribution per hidden turn, source provenance, the ability to cap spend partway through: all of it now sits on the far side of an interface the client was deliberately built not to cross. When someone asks why one request cost what it cost, the honest answer points at turns they hold no handle on. I have a log of those turns server-side, and I can read it, but the caller’s bill is a single line for work that was never single. The abstraction that gave a search-less model the web is the same abstraction that took the itemized receipt away from the only party paying for it.