Keeping an LLM gateway from causing its own 429s

Ryle · Oct 24, 2026

My coding agents send their model traffic through a gateway that sits in front of the Anthropic API (and a few others). At one point I was helping debug a different gateway, someone else's implementation, that got HTTP 429 (Too Many Requests) errors all the time, and I wrote down the things I thought a gateway should and shouldn't do to avoid causing them. This post is that list, cleaned up, plus what I learned when I went back and checked each item against my code and the provider's current documentation. One of the items turned out to be wrong.

The short version: a 429 is the provider deciding your organization or key has used too much. A gateway can't make the limit bigger, but it can easily make the provider count more load than your clients actually generate. Nearly every cause I found is one of three things:

The gateway sends more requests than its clients did, usually by retrying.
The gateway makes the same request count for more, usually by breaking prompt caching.
The gateway sends the same load in a burstier shape.

This post is about the plain forwarding path, where the gateway passes each request through with the caller's own key. Some of my traffic goes through other routes with their own handling, which I'm not covering here. The prompt cache side has its own post, so here it only comes up where it touches rate limits.

What the provider actually counts

Some vocabulary first, since the limits are named by acronym:

RPM is requests per minute.
ITPM is input tokens per minute, and OTPM is output tokens per minute. Many providers have a single TPM (tokens per minute) limit instead.
A token bucket is how the limit is enforced. Your allowance refills continuously up to the maximum, instead of resetting at the top of each minute, so a burst can empty it even when your per-minute average is fine.

Anthropic's rate limit docs say a few things that matter for a gateway:

Limits are set per organization and per model class, and each class has its own RPM, ITPM and OTPM.
A limit of 60 RPM might be enforced as 1 request per second, so short bursts can trip it.
For most current models, only uncached input counts toward ITPM: the tokens after the last cache breakpoint plus the tokens written to cache. Tokens read from cache don't count. Their example is a 2,000,000 ITPM limit with an 80% cache hit rate, which lets you process about 10 million input tokens a minute.
OTPM counts the output tokens actually generated, as they're produced.
A sharp increase in usage can hit separate acceleration limits, and they recommend ramping traffic up gradually.
Not every 429 is a rate limit. When an organization reaches its monthly spend cap, requests also come back as 429 with error type rate_limit_error, but with no retry-after header, and retrying doesn't help until the cap resets. The error body has error_code: enforced_spend_limit_reached.

So the question for a gateway is never "how do I avoid 429s" in the abstract. It's "what am I doing that makes the provider count more requests, more uncached input, or a burstier shape than my clients sent".

Don't retry in the gateway

This is the biggest one. When the provider returns a 429 on my forwarding path, the gateway forwards it to the caller unchanged, retry-after header included, and the caller's API client backs off. The gateway doesn't retry it. It doesn't retry a 5xx either, and if the upstream connection fails before any response arrives, the caller gets a 502 instead of a second attempt.

The reason is that retries multiply. Anthropic's official SDKs retry 429s, 5xx errors and connection errors twice by default, so one logical call can already be three requests. If the gateway also retries twice, every one of those attempts becomes three upstream requests, and one call that keeps failing is nine requests against a limit that already said no. Every extra one counts. OpenAI's rate limit docs say it directly: "unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work." A gateway that retries its own 429s keeps the bucket empty and turns a short limit into a long one.

Top: three client attempts, each retried twice by the gateway, make nine upstream requests. Bottom: only the client retries, so three attempts make three requests, each waiting out the retry-after. — Two retry layers with two retries each turn one failing call into nine upstream requests.

The rule I ended up with is that the whole chain has exactly one backoff owner. Usually that's the client, because it already has a retry loop and it can see retry-after. The gateway's job is to deliver that header intact.

5xx errors get the same treatment for a different reason. When the provider is having a brownout, retrying in the gateway adds load at the moment it's least able to take it. My gateway forwards the 5xx and counts it by status code, and the client decides.

There is one deliberate exception, and it follows the same rule. One other provider I use answers short concurrency limits and capacity sheds with a 429 or a 5xx that clears within seconds. On that route the gateway is the backoff owner instead of the client: it holds the request and resends it on the same key, waiting 1 second, then doubling up to 16, plus up to half a second of jitter so a group of held requests doesn't resend in lockstep. It stops after 300 seconds by default (configurable, capped at 900), retries a 5xx at most four times since a 5xx might be specific to that request, and passes a real quota exhaustion straight through. Only after the deadline does the caller get a 429 to back off from, so in practice there's still one retry loop at a time.

One more place retries hide is your HTTP library. Recent 0.12 versions of reqwest, the Rust HTTP client I use, retry by default when an HTTP/2 server sends a graceful shutdown (GOAWAY with no error) or refuses a stream (REFUSED_STREAM), up to two extra attempts per request. Both of those mean the server didn't process the request, so the retries are safe, but it's worth knowing your client does it. On the path that sends Claude requests I turned it off along with redirects, so each request the gateway forwards is exactly one upstream send.

Don't make the same request count for more

Keep the request byte for byte

The forwarding path passes the client's body and headers through essentially untouched, so the cache_control breakpoints and beta headers the client set up keep working. That matters for rate limits as much as for cost. Since cache reads don't count toward ITPM on most models, a long agent conversation that reads 95% of its input from cache only counts the other 5%, plus whatever it writes to cache. With caching broken, the same conversation counts all of it. A gateway change that breaks caching, like re-serializing JSON in a different key order, stripping a beta header, or moving a conversation to a different account, raises your bill and moves you toward ITPM 429s at the same time. The cache post covers how that happens and how to catch it.

max_tokens, and what I had wrong

My original note said Anthropic charges the requested max_tokens against the per-minute token limit when the request arrives, so a call with max_tokens: 64000 that produced 200 tokens would still take a big bite out of the bucket. I wrote that a gateway that rewrites max_tokens upward could push you into 429s at a fraction of real throughput.

That's not what Anthropic's docs say now. They say OTPM counts only the tokens actually generated and that max_tokens doesn't factor into it, "so there is no rate limit downside to setting a higher max_tokens value". ITPM is estimated when the request starts and corrected as it runs. I don't know whether this changed or I had it wrong from the start.

It's still true elsewhere. OpenAI's docs say your rate limit "is calculated as the maximum of max_tokens and the estimated number of tokens based on the character count of your request", and suggest setting max_tokens close to the response size you expect. So on an OpenAI-style TPM limit, a gateway that inflates max_tokens, or a client that always asks for the model's maximum, really does use up the bucket faster. My gateway doesn't raise max_tokens anywhere. The only place it touches it is one free-usage path that caps it.

Reuse connections, and know when not to

The gateway builds its HTTP client once at startup and shares it across routes, instead of making a new one per request. The standard settings:

Connect timeout of 5 seconds, and no overall request timeout, because a streamed response can legitimately run for minutes.
TCP keepalive every 30 seconds.
HTTP/2 keepalive PINGs every 20 seconds, sent even when the connection is idle, with a 20-second timeout. Without these, a stream that pauses for a long stretch of model thinking can be reset by something in between, and the client sees "socket connection was closed unexpectedly".
Idle pooled connections are closed after 30 seconds. A comment in my code says servers behind a CDN typically drop idle connections after 60 to 100 seconds. I haven't measured that for any particular provider, but the point is to close idle sockets before the other side does, because reusing a socket the far side has already half-closed fails the next request as soon as it writes, which shows up as "error decoding response body".

Most of this prevents connection errors rather than 429s directly. It touches rate limits in two ways. A client per request means a new TLS handshake and a new connection for every call, and under load that's a lot of connections, which some edges throttle. And a request that fails on a stale socket gets retried somewhere, and that retry is the thing that trips the limit. Once retries are involved, a connection bug can look exactly like a rate-limit bug.

Later I changed this for the path that sends Claude requests. Two requests on the same replica failed with connection resets 1 and 3 milliseconds into the send, before any response headers arrived. A reset like that is ambiguous: the provider may or may not have accepted the prompt, and if it did, resending it means paying for and counting that input twice. A shared HTTP/2 connection also means one bad connection can fail several unrelated conversations at once. So that path now opens a fresh HTTP/1.1 connection for every request, with no idle pool, no automatic retries and no redirects, and a reset is reported instead of replayed. That costs a TLS handshake per request. I never found out which hop sent those resets, and I can't prove each one was on a reused connection, so this is a mitigation, not a diagnosis.

Request path: the agent's client, the only layer that backs off; the gateway, which forwards bytes unchanged, sends once, reuses a client and records rate-limit headers; the provider's token buckets; and what the client gets back. — The forwarding path: the client owns backoff, and the gateway sends each request once and passes the answer back unchanged. The Claude path now opens a fresh connection per request instead of reusing one.

Read the rate-limit headers

Every response from Anthropic, successful or not, carries headers describing the limits. For a long time my gateway threw them away. Now it parses them and exports them as metrics:

anthropic-ratelimit-requests-limit, -remaining and -reset, for RPM
anthropic-ratelimit-tokens-limit, -remaining and -reset
anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining
retry-after on a 429

For providers that send OpenAI-style headers instead, it falls back to x-ratelimit-limit-requests, x-ratelimit-remaining-tokens and so on.

Parsing the reset times was fiddlier than I expected, because providers format them differently. Anthropic's are RFC 3339 timestamps. OpenAI's are durations like 1s or 6m0s. Some headers are bare numbers, which can be either seconds from now or an absolute Unix timestamp. My parser handles the timestamp, ms and s suffixes and bare numbers (anything above a billion is treated as a Unix timestamp), and clamps the result at zero.

When a 429 shows up, the first thing to look at is which remaining counter hit zero, because each points at a different cause:

At zero	Likely cause
Requests remaining	Too many calls: fan-out, a burst, or a retry layer you didn't know about
Input tokens remaining	Too much uncached input; check the share of input read from cache
Output tokens remaining	Real output volume on Anthropic; possibly `max_tokens` on an OpenAI-style limit
Nothing, and no retry-after	Possibly not a rate limit at all; read the error body (a spend cap, for example)

The token remaining values are rounded to the nearest thousand, per the docs, so don't read too much into small changes.

Concurrency shape

There's no in-flight limit on my forwarding path. It stays under the request limits because it doesn't add requests, not because anything caps them. The things I watch for instead:

Fan-out. One logical operation that becomes many parallel sub-requests multiplies request rate, and token buckets punish the burst even when the average is fine.
Everything on the same tick. Agents or scheduled jobs that all start at the top of the minute or hour hit the per-second enforcement together. Spreading start times out is cheaper than any amount of retry logic.
Ramping up. Going from idle to a full fleet at once is the "sharp increase in usage" that the acceleration limits are for.

If I ever need a hard ceiling, I'd add an explicit concurrency limit with a small queue in front of it. Holding a request for a moment in the gateway is better than letting it hit the provider, bounce off a 429 and come back.

When the provider is down

The last piece was writing down what the gateway does when the provider is degraded, so I'd know the failure modes from the code instead of finding out during an outage:

Upstream	Gateway	Caller sees
429 with headers	Forwards it, records the headers	The provider's 429 and `retry-after`
500, 502, 503	Forwards it, counts it by status	The provider's response
Connect timeout, reset, DNS failure	No retry, records the failure kind	A 502 with a `proxy_error`

The failure kinds it records include connection reset, connection timeout, connection refused, HTTP/2 reset, HTTP/2 GOAWAY and DNS, which is usually enough to tell a provider problem from a network problem at a glance.

The case to avoid is a caller that hangs or gets a misleading error. A 429 with an accurate retry-after lets the client wait the right amount of time. A 403 or a generic error can make a coding agent decide its credentials are bad and stop.

What's still open

My reset parser doesn't handle compound durations like OpenAI's 6m0s. For the one provider where I read OpenAI-style reset headers, a reset like that gets recorded as unknown.
I don't have a clean long-run 429 rate to quote. The counters reset whenever a replica restarts, and I haven't written the query that stitches them together.
The fresh-connection change trades handshake latency for fewer ambiguous failures, and I haven't measured what it costs per request.

A checklist

Have exactly one backoff owner in the chain. If it's the client, forward 429s with retry-after intact and don't retry in the gateway.
Don't retry 5xx in the gateway either; during a brownout it only adds load.
Never replay a request after an ambiguous failure, where the provider may already have accepted it.
Check whether your HTTP library retries for you.
Forward request bodies and headers unchanged. On Anthropic, cache reads don't count toward ITPM, so broken caching costs rate limit as well as money.
Don't inflate max_tokens. On OpenAI-style limits it counts against TPM up front.
Build one long-lived HTTP client, close idle connections before the far side does, and send keepalives on long streams.
Export the rate-limit headers and look at which remaining counter hit zero before changing anything.
Avoid fan-out bursts and synchronized start times, and ramp up gradually.
Treat a 429 without retry-after as possibly something other than a rate limit, and read the error body.

Most of this is about the gateway not creating load. The provider counts what it receives, and a gateway that retries, rewrites or reconnects too eagerly sends more than its clients asked for.