Redacting secrets from LLM requests without breaking the prompt cache

Ryle · Oct 18, 2026

In the setup I'm running, all model traffic goes through one gateway, and the agents behind it handle real credentials. API keys show up in tool output, config files and shell history, and all of that ends up in the conversation. So I added a step to the gateway that swaps every known secret for a placeholder before the request goes out to the model provider, and swaps it back in the response. That part worked fine. The problem was it wrecked prompt caching on every conversation that touched a secret, and I ended up turning it off. Once it was disabled, cache reuse on the main model traffic went back above 90%. The fix ended up being small, but it changed how I think about any proxy that rewrites prompts.

How provider prompt caching works

Providers cache the attention state for a prompt prefix, so the next request that starts with the same tokens can skip recomputing them. These are the rules that ended up mattering for me:

• Matching is on an exact prefix. Anthropic builds the prefix in a fixed order, tools, then system, then messages, and hashes it cumulatively up to each breakpoint. One changed byte anywhere invalidates everything after it.
• On Anthropic you mark up to four breakpoints with cache_control. A write happens at a breakpoint, and a later request finds it by looking back up to 20 blocks from its own breakpoint.
• Entries expire. Anthropic's default TTL is five minutes, with a one-hour option, and every read refreshes it. OpenAI's current models keep entries for 30 minutes after the last write or reuse.
• Reads are cheap and writes cost extra. On Anthropic a read is 0.1x the base input price on most models (0.05x on the newest Opus and Sonnet), while a five-minute write is 1.25x and a one-hour write is 2x.

A long agent conversation resends its whole history every turn, so nearly all of its input should come from cache. That only works if every byte of the history is the same from one turn to the next.

How the first version broke it

My first version loaded a snapshot of the secret store, did a substring replace for each value, and generated placeholders as ${SECRET_<request counter>_<index>}, where the counter was a per-process number that incremented on every request. The response side kept a request-local map from placeholder back to value, so the client always saw the real text.

The restore step is actually what makes the bug so bad. The client never sees a placeholder, so its history has the real secret in it, and on every turn the whole history goes back through the gateway and gets redacted again. With a new counter value each time, turn 2 forwards ${SECRET_0007_3} where turn 1 had forwarded ${SECRET_0006_3}. So the provider's cached prefix ends at the first occurrence of any secret, and everything after that gets written to cache again at 1.25x on every call. If the secret is in the system prompt or an early tool result, that's most of the conversation.

Nothing errors out, and the responses are still correct. The only thing you can see is the cache-read field in the provider's usage numbers dropping toward zero while cache writes go up.

There was also a second, less obvious source of nondeterminism. The secrets came out of a hash map, and I only sorted the replacement order by length. Rust's default hash map randomizes iteration order per process, so two values of the same length could get different indices on different gateway replicas. Even with a fixed counter, a request that landed on another replica would have produced different bytes.

What a miss costs

I don't have a clean before-and-after cost figure for the time it was on, so here is some rough arithmetic at current Anthropic list prices for Opus 5.5: $4 per million input tokens, $5 for a five-minute cache write, $0.20 for a cache read. Take one agent turn with a 100k-token history and 2k new tokens.

That's about 17x on input for the same answer. Time to first token also gets worse, since the provider has to recompute the whole prefix.

The design that keeps prefixes stable

What I needed was for the same secret to map to the same placeholder bytes on every request and every replica, as long as the secret store doesn't change.

Deterministic placeholders. I changed each placeholder to be derived from the snapshot. The gateway hashes a version tag plus every eligible value, length-prefixed, into a salt, then hashes the salt with each value and takes 8 bytes of the digest as a 16-hex-character ID. Any replica that loads the same secrets produces the same placeholder for the same value, and the snapshot refreshes every 60 seconds without changing anything unless the store changed. Because the salt covers every value, hashing a guess at one secret can't confirm it.

The downside of keying on the whole store is that rotating or adding any secret changes every placeholder at once, so every conversation with a secret in it gets one cold cache fill. Secrets don't change very often, so I was fine with that. If they changed often, I'd switch to an HMAC under a dedicated key shared by the replicas, so each placeholder depends only on its own value.

A total order. Values are sorted longest first, and ties are broken by their bytes. Going longest first means a secret that contains another secret gets replaced as a whole, and the byte tiebreak gets the per-process hash order out of the output.

Replace the encoded form. Matching uses each value's JSON-string encoding. That way a password with a quote in it is found as \", and swapping it keeps the JSON valid in both directions.

Leave untouched bytes alone. When no secret occurs in a compressed request, the gateway forwards the original compressed bytes exactly. Inline base64 images get set aside before scanning, because a short secret can show up inside base64 by chance, and replacing it there just corrupts the image without protecting anything.

Fail closed on what you can't read. When redaction is on, a request with a content encoding the gateway can't decode (it handles gzip and zstd), a malformed body, or one that decompresses past 64 MB gets rejected instead of being forwarded without a check.

Hold back on streams. Streaming responses come in arbitrary chunks, and a placeholder the model echoes back can get split across two of them, so replacing per chunk would leak the placeholder to the client. I made the restore stream keep a pending buffer and only emit up to the point where no placeholder could still be in progress, which means holding back the last n − 1 bytes for the longest placeholder, plus any suffix that is a prefix of a placeholder. The test for this one feeds a redacted body through the restore stream at every chunk size from 1 byte to the full length and checks the output is byte-identical to the original.

The other tests check that the same request redacts to identical bytes twice, that a second snapshot built from the same secrets produces identical bytes (the stand-in for another replica), that changing one secret changes the placeholder of another, and that the gzip and zstd paths round-trip. There's also one that reads the gateway's source and fails if any provider forwarding path skips the redaction call.

Where it stands

The deterministic version is in the gateway code now, with those tests. The deployment config still has redaction turned off, because I want to watch how it does on cache with live traffic before I turn it back on. While writing this post I also found a gap I haven't fixed yet: when a request contains an inline image, the gateway re-serializes the whole JSON body to set the image aside, and the JSON library it uses sorts object keys. That output is deterministic for a given input, but the turn where the first image appears forwards the earlier history with a different key order than the turns before it. Anthropic's docs warn that key order inside tool-use blocks is part of the cached prefix, so that turn probably costs a cold fill. I haven't measured it. The fix would be to cut images out by byte offset instead of re-parsing.

A checklist for any gateway that rewrites prompts

The same thing can happen with context injection, prompt compression or tool filtering. This is what I check for now:

• Every inserted value is a pure function of the content plus state that all replicas share. No counters, nonces, timestamps, request IDs, or per-process random salts in the forwarded body.
• Every iteration over a map has an explicit sort with a total tiebreak.
• Serialization is byte-stable: key order preserved or canonical, no reformatting of parts you didn't change. Splice edits in instead of round-tripping through a parser when you can.
• Volatile context (time, status, per-request notes) goes after the last cache breakpoint, never before it.
• Streams are restored with a hold-back buffer and tested at every chunk size.
• A test sends the same conversation twice through two independently built instances and compares the forwarded bytes.
• Cache reads per request are monitored, with a breaker. I added one to the gateway that tracks large prompts per caller and model, counts an expensive miss as fewer than 4,096 cached tokens read, or at least 16,384 fresh tokens that are more than twice the reads, and stops forwarding after three misses in one cache lifetime.

The main thing I took from this is that with a prompt cache, "means the same thing" doesn't count for anything. A rewrite that keeps the meaning but changes one byte early in the prefix costs about as much as sending a new conversation every turn, and the usage numbers are the only place it shows up.