Giving coding agents credentials without giving them the secrets
Ryle ·
My coding agents push branches, open and merge pull requests, call paid APIs, read databases and deploy, and every one of those needs a credential. The easy way to hand one over is the way you'd do it for yourself: a token in an environment variable, a dotfile or a git credential store. For a while that's roughly what I did.
The trouble is that an agent isn't a person with a terminal. Everything it sees goes into a transcript, and that transcript gets sent to a model provider, stored, searched, summarized into memories and sometimes exported as training data. A secret an agent can read ends up in places you don't control. Over a few months I moved most of my agents to a setup where each one holds a single secret, its own identity, and everything else is used on its behalf by something that holds the real credential. This post goes through each technique I tried, what it cost and whether it stuck.
What goes wrong when agents hold secrets
Here's what I ran into. None of the incidents involved an agent trying to do anything bad.
- The environment ends up in the transcript. A model provider token sat in the agent pods' environment, and one day it showed up in a transcript. While I was investigating that, a debug
echoprinted it again. - Probing prints prefixes. A red-team agent I ran against my own setup wanted to know which secret names existed. It ran the "get a secret" command piped through
head -c 12, which put the first 12 characters of a real key in its transcript. - "Masked" isn't masked. My LLM gateway labelled rate-limit metrics with the first 16 characters of the provider key, to tell keys apart in dashboards. Past the vendor's fixed prefix, that was about three bytes of real secret. The label was also a database primary key, so when a reindex failed on a duplicate, the error message carried it into the cluster's log store.
- Agents go hunting. If a secret isn't where an agent expects it, the agent looks for it: shell history, dotfiles, other processes' environments, or it asks me to paste the value into chat. One agent saw a literal
${NAME}placeholder in some output, decided the vault was broken, and created a new vault item to "fix" it. - A token's blast radius is the token, not the task. A token that lets an agent comment on its own pull request can comment on every pull request in the repo. A write token for one branch is a write token for all of them.
Anything an agent can read, it will eventually print. So the goal became leaving it nothing to print.
Sort secrets into tiers
The most useful early step was writing down where each kind of secret should end up. This list from the planning note held up:
| Kind of secret | Where it lives | How an agent uses it |
|---|---|---|
| LLM provider keys | The LLM gateway | Calls the gateway as itself; the gateway adds the key |
| Outbound HTTP API keys (search, data feeds, SMS) | An egress proxy | Names the upstream; the proxy adds the key |
| Infrastructure (git forge, cluster, SSH) | Brokers | Asks a broker to act for it |
| Whatever must be local (client-side signing, a local test database) | The vault | Injected into one child process |
The last row is the only one where an agent's process ever holds a value, and the note called it "accepted as weakest tier". Most of the work since then has been moving things up the table.
One vault, and names instead of values
Every secret lives in one self-hosted Vaultwarden vault. That's the only allowed source. The agents' instructions say not to read secrets from environment variables, dotfiles, shell history, process memory or values pasted into the conversation, and if a name isn't in the vault, to report that and stop. An agent that's told to stop doesn't go through ~/.bash_history.
Agents can list secret names but not values, and there's a separate "exists" command that answers present or absent from the name list alone, without fetching anything. That came straight out of the red-team transcript: the agent only wanted to know whether a name existed, and its only tool fetched the value.
Getting agent pods into the vault took two tries. First, a scheduled job kept a pool of non-admin vault accounts topped up to match the number of agent pods, and each pod copied its account in at boot. The first version copied into a read-only mount and crash-looped the agent container. Later I added a vault broker: a pod presents its own API key, and a server-side broker that holds the vault passwords decrypts for it. Vault logins are slow on purpose (600,000 rounds of PBKDF2), so the broker keeps the decrypted items in memory for five minutes. It refuses writes.
Inject into one child process
For the bottom tier, my agent CLI has a run command that puts a secret into the environment of exactly one child process:
cli run --with-secret DEPLOY_TOKEN=API_TOKEN -- ./deploy.shThe CLI fetches DEPLOY_TOKEN from the vault and starts ./deploy.sh with API_TOKEN set. The agent's shell never has it, and the command line only carries names. There's a variant that writes the value to a file readable only by the owner in the user's runtime directory and deletes it afterwards, for tools that want a file.
The child is the agent's tool, though, and tools print things. So when any secret is injected, the CLI pipes the child's stdout and stderr through a scrubber that replaces each value with ${NAME}. It holds back a few bytes between reads so a value split across two reads still gets caught, and it also matches the forms a value takes in diagnostics: JSON-escaped, a Rust debug string, a Python bytes repr. Values shorter than four characters make the command refuse to run, because scrubbing abc out of all output mangles it, and silently not scrubbing it would be a leak.
The scrubber also broke something. A pipe isn't a terminal, and interactive CLIs check for one. A coding agent CLI I start through run saw a pipe, switched to its non-interactive mode, found no prompt and exited. Six of my launch shortcuts stopped working. The fix was to skip the pipe when a terminal is attached, on the reasoning that a human watching a terminal isn't a log. That was wrong, because agent harnesses run commands in pseudo-terminals too. For a long stretch, output from run inside an agent session wasn't scrubbed. The CLI now checks for the environment markers that agent harnesses set and scrubs whenever it finds one, terminal or not.
That's also where the ${NAME} confusion came from: the placeholder means the scrubber worked, which the instructions now say. And the scrubber has a hard limit. The child can read its own environment and print the value in an encoding the scrubber doesn't know, like base64. That's why this tier is the weakest.
Keep tokens out of argv and clone URLs
A URL like https://user:token@forge/repo.git is the classic way to give git a token, and it's bad in two ways. Anyone who runs ps on the machine sees it, and git saves it into the clone's .git/config, where it stays. It caused one visible bug: an automated fixer's push went out as the agent's own forge user instead of the bot account it was supposed to use, because its throwaway clone had the agent's token baked into the remote URL, and anything that ran git push in that checkout used it.
The replacement is git's --config-env, which takes a config value from an environment variable. So argv only names the variable:
AUTH_HEADER="Authorization: token $FORGE_TOKEN" \
git --config-env=http.https://forge.example/.extraHeader=AUTH_HEADER \
push origin my-branchA later sweep found URL tokens or insteadOf rewrites in CI, the merge queue, the fixer, pod init scripts and a provisioning service. Other tools got equivalents: docker login --password-stdin, and an owner-only curl config file instead of -u user:pass. The CLI that routes git through the forge broker (next section) also points git's global config at /dev/null, so a stray insteadOf rule on the machine can't quietly send a push somewhere else with a different credential.
Brokers instead of forge tokens
The forge was the biggest source of tokens. At the start each agent had its own forge user and access token, test harnesses had a pool of extra users, each repo needed collaborator grants for each user, and merges used an admin token. The note that removed the review token from agent pods says what was wrong with even the small ones: a pod-resident token that could comment could post on any PR in the repo.
So I replaced them with brokers, one operation at a time:
- Comments. The agent sends its comment and its own API key to a gateway route, which posts as a bot account.
- Pull requests. Same idea. Every PR is then authored by the bot, which loses track of which agent did the work, so the broker refuses to open a PR unless the body names the agent and the machine it ran on.
- Push. The agent uploads a git bundle of one branch. The broker checks that the bundle's tip is the commit the agent said it was pushing and refuses
main,master,devand release branches. Bundles were capped at 24 MB, which was fine for small repos and not for the big one, so this started on test-fixture repos. - A git proxy. Eventually the broker became a proxy that speaks git's smart HTTP protocol. It reads the ref updates in each push and refuses protected branches, release branches, tags and anything that isn't a branch. REST writes are limited to opening and editing PRs, comments and commit statuses. Forced merges are limited to two agent roles and an explicit list of principals, and an empty list means nobody.
Which repos a key could reach was first a global allowlist in the broker's environment. That kept returning 403s whenever someone created a repo and forgot to add it. Now grants are rows per API key: a new key has no access, revocation is checked on every request, and if the database is unreachable the broker refuses. Switching that on meant writing explicit grants for every existing key first, because there was deliberately no fallback to the old allowlist.
The broker had its own costs. It buffered whole git packs in memory, and a big push ran the gateway out of memory until I made it stream them. There was a stretch of 502s and 404s through the proxy. It stuck anyway, and it's the change I'd keep if I could keep only one.
Inject at the edge
The top two tiers work like the forge broker: the agent authenticates as itself, and something at the edge adds the real credential. That was the fix for the transcript leak: instead of a provider token in their environment, agent pods authenticate with their Kubernetes service account token, which the gateway checks with a TokenReview before adding the provider token itself. Outbound API calls go through an egress proxy: the agent names the upstream service in the request path, and the proxy forwards a filtered set of headers plus the credential.
The hard part is how the edge knows who's calling. My first idea was the pod's source IP, so agents would hold no secret at all. I tested it before relying on it, and it didn't hold up. My network setup didn't stop a pod from spoofing its source IP, the agent pods still had the capability to send raw packets, and the IP-to-pod mapping was racy when IPs got reused. Source IP is fine for attributing traffic, but it's no boundary against a pod that misbehaves. So the IP check became advisory and a per-agent bearer key became the real check. I added Ed25519 request signing on top, which went unused once each agent got its own pod. For the most privileged credentials, the proxy now also checks that the key belongs to a pod that's actually running and to the deployment that owns it.
The edge also has to watch what it sends upstream. A security audit found a route to a third-party API that copied request headers by hand and forwarded internal ones, including one carrying an internal key. Outbound headers are an allowlist now.
Scoped identities for scheduled bots
Some of my agents run on a schedule: research bots that each add findings to one project's repo and aren't supposed to touch anything else. At first each one got its own forge user, created by hand through the forge's admin API, plus database rows inserted with raw SQL.
Now a bot gets a broker key with a scoped grant: a branch prefix, a path prefix and the account it acts as. Pushes and PR branches have to sit under the prefix. Before merging, the broker pages through every changed file, including the old path of a rename, and refuses if any of them is outside the path prefix. The PR head has to be the exact commit requested, and base and head have to be in the same repo. The project repos add branch protection and their own delivery script on top. The instructions now say in so many words to never create per-bot forge users.
CI and Kubernetes
CI used to get secrets from a service that copied about 164 vault items into two repos' Actions secrets every hour. Workflows actually used about 50, and each repo got all of them. The replacement uses the CI runner's OIDC token: a job asks for a token, a small broker checks its issuer, audience and repo, and returns only the secrets mapped to that identity, which the job masks in its logs. My runners didn't issue OIDC tokens at first, so workflows needed fallbacks, and the migration is still partial.
In the cluster, the generated manifests only reference Kubernetes Secrets by name. External Secrets Operator fills a few hundred of them from the vault through a small bridge service. The lesson there was that the sync path is a vault too. The bridge would return any decrypted item to any pod in the cluster that asked, with no authentication. An audit found it, and now it requires a bearer token minted at startup, fails closed without one and doesn't list items by default.
Everything that copies transcripts
The gateway also redacts known secret values from requests before they reach a model provider. That broke prompt caching badly the first time, and I wrote that up in the gateway injection post, so I won't repeat it here.
Redaction at one point isn't enough, though, because transcripts get copied. Fine-tuning exports, long-term memory extraction, retained review output and CI logs are each a new place a secret can land, and each needed its own secret screen. The browser login helper is built the same way: the password goes into its child process from the vault, and the logged-in session goes to a file whose path is handed to the next command, so its logs show only the user, the URL and the path. For some other tools, "never print cookies or tokens" is still only a rule in the instructions.
What I'd do from day one
- Give each agent exactly one secret: its own identity. Tie it to the workload (a service account, or a key checked against the running pod), not to a network address.
- Broker every write. Push, PR, comment and merge go through something that holds the real token and checks the request. Grants belong to keys, start empty and fail closed.
- Make the vault answer "does it exist" without fetching. Let agents list names. Tell them to stop when a name is missing, instead of letting them hunt.
- Inject into one child process only for what has to be local, and scrub its output whenever an agent might be reading it. Don't treat a terminal as proof that a human is the only reader.
- Never put a token in argv or a URL. Use
--config-env, stdin and config files readable only by their owner. - Treat every copy of a transcript as a log that leaves your control: exports, memories, metrics labels, error messages. Hash what you need to tell credentials apart instead of truncating them.
- Authenticate the plumbing. The service that syncs secrets into the cluster and the routes that proxy to third parties need the same care as the vault.
None of this stops a determined agent that can run arbitrary code with a secret in its environment. What it does is make the honest mistakes, which are all I've actually seen, mostly print ${NAME}.