Running code I didn't write: WASM, gVisor, Firecracker and the machines they need

Ryle · Jun 13, 2026

A couple of my side projects run code I didn't write. One is a dataflow system where a pipeline stage can be a user-defined function, and the person (or, more often, the coding agent) who wrote that function is not someone I want running things inside my processes. Another runs coding agents for people other than me, which means arbitrary shell commands, package installs and browsers. Then there are builds of other people's repositories, which run whatever their build scripts say.

I ended up with several tiers, each a different trade between trust, isolation and startup cost. The tier I'm willing to give a piece of code decided which machines I need. The short version is that WASM and gVisor run anywhere, and the hardware-virtualized tiers need /dev/kvm, which my cloud VMs don't have. So dedicated servers, which I'd already bought for builds, also became the only place the strongest sandbox can run.

The tiers

Native operators. Filters, joins, projections and so on, compiled into the executor. I wrote them, they get reviewed and tested like the rest of the code, and they cost nothing extra to run.

Trusted dynamic libraries. A compiled transform loaded with dlopen into the executor process. The loader checks an ABI version and catches panics at the boundary, but that's about robustness, not isolation: once a library is loaded, it can do anything the executor can. So the only real control is who's allowed to register one. The default policy authorizes nobody; registration takes an internal route presenting a trusted dispatch token, or an owner I've explicitly allowed. The library's bytes are pinned by hash when it's registered, so swapping the file afterwards doesn't work. Nothing checks a signature at load time.

WASM in-process. A function compiled to WebAssembly runs under Wasmtime inside the executor. Each call gets a fresh store with 100 million units of fuel, 64 MB of linear memory, one memory, one table, one instance, a 30-second wall clock and a 64 MB output cap. There's no WASI, so the module has no files, sockets, clock or environment beyond what I hand it: the only imports are a log function capped at 8 KB per message and a millisecond clock. SIMD and threads are off and NaNs are canonicalized, so results don't depend on the CPU. Modules are compiled once per content hash and cached, and the bytes are hash-checked before compiling. Input and output cross the boundary as Arrow IPC buffers.

One detail I got wrong first: Wasmtime's async call runs a guest to fuel exhaustion without yielding. So with a big fuel budget, a looping guest starves the async worker it's on, and the outer 30-second timeout never gets a chance to fire. The fix is to make the store yield every million units of fuel.

gVisor pods. Agents I run for other people, and builds of other people's code, run as Kubernetes pods under gVisor's runsc. It's a user-space kernel: the container's syscalls go to a Go process (the Sentry) that implements most of Linux itself, and only a small set ever reaches the host. More on how that went below.

A Firecracker microVM per call. For an untrusted function that isn't WASM, the executor boots a Firecracker microVM, runs that one call and throws the VM away. The guest is a small Debian image with Python and pyarrow. It has no network device and no environment variables, and it gets exactly four block devices: the root filesystem, the input and the code (all read-only), and a writable output drive. The runner in the guest checks the code's hash before executing it. The host side decides success by reading the output drive, not by trusting the VM's exit code. The VMM runs under Firecracker's jailer, chrooted, as an unprivileged user. Each VM gets one vCPU and 512 MB by default and is killed after five minutes.

Two caveats so this doesn't sound like more than it is. The guest only runs Python right now. The docs I wrote early on say "untrusted native or Python code", but there's no native guest, so untrusted native code just has no tier. And I don't set jailer cgroups, so the VM's size is the only resource limit, not a host-side cgroup.

Startup cost goes up sharply as you go down the list, and I don't have good measurements for most of it. Native and loaded libraries cost nothing per call. WASM costs a module compile the first time a hash is seen and a store per call; I haven't benchmarked either. Firecracker is usually quoted at around 125 ms to boot. That's from published numbers, not mine. My only measurement is end-to-end: the first working run pushed five rows through the whole path, from submitting the job through the jailer, the VM and the Python runner and back, in about three seconds. Each launch also copies the whole root filesystem into the jail, and as far as I can tell every call is a cold boot, so I'd guess the real per-call cost is well above the headline boot time. Snapshot restore is only partly written.

What fail-closed means here

The rule I wanted was that an untrusted function never runs in the executor's process, even if the sandbox is misconfigured. The way it's built: the process-wide sandbox starts out as a denying implementation, and only a successful registration replaces it. A stage that reaches the denying sandbox gets an error:

refusing to execute untrusted UDF in-process: no sandbox backend registered
(requested runtime FirecrackerMicroVm, udf <identity>). ...

The Firecracker backend registers only if all of these hold. The deployment opted the node in with an environment variable, /dev/kvm exists, and the kernel, root filesystem and Firecracker and jailer binaries are staged where expected. If the opt-in is missing, or /dev/kvm isn't there, the executor logs it and keeps the denying sandbox. If the node opted in but the images are missing, that's an error in the log too, and the sandbox still denies. The registration can only happen once, and there's no flag that forces in-process execution. An audit specifically looked for one.

That's the whole of what fail-closed means here: refuse rather than fall back. It doesn't route the stage to a machine that could run it. Getting untrusted stages onto a KVM-capable executor is a separate problem, and the answer for now is just that only one executor is set up to run them.

CI checks that each tier dispatches as documented and that a microVM stage without a backend gets that refusal. The test that actually boots Firecracker needs KVM, so it only runs locally; CI never crosses the real boundary.

The worst bug in all of this wasn't in a sandbox. An audit found that the trusted-library tier treated a missing owner header as "this is an internal call". Leave the header off, send a dataflow whose function names a library by hash, and the executor would fetch your library, check it against the hash you supplied and dlopen it. That was a full sandbox bypass. The fix was an explicit "internal" flag, so that a missing value authorizes nothing. If your trust check has a "none" case, look at what it defaults to.

Cloud VMs don't have /dev/kvm

Firecracker and Kata Containers both need hardware virtualization, the same KVM I once used for GPU passthrough to a Windows VM. On a cloud VM that means nested virtualization, and my cloud provider, Hetzner, doesn't offer it on any of its cloud server lines, including the dedicated-vCPU ones. There's no /dev/kvm and no vmx or svm flag. It's quick to check on any machine:

ls -la /dev/kvm                       # missing on a VM without nested virt
grep -cw -e vmx -e svm /proc/cpuinfo  # 0 when the CPU flags aren't exposed

So gVisor is the most isolation I can get on a cloud node, and anything that needs a real VM boundary has to run on bare metal. My node join tooling enforces that. Joining a node as a dedicated worker first checks for /dev/kvm and stops if it's missing, with a message saying gVisor is the ceiling on a cloud VM.

There's a second, sneakier version of the same problem. My first plan was to run the executor itself in a Kata pod backed by Firecracker, and have it start a microVM per function. That can't work: a Kata Firecracker pod is a Firecracker guest, and it doesn't have /dev/kvm either. So the executor that launches microVMs runs as an ordinary container on a dedicated node. It has to be privileged, because the jailer starts as root to build its chroot and device nodes before dropping to an unprivileged user. In practice that means the boundary is Firecracker plus the jailer, and if someone escapes the VM, they land in a privileged container. Whatever that container can reach is the blast radius, so the rule is that the launcher holds nothing beyond what it needs to start VMs: no database or object-storage credentials.

Not every dedicated machine can run it either. Of my three dedicated servers, only one had everything the microVM executor needs when I turned it on. Another had the Firecracker binaries but couldn't reach the object store where the guest images are staged from. The third had no install at all. A node selector on "dedicated" would have sometimes landed the pod on a machine where its host mounts didn't exist, so it would hang starting up. Instead, a node only gets the readiness label after I've checked it by hand, and the executor selects that label.

Turning the tier on took 14 bug fixes over about 16 hours, though every component already existed. Two of them were Firecracker behavior worth knowing in advance. Block devices are sized in whole 512-byte sectors, so a 250-byte function became an empty code drive until I zero-padded it. And the VMM exits when the guest reboots, not when it powers off: a guest that powered off after writing its output left Firecracker running with a halted vCPU, and the executor waited forever.

gVisor in practice

gVisor runs on every worker node, cloud or dedicated. Agents I run for other people are forced onto it whatever their configuration asks for, and there's a test that a request for plain runc gets ignored. Tenant namespaces have an admission policy that rejects any pod that doesn't name a sandboxed runtime class, so a namespace admin can't opt out. A few things about installing it bit me:

• Pin the release. Installing from the rolling channel, a node once joined with no runsc handler at all. Now the version is pinned and the install fails before the kubelet joins.
• Label a node as gVisor-capable only after checking the runtime loaded the handler. A pod whose runtime class has no handler doesn't fall back to runc, it just never starts. One deployment asked for a systrap handler the cloud nodes didn't have, and sat there.

Most of my software ran unchanged. What didn't:

• openat2 isn't implemented. It returns ENOSYS, and every agent workflow run started failing its first step once the runner moved from a Kata pod into gVisor. That code used openat2 with RESOLVE_BENEATH and no-symlink flags to stay inside its workspace. The fallback walks the path one component at a time with openat(O_NOFOLLOW), which gives the same guarantees.
• procfs and terminals look different. gVisor reports a zero foreground process group and tty_nr, /proc/.../children lists thread IDs as well as child processes, and SO_PEERCRED returns a thread ID where Linux returns a process ID. All of these broke a terminal supervisor that worked out what was running in a tmux pane.
• UDP segmentation offload. gVisor accepts the probe for UDP_SEGMENT, then rejects segmented sends. A QUIC client that used it got errors until I turned GSO off.
• No Docker inside. Docker-in-Docker doesn't work under runsc, so I assemble images with crane.
• Debugging from the node. A gVisor pod's host process is the Sentry, so nsenter into it doesn't give you the container's view. I use the container runtime's exec instead.

Performance was mostly fine for agents, which spend their time waiting on a model. The exception was process creation. Each tmux client a supervisor spawns to capture a pane costs a lot more under gVisor than under runc. Polling every 1.5 seconds, idle sandboxed agent pods used up to 2.4 CPUs, two thirds of it in those tmux clients. Now an idle pane is polled as slowly as every 12 seconds, and a send wakes the loop right away. I haven't re-measured since. I've seen 5 to 20% overhead quoted for syscall-heavy work under gVisor, and more for I/O. I haven't measured that myself.

The biggest surprise was accounting. cAdvisor reported zero memory for gVisor containers, because the usage shows up in a sandbox cgroup outside the pod's slice. Meanwhile one sandboxed agent requesting 512 MB was really using 2.25 GB at p95 and 2.6 GB at peak. The kubelet's resource metrics endpoint does report gVisor memory, so I switched to that. gVisor's CPU counters still overstate real CPU use, so I size nodes from node-level CPU instead. If you pack nodes on what Kubernetes thinks gVisor pods use, you're packing on wrong numbers.

Kata, and a node that disappeared

For a while I also ran Kata Containers on the dedicated nodes: warm executor pods under both the QEMU and the Firecracker runtime classes, and Kata Firecracker for some tenant pods. Each Kata pod boots a guest with 2 GB of memory, which is why its runtime class overhead is around 2.3 to 2.5 GB per pod.

Then one dedicated node went unresponsive for over three hours. SSH couldn't even finish its banner exchange, the provider's soft power button did nothing, and a hardware reset brought it back. At the time of the crash the root cgroup showed 64 GB in use against 9.5 GB in the Kubernetes pods slice. About 54 GB was memory the scheduler, pod limits and kubelet eviction couldn't see. Kata was running its VMMs in a separate overhead cgroup, not the pod's, and force-deleted or evicted Kata pods could leave their VMM running. On another node I found 36 orphaned Firecracker processes holding 19 GB, some of them weeks old. The warm executor pods got recreated on every image rollout, so each teardown that failed under memory pressure leaked one more VM.

The fixes were to put the VMM in the pod's cgroup (Kata's sandbox_cgroup_only), cap the overhead cgroup so a future leak runs out of memory inside it and not on the node, and add a reaper for VMMs with no live pod. The warm Kata pods went to zero replicas and haven't come back. Kata now covers a few opt-in cases, like one kind of mobile app build. A plain runc pod that starts a jailed microVM only when a call arrives turned out to be much easier to live with than a fleet of long-running VMs.

Dedicated servers versus cloud VMs

This is where the sandboxing choices meet the bill. My cluster has two kinds of nodes:

• Dedicated servers. Three of them, together about $195 a month on the contracts I have (an estimate, not an invoice). New orders cost more after a price rise: a 16-core Ryzen with 128 GB lists around $300 a month new, plus setup. They have /dev/kvm, big local NVMe and no autoscaling: getting another one means ordering it or picking one from the provider's server auction, which takes minutes to hours, and then joining it to the cluster by hand.
• Autoscaled cloud VMs. The burst pool uses 16 vCPU / 32 GB VMs at about $0.24 an hour after the same price rise, which is about $173 a month if one ran all the time. They come up in minutes through an API and go away when idle. No /dev/kvm.

So a 16 vCPU cloud VM that runs all month costs more than half of a new 16-core dedicated server with four times the memory, fast local disk and /dev/kvm, and about what those servers cost on older contracts. Cloud only wins for capacity you'd otherwise leave idle, and the CI cost post is mostly about how hard it is to make an autoscaler actually give that capacity back.

So the split I've ended up with:

• Bursty, high-density and moderately untrusted work runs under gVisor, so it can go anywhere, including the elastic cloud pool: other people's agents and builds of other people's code. gVisor is the ceiling on cloud nodes, and for most of that work it's enough. My burst build servers also use cloud nodes, but they only build my own code.
• Work that needs a hardware boundary goes on dedicated servers: per-call Firecracker for untrusted functions, and the occasional Kata pod. That capacity is fixed, and adding more is my decision, not an autoscaler's.
• Trusted, stateful and heavy work also goes on dedicated servers: warm build caches, databases, and my own development agents, which use plain runc because some of them need things like FUSE.

My guess, without traffic to prove it, is that untrusted Python functions stay rare enough that one or two KVM-capable machines are plenty. If that changes, a bare-metal autoscaler against the provider's ordering API is possible, but it would be a much slower loop than the cloud one.

What I'd tell someone starting

• Decide the tier by how much you trust the author, not by the language. Then make the default for unknown code the most restrictive tier, or refusal.
• If people can compile to WASM, use it. A capability-free Wasmtime store with fuel, memory and time limits runs on any node and needs no privileges. Make sure your timeout can actually fire.
• Fail closed means refuse. Write the refusal path first, test it in CI, and look for any "missing value means internal" shortcut in your trust checks.
• Check for /dev/kvm before planning on Firecracker or Kata. On a cloud VM without nested virtualization, gVisor is the ceiling. A VM-based pod can't host further VMs either.
• Budget for gVisor's syscall gaps (openat2, procfs quirks, GSO, no Docker-in-Docker) and fix your memory metrics before you trust them for packing.
• If you run VM-based pods, charge the VMM's memory to the pod and reap orphans, or the scheduler will pack nodes that are already full.
• The process that launches microVMs is part of your trusted base. Keep it as small and credential-free as you can.

Related posts

• The importance of remote build servers for agents
• What CI costs on Hetzner versus GitHub Actions
• What I learned running a fleet of AI coding agents
• Giving coding agents credentials without giving them the secrets