Sharing in-memory data with code you don't trust

Ryle · Nov 4, 2026

For years I've wanted a data cluster that many people can share the way they share a database: one copy of the hot tables in memory, each person running their own functions over it, and each person seeing only the rows and columns their policy allows. The parts exist separately. Spark lets everyone on a cluster share cached data. MicroVMs and WebAssembly run untrusted code safely. Row-level security decides what each user reads. Putting them together is where it falls apart. Once a user's own code runs next to the shared data, the policy only holds as long as that code behaves.

I've come at this from several directions, starting with a Spark fork, and I haven't solved it. This post covers how I got here, why the obvious fixes don't work, the approach I've been working on, and where sharing memory with sandboxes gets hard.

One version of each library

A Spark application's executors are long-running JVMs, and every task on an executor loads user code through the same classloader. SparkContext.addJar can add a jar while the application runs, but it can't replace one. Add the same path again and Spark logs that "Overwriting of added jar is not supported in the current version" and ignores it. Add the new version under a different file name and it goes on the end of the classpath. The classloader searches jars in the order they were added, so any class both versions contain still loads from the old jar, and only classes added in the new version load from the new one. You get a mix of both. So a long-lived shared cluster runs one version of each library, the one it started with, until someone restarts it and everyone's cached data goes with it.

The Spark shell has a way around this for code typed at the prompt. Each line compiles into new classes under new names, inside a wrapper generated for that line. The driver writes those classes to a directory it serves, and each executor fetches them from the driver through a classloader of its own. Redefining a function in the shell creates a new class instead of changing an old one, which is why it works. That only covers what you type. Libraries on the classpath are still one version at a time, and the Scala REPL's :require command, which adds a jar from the prompt, refuses any jar that contains a class already on the classpath.

I wanted a shared cluster to treat whole libraries the way the shell treats lines, so I forked Spark to give each user their own classloader on every executor. Two people could load different versions of the same library on one cluster and redefine their pipeline functions whenever they liked, as in a REPL. They could still share cached tables and datasets with each other, as long as both could decode the rows. DataFrames make that easier than it sounds. Spark caches them in its own binary format, independent of the classes that produced them, so code built against a different version of a library can read them back through an encoder for a compatible schema. A cached RDD of objects is harder, because those objects belong to the classes that made them.

Shading doesn't save you

The usual fix for two versions of one library is shading: rewrite one copy's bytecode so com.example.lib becomes, say, shaded.v2.com.example.lib. Both copies then fit in one classloader because they no longer share names. Spark does this to its own copies of Guava and Jetty, moving them under org.sparkproject so they can't clash with the versions an application brings. It works well for a dependency hidden inside one jar. It works badly when the library you want two versions of is your own, the one whose classes describe your data.

A renamed class is a different type. A shaded.v2...Trade isn't a Trade, so anything passed between versions has to be converted. Code that finds classes by name breaks unless every string naming one was rewritten too: reflection, ServiceLoader files, serializers that record class names, Kryo registrations. And keeping two versions of your own code apart means shading every build under a new package name, so each release becomes a new set of types that can't talk to the last one.

So you end up with one of three options, all bad. Do everything in the REPL. Split the library into a stable part that's loaded once and a part you reload, and keep moving code between them. Or relink and restart every time something changes. Each one bends how you structure and develop the library around a limit of the cluster.

Pitching it to Databricks

At one point I talked to Databricks and proposed doing this properly upstream. They didn't consider it a priority, so I gave up on the fork. Without upstream support, a fork like that has to be carried forward through every Spark release.

Spark later grew part of it. Spark Connect, added in 3.4, splits the client from the cluster, and since 3.5 each Connect session can add its own jars and class files, which executors load through a classloader for that session. The ticket that added it describes my old problem exactly: "a Spark cluster would only be able to support a single REPL at a time", because "REPL-generated classfiles as well as user-added JARs may conflict if there are multiple users of the cluster." Spark 4.0 extended it to ordinary sessions, though the shell and classic PySpark turn it off. That's close to what my fork did, and like my fork it separates class names and nothing else. Spark's own jars still come before a session's, and every session's code runs in the same executor JVMs with the same access. Spark's security page says that "Open-source Spark Connect does not provide user isolation."

Then customers wanted to run code

Later, at several different companies, I heard the same request from the product and business side: let our customers run code on our Spark clusters, against the datasets we already hold in memory. You'd expect that to be routine. Sharing in-memory datasets is the main reason to run a big shared cluster.

For SQL it's workable. The engine parses the query, applies row filters and column masks, and only the engine's own code touches the data. It stops working the first time a query needs a user-defined function (UDF) or an arbitrary lambda passed to map. That function is bytecode running inside the executor JVM, and it gets everything the executor has:

the executor's files, environment variables and network access;
its cloud credentials, which can usually read every bucket the cluster can;
every other user's cached blocks, through Spark's internal APIs or reflection;
the ability to start processes that outlive the task.

So letting a customer run a UDF on a shared cluster is letting them run arbitrary code on it, remote code execution (RCE) in security terms, and from there it's a short step to reading data their policy says they can't see. Spark's security page says so directly: "remote code execution is fundamental to Apache Spark's design and purpose", and anyone who can use a cluster "generally already has total control over the resources assigned to their Spark application." The JVM used to have an in-process sandbox, the SecurityManager, but Spark never used it to contain user code, and the JDK deprecated it in version 17 and disabled it permanently in version 24. PySpark gets a little further because Python UDFs run in separate worker processes, but those run as the same OS user on the same machine and can read whatever the executor can.

I spent a while looking for a way around this inside Spark and didn't find one I'd trust. Databricks came to the same conclusion. Its 2024 announcement of Lakeguard, the isolation layer behind its shared (now "standard") clusters, says that "Spark does not offer a mechanism for isolating user code", and the paper describing it says that before Lakeguard "all user code was executed directly in the engine's JVM, allowing users to access all data, credentials, and secrets processed in the environment."

Lakeguard uses Spark Connect to separate each user's client code from the driver's JVM and runs it in its own container on the driver node. Python and Scala UDFs run in sandboxes that Databricks' cluster manager creates, and all traffic between Spark and a sandbox goes through the cluster manager. Each sandbox runs one user's code, is reused for that user's session, and gets its input by having batches of rows moved into it.

That's a real improvement, and its limits are the ones this post is about:

The sandboxes are containers on the cluster's own VMs. The paper explains that Firecracker needs direct access to the machine, while Databricks deploys clusters to standard cloud VMs where bare metal isn't feasible, and it describes the result as sharing resources "between users in the same organization". Different customers get separate machines instead: Databricks' serverless compute is "dedicated exclusively to that workload, and securely wiped" afterward.
The engine still holds every user's data and credentials in one JVM. Jars an administrator allowlists load into it, and Databricks' documentation warns that "Allowlisted artifacts can access cluster resources and user data."
Some workloads can't run there at all. Standard clusters don't support R, GPUs, the machine-learning runtime or the SparkContext calls that create RDDs.
Dedicated clusters, assigned to one user or group, run classic Spark, so they can't apply row filters or column masks "without risking over-fetching". They send those reads to serverless compute, which hands the filtered rows back through temporary files in cloud storage: a copy per query.
Python UDFs may open connections on ports 80, 443 and 53. A shared function can send out whatever rows its caller's query gives it, so the sandbox protects the cluster from a UDF's author while the function's callers still have to trust that author with their rows.

DataFusion has the same problem

Next I looked at Rust. I find DataFusion and Polars-style engines more interesting than Spark now: columnar, vectorized and native from the start. Spark's fast paths have gone native anyway. On Databricks, Photon replaces the JVM's execution of Spark SQL with a C++ engine for the operations it supports. For open-source Spark, Gluten hands execution to native engines such as Velox, a C++ library, and DataFusion Comet runs many operators as Rust code on DataFusion.

The trust model stays the same. A DataFusion UDF is Rust code compiled into the query process, and DataFusion's security guide says "the end application is responsible for security decisions." In datafusion-python, a Python UDF runs in the same process, called while holding Python's interpreter lock. A Polars plugin is a Rust shared library that the engine loads into its own process and calls directly. Moving from the JVM to Rust makes UDFs faster, and they still run next to the data. Native code even makes it easier to go looking, since it can read any byte in the process without reflection.

People are working on sandboxed UDFs for DataFusion. InfluxData's datafusion-udf-wasm runs them under Wasmtime, including Python as CPython compiled to WASM, and passes Arrow data in and out as IPC bytes. The code is contained, and every batch is copied into the sandbox and back.

Sandboxes sit apart from the data

Most current work on running untrusted code uses sandboxes. AWS built Firecracker, a minimal hypervisor, to run Lambda functions in microVMs, and agent sandboxes such as E2B and Vercel Sandbox use it too. gVisor, which Modal's sandboxes currently use by default, puts an application kernel that runs in user space between a container and the host. Cloudflare runs many customers' Workers in one process as V8 isolates, and its tools for AI agents use both kinds of sandbox: Code Mode has a model write TypeScript that runs in a fresh isolate with internet access cut off, and the Sandbox SDK runs each sandbox in its own microVM.

These boundaries differ in strength. A plain container shares the host's kernel, so one kernel bug reachable from inside it is enough to escape. In gVisor that kernel is the Sentry, a Go program that handles the container's system calls and makes only a minimal set of host system calls itself. It's still a process on the host and not a virtual machine. By default it catches system calls with seccomp, and its optional KVM platform uses hardware virtualization only to make address-space switches faster and better isolated. Escaping takes a bug in the Sentry and then a bug in one of the host system calls it's still allowed to make. Kata Containers and Firecracker run the code in a real virtual machine with its own guest kernel. The host sees only KVM and a virtual machine monitor, and the CPU's virtualization hardware keeps the guest's memory apart. I treat gVisor as a step below that boundary. It's the most isolation I can get on cloud VMs without nested virtualization, and code that needs a hardware boundary runs on machines with KVM. None of them stops hardware side channels; gVisor's security model says plainly that it "does not provide protection against hardware side channels", and that comes up again below.

They isolate code well, but they aren't wired into a data pipeline. A sandbox takes in a request, a message or some files and returns output. Data gets in by being serialized and copied, and results come back the same way. The warehouses use the same model for customer code. Snowflake described its sandbox for Java, Scala and Python functions in 2022 as Linux namespaces, a seccomp filter on system calls, and a proxy that all outbound traffic goes through, and a 2026 paper from its engineers says Snowpark now uses gVisor. BigQuery runs Python UDFs in containers with the network blocked unless you attach a connection, and its remote functions are HTTP calls to a Cloud Run service the customer deploys. Lakeguard's UDF sandboxes reach Spark only through the cluster manager that created them. Each makes the code safe by putting a boundary between it and the shared memory, and every call pays to move its data across.

Security-first: a copy per user

The most security-focused organizations don't try to share at all. In a public presentation at Bloomberg, speakers from the NSA, where security outranks everything, described their answer for big Spark and Jupyter clusters: copy the data out per user. An export job applies each user's row-level security (RLS) and role-based access control (RBAC) policy and writes that user their own copy of the table, and each user gets their own environment on the cluster with capped memory and CPU. Nobody shares memory with anyone, so a UDF can only reach data its author was cleared to see.

That's the right call when a leak is the worst possible outcome, and it's enormously wasteful. Say 40 analysts read a 2 TB hot table: that's up to 80 TB of exports and memory, most of it identical, each copy going stale on its own schedule, and each user waiting for a warm-up before their first query. The storage side has had per-cell security labels for a long time; Apache Accumulo, which came out of the NSA, labels every cell. Those labels stop helping once arbitrary code holds the rows in memory, so the compute side falls back to copies.

The first three are what people run today. The fourth is what I'm after, and the rest of this post is about why it's hard.

My attempt: sandboxes in the pipeline, reading Arrow

My attempt starts from the pipeline. Every function a pipeline runs goes through the executor, and the executor picks a sandbox for it from who wrote the code and what it's compiled to. I described the tiers in an earlier post: built-in operators and a few trusted libraries run in the executor's process, anything compiled to WebAssembly (WASM) runs under Wasmtime in the same process, and untrusted Python goes to a Firecracker microVM. Three things make this different from a Spark executor running a UDF.

Policy is applied before the sandbox sees anything. A stage's input comes from the engine's read service, and that's where the user's policy is enforced, whether the rows come from Postgres or from Arrow files in columnar storage. The executor only gets rows the read service already decided the user may read, and each sandbox only gets its own stage's batches. A WASM guest imports almost nothing from the host. It has no way to ask for more data, open a file or reach the network.

Arrow IPC is the only interface. Arrow's IPC format is a byte stream of record batches whose buffers are found by offsets within each message. For WASM, the executor writes a stage's input as one IPC stream into the guest's memory and reads one back. A microVM gets the same bytes on a read-only virtual disk and writes its result to a second disk, which the host reads after the VM exits, without trusting the VM's exit code. Output goes through the arrow-rs IPC reader with validation on, so a malformed batch fails there and goes no further.

Code is identified by its hash. Sandboxed functions are stored and loaded by the hash of their build, so two people's versions of the same function run side by side in one executor. That's what my Spark fork used classloaders for, and it comes for free when nothing is looked up by name. A WASM module also carries its own copy of every library it was built with, so library versions can't collide either.

Copying each stage's input into every sandbox is the sandbox-per-call model from earlier, with the policy applied once by the engine, outside anyone's code. The idea on top of that is shared memory: a read-only cache of Arrow batches that every sandbox maps without copying, split so each sandbox sees only the parts its user may read. That's where the hard problems are.

Where shared memory breaks it

Mapping one copy into many sandboxes removes the copy, and the copy was doing work: the engine wrote exactly the rows a user could see into memory only that user's code could reach. With shared memory a sandbox can read every byte of every page it's given, so what matters is what else is on those pages. These are the problems I keep running into.

Pages are coarser than policies. Memory protection works on pages, 4 KiB on most x86 machines, and policies work on rows. Arrow stores each column as a contiguous buffer, so one page of 8-byte values holds 512 rows, and in arrival order nearly every page mixes rows that different people may read. A sandbox can't be given part of a page. The data has to be laid out by policy first: group the rows by the set of people who may read them, and pad each group's buffers out to a page boundary.

A column whose rows belong to policies A and B. In arrival order every page holds both, so none can be mapped for A alone. Grouped by policy and padded, pages 1 and 2 hold only A and pages 3 and 4 only B. — Schematic. A real 4 KiB page holds 512 eight-byte values, so without a layout by policy almost every page mixes the two.

Each row is still stored once, and each user maps every group they're allowed to read. Column policies fit this well, since Arrow keeps each column in its own buffers and leaving a column out means leaving its buffers unmapped. Row policies get worse as they multiply. If 10,000 customers each see only their own rows, a 50-column table becomes 10,000 groups of 50 columns, about a million buffers at two per column, and each buffer can carry up to a page of padding: as much as 4 GB of padding before any data. With overlapping policies the number of groups can approach the number of rows.

Filtering in place leaves the filtered rows behind. Arrow libraries avoid copies wherever they can, and here that works against the policy. A slice is an offset and a length into the original buffer, so everything outside the slice is still in memory. In arrow-rs, the Rust Arrow library under DataFusion, filtering a dictionary-encoded column filters the keys and keeps the whole dictionary, so every distinct value in the column survives the filter. Filtering a string-view column drops the 16-byte views and keeps all the original string data they pointed into. The IPC writer sends those buffers as they are, and a comment in its source says to garbage-collect view arrays first if you want them compacted. So the policy layout has to be built by copying into fresh, compact buffers when the data is loaded. That's one copy per group of readers instead of one per user, and every new Arrow layout or kernel is another place a filtered row could survive.

WebAssembly can't map read-only memory yet. A WASM module's linear memory is one read-write buffer with no protection inside it. Wasmtime lets the host decide where that buffer lives, but it documents no way to make part of it read-only. The WebAssembly memory-control proposal sketches read-only sections and mappable regions and names the problem as "a heavy reliance on copies when interacting with the host", but it's at phase 1, an early stage, and no engine ships it. So there's no supported way to hand a WASM guest a shared cache it can read and not write. A microVM can have one: since December 2025 Firecracker has had a persistent-memory device that maps a host file into the guest's memory, with a read-only mode.

Validation races with writable memory. Results come back the same way. If a sandbox writes Arrow buffers into memory the engine reads, the engine has to check them first: offsets in bounds and increasing, dictionary keys inside the dictionary, strings valid UTF-8. A check only holds while the memory can't change. If the sandbox can still write to that memory, it can change an offset after the engine checks it and before the engine uses it. Kernels and hypervisors have had this bug for years under the name double fetch, and the fix carries over: copy the result out, or take away the sandbox's write access before validating. On Linux, a memfd can be sealed against writes, and the seal can only be applied once no writable mapping of it remains.

The Arrow C Data Interface, the usual way to hand Arrow memory over without a copy, doesn't fit this boundary at all. It's meant for libraries in one process that trust each other: it passes raw pointers and a release callback for the consumer to call, and Arrow's security guidance says to "never consume a C Data Interface structure from an untrusted producer." IPC locates buffers by offsets within each message and has nothing to call, which is the right shape for a boundary.

Shared hardware leaks through timing. With every page right, sandboxes that share a core or a cache can still learn about each other's data from how long things take. Spectre, disclosed in 2018, showed how far that goes. The CPU guesses which way a check will go and runs ahead before it knows; when the guess was wrong it throws the results away, but the memory it touched stays in the cache, where timing can reveal it. Google's V8 team concluded the next year, in a paper called Spectre is here to stay, that these attacks "defeat all language-enforced confidentiality", and wrote that "the only effective mitigation is to move sensitive data out of the process's address space." Chrome did that by putting different sites in different processes.

That covers WASM in the executor's process too. The guest's memory bounds hold for normal execution, and speculation can run past them. Wasmtime includes "a few forms of basic spectre mitigations" and says that mitigating Spectre "continues to be a subject of ongoing research". An attack like this also needs a precise clock, so one defense is to take the clock away. Cloudflare's Workers do that: Date.now() doesn't advance while code runs, and threads and shared memory aren't allowed.

MicroVMs help, and they share hardware too. Firecracker's guide for production hosts recommends turning off simultaneous multithreading (SMT), which runs two hardware threads on one physical core, wherever tenants have to be kept apart, because SMT is "frequently a precondition" for Spectre-style leaks between them. The persistent-memory device has a catch of its own. Firecracker's documentation warns against giving several VMs the same backing file, because that "causes the same physical pages to be mapped to different VMs, which could be exploited as a side channel". A VM sharing physical pages with another can time its own reads to tell which parts the other touched recently, and what a query touches can depend on data the first VM isn't allowed to read. A shared Arrow cache mapped into many microVMs is exactly that setup.

Outputs carry what went in. A sandbox that reads one group's data and writes a result has made something only that group's readers should see. If the result lands in a table or cache that other people read, the policy leaks through the front door. A function can also try to signal through anything it controls: how long it runs, how much memory it uses, whether it fails. Butler Lampson called this the confinement problem in 1973, and I think it's still the hardest part. A coarse version of the labeling is cheap: if a function can see any private column, mark its whole output private, since an opaque function could have read any of its input.

Designs for sharing memory more safely

None of these problems has a complete answer, but I've sketched designs that narrow them.

Share only what everyone may read. Laying out every private row by policy gets expensive fast, so the design keeps two tiers. One physical copy of each common dataset lives in a shared read-only Arrow cache: public data such as a price feed or a dimension table, plus aggregates that passed a release policy. Each tenant's private data lives in that tenant's own partition and is never shared by reference. A plan may name common datasets and the caller's own private ones, and a plan that names anyone else's private data is rejected before it runs. Private rows give up zero-copy sharing, but that's where the policy groups multiply, and the big tables many people scan are usually the common ones.

Labels come from how data was computed. A pass over the plan works out each column's label from how it was produced: public, private to one tenant, or reduced, meaning derived from private data through an aggregation whose conditions hold. The same price column can be a tenant's secret raw input and also the basis of an average that's safe to release. Private data reaches another tenant only through a reduction like that. A sum or count needs groups of at least k rows, checked on the groups actually emitted, since a join can shrink a group to one row. A moving average needs a stride at least as long as its window, since overlapping windows can be differenced back into the raw series. A budget limits how many aggregates one caller can combine, and differential privacy fits in the same place. Aggregates that pass can be promoted into the shared tier, so one tenant can join against another's released aggregates and reach nothing else of theirs. The guarantee is what information-flow research calls delimited release: nothing leaks beyond what the release policy declares.

Code holds handles. User code names datasets by handle and never sees an address. The function that turns a handle into memory depends only on the shared tier and the caller's own partition, which is a property small enough to state and prove. The read path helps the same way. The owner filter is applied at the leaves and every operator above it is monotone, so rows the caller can't see can never change what the caller gets back. Only an operator that reacts to a missing row could carry another tenant's data into a result, and those are the operators a planner already has to find for CALM. Memory a sandbox does get comes as a capability: read-only, valid for one call and revoked after it, with guard pages between regions and copy-on-write for anything the sandbox may change, merged only after validation.

Keep the hardware quiet. Private data goes in a separate cache arena per tenant, so one tenant's queries can't evict another's pages and time the result, and shared caches hold only data everyone may read. Response times and request rates can be normalized so a query's timing says less about what it touched. Pools of snapshot-restored sandboxes are keyed by tenant as well as code, so two tenants with byte-identical functions never resume the same guest memory.

What I think a solution needs

This is the list I check new designs against:

The engine evaluates every policy. User code never decides what it reads. It gets buffers the engine already filtered, and no handle to storage, credentials or the network.
Shared memory holds only what all its readers may see. Common data and released aggregates go in one read-only tier. Private data stays in each tenant's arena, or is grouped by the set of people who may read it, compacted and padded to page boundaries where sharing it is worth that layout.
Buffers go one way. Input is mapped read-only. Output goes to separate memory that the sandbox loses write access to before the engine validates it.
Code gets handles and short-lived capabilities. A handle resolves only into the shared tier or the caller's own data, and access ends with the call.
Sandboxes are cheap to start. Snapshot-restored microVMs and precompiled WASM modules make a fresh sandbox per query or per call affordable, and no sandbox is reused across tenants.
Scheduling knows about clearance. Sandboxes holding different groups' data don't share a physical core or a private cache, and the most sensitive ones don't share a machine.
Outputs carry their inputs' labels. A result goes only to people allowed to read everything the function was given, whether or not it used all of it, unless a reduction that meets its conditions releases it.

Research systems have built pieces of this. Airavat (NSDI 2010) ran untrusted MapReduce mappers under mandatory access control and used differential privacy, which adds noise to results, to release aggregates to people who couldn't read all of the inputs. Ryoan (OSDI 2016) confined untrusted modules in sandboxes protected by Intel SGX enclaves, so that neither the code nor the machine it ran on could leak the data, and each module processed a request once without keeping anything from it. I haven't found anything that does all of it for in-memory analytics at Spark's scale, and I'd like to. A shared cluster where everyone runs their own code over the same hot tables, each under their own policy, would replace a lot of copying, waiting and per-team clusters.

What I'd tell someone starting

If SQL is enough, stop there. Row filters and column masks applied by the engine are well understood, and nothing the user writes runs.
If customers need their own code, assume it can read everything its process can reach: memory, files, environment variables, credentials and network. Pick the boundary from that, not from what the code is supposed to do. A VM boundary such as Firecracker or Kata is stronger than gVisor, and gVisor is stronger than a plain container.
Filter before the boundary, in code you wrote. The sandbox should receive data that's already safe for it to see, and nothing that lets it ask for more.
Use Arrow IPC across the boundary and validate what comes back. Keep the C Data Interface for code you trust.
Compact before you share. Slices, dictionaries and string views can all carry rows the filter removed.
Start with a copy per call. It's simple, and for many user functions, Python ones especially, the copy costs less than the function. Work on shared memory once many users scan the same large tables and the copies show up as the cost.