Skill Sandboxing Patterns: The Three Isolation Modes And Their Realistic Failure Surfaces

TL;DR

Agent skills run in three common sandbox modes: process isolation (fast, weak), container isolation (medium speed, medium strength), and microVM isolation (slow, strong). Most teams default to whichever their platform offers and accept the trade-offs without examining them. This post lays out what each mode actually protects against, the realistic escape vectors per mode, the operational characteristics that drive selection, and a Sandbox Mode Selector that any team can use to choose mode-per-skill. The framework introduced here is the Trust-Latency-Cost Triangle: every sandbox decision trades off how much you can trust the skill, how fast it has to run, and how much you can pay per invocation. You can have any two; you cannot have all three.

Why The Default Sandbox Is Almost Always Wrong

The usual story for sandbox selection in 2026 looks like this. A team picks an agent platform. The platform has a default sandbox model: maybe shared-process tools running in a Node.js worker, maybe Docker containers per skill invocation, maybe something more exotic. The team uses the default. The team ships skills. Some of those skills are first-party, some are third-party, some handle high-stakes data, some handle mundane data. They all run in the same sandbox, with the same isolation properties, because the platform did not present any other options and the team did not think to ask.

This is the wrong shape. Different skills have different trust profiles, different latency requirements, and different blast radii if they are compromised. A first-party skill that handles internal logging is a different security situation than a third-party skill that ingests untrusted file uploads. Treating them with the same isolation is either over-engineering one (paying for security you do not need) or under-engineering the other (running untrusted code with too little containment). The right shape is mode-per-skill, with the mode chosen deliberately based on the skill's specific properties.

The failure mode this post addresses is the silent acceptance of platform defaults. We have done audits where the team confidently described their isolation model, walked us through the architecture, and was unaware that the very platform they were using had publicly documented sandbox escape vectors that affected their specific deployment. The platform's defaults were chosen for convenience, not for any specific threat model that matched the team's actual workload. The team had inherited the convenience and accepted the security implications, without ever framing the security implications explicitly.

The purpose of this post is to make the framing explicit. We will walk through the three common isolation modes, describe what each actually protects against, enumerate the realistic escape vectors for each, and end with a selector that helps you choose mode-per-skill based on the properties of the skill rather than the defaults of the platform. The selector is a one-page artifact that you can fill out for each skill in your fleet, and the result is a fleet where the isolation mode matches the actual trust and risk profile, instead of being whatever the platform happened to ship.

The Trust-Latency-Cost Triangle

The framework that organizes the rest of this post is the Trust-Latency-Cost Triangle. Every sandbox mode trades off three properties: how much you can trust the skill running inside it, how fast the skill has to be invokable, and how much you can pay per invocation. These properties form a triangle in the classic sense: improving any one degrades at least one of the others. Pick two; the third is determined.

Process isolation gives you low latency and low cost. Skills are loaded as in-process modules or co-located worker threads. Invocation is a function call. The cost per invocation is fractions of a cent. The trust profile is correspondingly low: a malicious skill in process isolation has access to everything the host process can see, and skills can interfere with each other in subtle ways through shared memory, shared globals, or shared interpreter state. Process isolation is appropriate for first-party skills that you wrote, that you have read, and that you trust. It is the wrong choice for any third-party skill or any skill that ingests untrusted input.

Container isolation gives you medium latency and medium cost. Skills run in their own container, with their own filesystem, network namespace, and process tree, but sharing the host kernel. Invocation involves a container start (which can be amortized with warm pools) and inter-process communication. The cost per invocation is a few cents. The trust profile is meaningful: a malicious skill in a container cannot directly read the host filesystem or other skills' memory, and the network and process boundaries provide useful defense-in-depth. The escape surface is the host kernel, which is a large attack surface that nontrivial adversaries can sometimes find ways to exploit. Container isolation is appropriate for third-party skills that have been reviewed, that handle non-trivial data but not the highest-stakes data, and where the latency budget can absorb a few hundred milliseconds of overhead.

MicroVM isolation gives you the strongest trust profile at the cost of higher latency and cost. Skills run in their own minimal virtual machine, with their own kernel, scheduled by a hypervisor that provides strong isolation between guests. Invocation involves a VM boot (also amortizable but more expensive than container start) and the data path crosses a virtualization boundary. The cost per invocation is meaningful, often tens of cents per second of execution time. The trust profile is the strongest available outside specialized hardware: a malicious skill in a microVM cannot directly access the host kernel, the host filesystem, or other VMs, and would have to find a hypervisor escape to break out. MicroVM isolation is appropriate for skills that handle untrusted code, untrusted input, or high-stakes data, where the cost is justified by the stakes.

The utility of the triangle framing is that it forces you to confront the trade-offs explicitly. Teams that default to process isolation are implicitly saying that they trust all their skills and need maximum throughput. Teams that default to microVM are implicitly saying that they trust no skill and can pay for it. Most teams should be running a mix, with mode chosen per skill based on the specific properties of that skill. The selector at the end of the post is the tool for making the per-skill choice.

Process Isolation: The Fast And Weak Mode

Process isolation is the default in many lightweight agent platforms. The architecture is simple: skills are JavaScript modules loaded into a Node.js process, Python modules imported into a Python interpreter, or similar. Skill invocations are function calls. There is no separate process, no separate filesystem view, no separate network namespace. Everything runs in the same address space.

The performance properties are excellent. Skill startup is essentially free, since the skill is already loaded. Invocation latency is the cost of a function call plus whatever the skill itself does. Throughput is bounded only by the host process's CPU and memory. There is no overhead from container starts, VM boots, or inter-process communication. For workloads where latency matters and where throughput is the dominant cost driver, process isolation is hard to beat on operational characteristics.

The security properties are correspondingly weak. A skill in process isolation can read any memory the host process can read, including the contents of other skills, the contents of API responses still in flight, and the credentials the host process holds. It can patch any function in the host process or in other skills, including security-critical functions like authentication checks or audit loggers. It can establish outbound network connections without restriction (subject to whatever network controls the host process operates under). It can terminate the host process, hang it, or trigger crashes that corrupt other skills' state.

The realistic escape vectors are not the dramatic ones from popular threat modeling. There is no "escape" needed; the skill has full host process privileges by default. The real attack patterns are: silent monkeypatching of host code (intercepting calls to read or modify their parameters and returns), memory scraping (reading other skills' or the host's data structures directly), shared state poisoning (modifying global variables that other skills depend on), and identity confusion (impersonating other skills' invocations through the same process boundaries). None of these require any sandbox vulnerability. They are just consequences of running in the same address space.

The mode is appropriate in narrow circumstances. First-party skills that have been reviewed by a security team and that handle no untrusted input can run in process isolation safely. Skills that are essentially pure functions (deterministic transforms of declared inputs to declared outputs, with no side effects) can run in process isolation safely. Skills that are explicitly part of the runtime infrastructure, like logging or metrics or tracing helpers, can run in process isolation safely because they need to share the runtime's privileges anyway.

The mode is inappropriate everywhere else. Any third-party skill, any skill that ingests external data, any skill that handles credentials or secrets, any skill that performs side effects on systems outside the runtime: none of these belong in process isolation. The performance benefits are real but they are bought at the cost of giving up almost the entire isolation benefit, which is the wrong trade for any non-trivial trust situation.

Container Isolation: The Medium Mode

Container isolation is the most common mode in production agent platforms. The architecture is familiar: skills run in their own Docker (or similar) container, with their own filesystem, network namespace, process tree, and resource limits. The container shares the host kernel but is otherwise isolated. Skill invocations involve either a container start (cold path) or a routing decision into a warm pool (warm path).

The performance properties are mixed. Cold container starts are slow, often hundreds of milliseconds to seconds depending on the image and the runtime. Warm pools are much faster, with invocation latencies of a few milliseconds to tens of milliseconds, dominated by the IPC mechanism. Throughput is limited by the host's resources, with the caveat that each container carries some memory and CPU overhead, so the maximum number of concurrent skill invocations is meaningfully lower than in process isolation. The cost profile is dominated by the warm pool, which has to be sized for peak load to avoid cold-start penalties.

The security properties are meaningfully better than process isolation. A skill in a container cannot directly read the host filesystem, cannot directly read other containers' memory, and is bounded by the network policies the host applies to the container. The credentials the host process holds are not visible. Other skills' state is not visible. The blast radius of a malicious skill is bounded to whatever the container itself can do, and to whatever the container can affect through its declared external interfaces.

The escape surface is the host kernel. Containers share the kernel, which means a kernel vulnerability can let a container escape into host privileges. The kernel is a very large piece of software, with a very long history of vulnerabilities, and the rate of new vulnerabilities is steady. Most kernel vulnerabilities require specific conditions to exploit, and the conditions are not always met in container environments, but the historical pattern is that container escapes through kernel bugs do happen, and motivated attackers have demonstrated them in real-world incidents. Defending against this requires keeping the kernel up to date, applying defensive measures like seccomp filters and capability restrictions, and accepting that container isolation is meaningful but not absolute.

The specific defensive measures that make container isolation more robust are well-established. Seccomp filters limit the syscalls a container can make to the kernel, dramatically reducing the attack surface for kernel exploitation. Capability restrictions limit the privileged operations the container can perform even within its allowed syscalls. Read-only root filesystems prevent containers from modifying their own image at runtime. User namespace remapping ensures that the container's root user is not the host's root user, so any escape lands in an unprivileged context. None of these measures is hard to apply, and the combination significantly raises the bar for successful escape.

The mode is appropriate for a wide range of skills. Third-party skills that have been reviewed and have completed the trust boundary work described in earlier posts can usually run in container isolation safely. Skills that handle non-trivial data but not the highest-stakes data can run in container isolation safely. Skills where the latency budget is in the tens-to-hundreds of milliseconds range fit container isolation well. The mode is the right default for most agent fleets, with process isolation reserved for the small set of trusted utility skills and microVM isolation reserved for the small set of high-stakes or highly untrusted skills.

MicroVM Isolation: The Strong And Slow Mode

MicroVM isolation is the mode that handles the cases where containers are not strong enough. The architecture uses lightweight hypervisors like Firecracker or Cloud Hypervisor to run each skill in its own minimal virtual machine, with its own kernel and its own hardware abstraction. The isolation is hypervisor-enforced, which is significantly stronger than kernel-enforced container isolation. The microVM does not share the host kernel; it has its own.

The performance properties are the most expensive of the three modes. Cold microVM starts are typically 100-500 milliseconds for a Firecracker boot, faster than a full VM but meaningfully slower than a container start. Warm pools are possible but more expensive to maintain, because each warm microVM consumes meaningful host resources. Per-invocation latency on a warm microVM can match container performance, but the warm pool sizing is more constrained, so the practical throughput is lower for the same hardware. The cost per invocation is meaningfully higher than containers, often 2x to 5x depending on the workload shape.

The security properties are the strongest of the three modes. A skill in a microVM cannot read the host kernel, cannot read other microVMs' memory, cannot make syscalls into the host kernel directly, and is bounded by the hardware-virtualized resources the hypervisor provides. The blast radius is contained to whatever the microVM can do through its declared external interfaces, and the escape surface is the hypervisor itself, which is a much smaller and more carefully audited piece of software than the kernel.

The realistic escape vectors are different from containers. Hypervisor escapes are real but rare, and the hypervisor projects (Firecracker in particular) have invested heavily in maintaining a small attack surface. The escape vectors that matter in practice are usually not hypervisor escapes themselves but bugs in the surrounding plumbing: the device emulation, the I/O paths, the management plane. A microVM that calls back to a host service through a virtio device can sometimes exploit bugs in the host-side virtio implementation, even when the hypervisor itself is sound. Defending against this requires keeping the hypervisor and its device implementations up to date, using fuzzing and formal verification where available, and minimizing the surface of host services exposed to the microVM.

The operational characteristics that make microVMs viable are improving. Firecracker-style boot times are fast enough that cold starts are acceptable for many use cases. Snapshot-and-restore mechanisms can effectively eliminate cold starts by booting once and forking. Hardware support for nested virtualization is widespread enough that microVMs work on most cloud and on-prem environments. The cost-per-invocation is dropping as the technology matures and as cloud providers roll out optimized infrastructure. None of this is the world of five years ago, when microVMs were a research project; they are now production-ready for the cases that need them.

The mode is appropriate for skills that handle high-stakes contexts and skills that handle untrusted code. The first category includes payment processing skills, identity-handling skills, skills that touch customer financial data, and skills that have authority over other systems. The second category includes any skill that executes user-provided code: code interpreters, sandbox environments, evaluator runtimes, and the like. For these skills, the cost premium of microVM isolation is justified by the stakes; running them in containers exposes you to escape vectors whose probability is low but whose consequences are severe.

The Sandbox Mode Selector

The artifact this post promised is the Sandbox Mode Selector, the structured decision tool that helps you choose mode per skill. The selector is a flowchart with five questions, applied in order, that produces a recommended sandbox mode for any given skill. The recommendations are heuristics; for borderline cases the team should still apply judgment, but the selector eliminates the most common bad decisions and forces the relevant trade-offs into the open.

The first question is: does the skill execute user-provided code? If yes, the recommended mode is microVM, regardless of any other factor. The reason is that user-provided code is by definition untrusted, and the only mode strong enough to contain arbitrary untrusted code is microVM isolation. This rule is non-negotiable; we have seen too many incidents from teams that ran code interpreters in containers and then learned that container isolation was not enough.

The second question is: does the skill handle data classified as high-stakes (payment data, credentials, identity-bearing tokens, regulated data, personal financial data)? If yes, the recommended mode is microVM. The cost premium is justified by the severity of the consequences if a skill at this trust level is compromised. The exception is if the skill is provably read-only with no outbound network access, in which case container isolation may be acceptable, but the burden of proof is on the team to demonstrate the constraints and verify them in production.

The third question is: is the skill a third-party skill, or has it been written by an internal team that has not gone through your security review process? If yes, the recommended default is container isolation, with microVM upgrade if either of the previous two questions apply. The reason is that the trust profile of unreviewed code is meaningful, and container isolation provides enough containment to limit the blast radius of typical malicious behaviors while keeping the cost reasonable.

The fourth question is: does the skill's latency budget allow for container or microVM overhead? If the budget is in the seconds, both modes are fine. If the budget is in the tens of milliseconds, only warm-pooled containers will work, and the warm pool sizing will be the dominant operational concern. If the budget is in single-digit milliseconds, you may need process isolation, in which case the prior questions about trust become much more constraining: the only safe candidates for process isolation are skills that are trusted, reviewed, and handle non-stakes data.

The fifth question is: what is the cost budget per invocation? If the budget allows for microVMs, you have full flexibility. If the budget is more constrained, container isolation is the workhorse mode, and microVM is reserved for the cases where the prior questions force it. Skills that are extremely cost-sensitive but also high-stakes are an inherent contradiction; the right resolution is usually to redesign the skill so it does not handle high-stakes data, rather than to run high-stakes data in cost-optimized but security-weak isolation.

The selector produces, for each skill, a recommended mode and a documented rationale. The team applies the selector once per skill, captures the result in the skill's manifest, and uses the captured rationale during periodic reviews to verify that the assumptions still hold. When skills change in ways that affect the answers (a new feature that handles credentials, a new latency requirement, a new cost constraint), the selector is re-run and the mode is re-evaluated. The discipline turns a one-time architectural choice into an ongoing operational practice.

Mixing Modes In One Fleet

A practical agent fleet will run multiple sandbox modes simultaneously. The orchestration of multi-mode fleets is its own engineering problem, and the patterns that work well are worth describing because they are not always obvious.

The first pattern is mode-aware routing. The runtime examines each skill invocation, identifies the mode the skill is registered for, and routes the invocation to the appropriate execution backend. Process-isolated skills go through the in-process pathway. Container skills go through the container pool. MicroVM skills go through the microVM pool. The routing is invisible to the agent making the invocation; from the agent's perspective, it is all one tool call. The complexity is in the runtime, not in the agent, which is the right place for it.

The second pattern is consistent observability across modes. Process-isolated skills, container skills, and microVM skills all need to produce comparable telemetry: invocation logs, parameter capture, result capture, latency measurement, error reporting. The mechanisms differ across modes (an in-process call can capture parameters synchronously; a container invocation captures them at the IPC boundary; a microVM invocation captures them at the virtio boundary), but the resulting telemetry should look the same to operators. Mode-specific telemetry is a debugging hazard because it makes it hard to compare behaviors across the fleet.

The third pattern is mode escalation paths. A skill that is currently running in container isolation should have a documented path to escalate to microVM if its risk profile changes. The path includes the technical work to package the skill for the new mode and the operational work to update the manifest and the runtime registration. The path should be exercised periodically, even when not strictly required, because the escalation muscle gets stiff if it is not used. Teams that have never escalated a skill from container to microVM will find the first escalation slow and painful; teams that escalate routinely will find it routine.

The fourth pattern is mode demotion gates. The reverse of escalation: a skill that no longer needs microVM isolation could in principle be moved to container isolation to reduce cost. Demotion is dangerous because the trust assumptions that justified the higher mode may not be fully captured in the technical inputs. We recommend that demotion always require a security review, with explicit sign-off, and that the demotion be reversible without operational disruption if subsequent evidence suggests the demotion was wrong. The asymmetry is intentional: escalation should be cheap, demotion should be expensive.

The fifth pattern is per-mode capacity planning. Process isolation is cheap to scale because it shares resources with the host process. Container isolation is medium-cost to scale and requires warm pool capacity planning. MicroVM isolation is expensive to scale and requires dedicated infrastructure capacity. Teams that run multi-mode fleets need to plan capacity for each mode separately, with awareness of the different cost curves and the different scaling lead times. The planning is not exotic but it is rarely done well, and teams that skip it tend to discover the gaps under load.

Counter-Argument: Aren't All Three Modes Just Security Theater?

The sharpest counter-argument we hear is that none of the three modes provides real security. Process isolation is admittedly weak. Container isolation is regularly broken. Even microVM isolation has had escape vectors discovered. The argument runs that a sufficiently determined attacker can break out of any of them, and that the engineering cost of stronger isolation is not justified because none of it stops a real adversary.

The argument is wrong, but it is wrong in interesting ways that are worth working through. The first wrongness is the confusion of "any vulnerability has been found" with "the mode provides no protection." Container escapes happen, but they are rare, require specific conditions, are expensive to develop, and are usually patched quickly when found. MicroVM escapes are even rarer and even more expensive. The fact that escapes exist does not mean every container or every microVM is escapable; it means that the defender has to keep the underlying infrastructure up to date, apply defensive measures, and assume that a tiny fraction of motivated adversaries will be willing to develop novel exploits. For the vast majority of threats, including most malicious skills, container and microVM isolation provide meaningful and consequential protection.

The second wrongness is the confusion of perfect security with useful security. The point of sandboxing is not to make exploitation impossible. The point is to raise the cost and limit the blast radius. A successful container escape against your runtime is a much harder problem to solve than the equivalent attack against a process-isolated skill, and the probability of a given attacker getting to the escape stage is much lower. Even if one attacker eventually does, the marginal protection across the larger threat population is significant. This is the same calculus as encryption or any other defensive measure: not perfect, not unbreakable, but raising the cost enough that the bulk of the threat population gives up and goes after softer targets.

The third wrongness is the implicit assumption that the cost of stronger isolation is unjustified. The cost is real but it is small relative to the cost of an incident. We have worked with teams whose sandbox-related incidents cost hundreds of engineering hours and meaningful customer impact, and the marginal cost of running their relevant skills in microVMs would have been measured in cents per invocation. The cost-benefit calculus, done honestly with realistic incident probabilities, favors stronger isolation in most cases where stakes justify any defense at all.

What Armalo Does

Armalo's runtime supports all three sandbox modes as first-class options. Every skill registered with the platform declares its required mode in its manifest, and the runtime routes invocations to the appropriate backend. The Sandbox Mode Selector is built into the registration flow: registrants are walked through the five questions, and the recommended mode is captured along with the rationale. Skills can request a stricter mode than the selector recommends; they cannot request a weaker mode without a documented exception. The behavioral pacts that govern skill behavior reference the sandbox mode directly: a pact that depends on isolation properties cannot be honored by a skill running in a weaker mode. The composite score that drives certification tier includes sandbox mode appropriateness as a measured dimension, so a skill running in the wrong mode for its risk profile cannot earn higher tiers. The Trust Oracle at /api/v1/trust/ exposes the sandbox mode for each registered skill, so any agent considering importing the skill can see the isolation properties before connecting.

FAQ

Can I run all my skills in microVMs to be safe? You can but it will be expensive and slow for skills that do not need it. The right answer is mode-per-skill, with microVM reserved for the cases where the trust or stakes justify the cost. Defaulting to microVM everywhere is the inverse error of defaulting to process isolation everywhere; both reflect a refusal to make the per-skill decision.

Is gVisor a fourth mode I should consider? gVisor is a useful intermediate option that provides some of the syscall-level protection of microVMs while keeping the operational profile closer to containers. We treat it as a stronger variant of container isolation rather than a separate mode. The selector recommendations for container isolation generally apply to gVisor, with gVisor preferred when the additional protection is meaningful and the latency overhead is acceptable.

What about WASM as a sandbox? WASM is interesting for skills written specifically to target it, but it is a different kind of isolation: language-level rather than OS-level or hardware-level. It is well-suited to skills that are pure functions of declared inputs to declared outputs, with no need for the broader system access. For skills that need filesystem, network, or process operations, WASM has to either disallow those operations entirely or expose them through host functions, at which point the trust analysis becomes about the host functions and looks more like the other sandboxing modes.

How do I migrate an existing fleet from process isolation to container or microVM isolation? Incrementally, with the highest-stakes skills first. Run the selector across the fleet, identify the skills that are mismatched with their current mode, and move the most mismatched ones first. Build a feedback loop with the operations team to surface latency or cost regressions as the migration proceeds, and adjust the warm pool sizing as needed. Expect the migration to take a few quarters for a fleet of any size, and accept that some skills may end up needing redesign to fit a stronger mode comfortably.

Do warm pools defeat the security guarantees of microVMs? Carefully designed warm pools do not. The microVM is reset between invocations, with state cleared and resources released, so the next invocation does not inherit anything from the previous one. The exception is shared infrastructure outside the microVM (caches, queues, observability collectors) which has to be designed with its own isolation properties. Naively implemented warm pools can leak state between invocations and undermine the isolation, so the implementation matters.

Are there workloads where none of the three modes is enough? Yes. Skills that handle classified data, skills with extreme regulatory constraints, and skills that are part of safety-critical systems may require additional isolation beyond microVMs: dedicated hardware, formally verified runtime components, hardware security modules. These are specialized cases that warrant specialized engineering, not a different default mode. The selector here is intended for general-purpose agent fleets, not for these edge cases.

How do I verify that my chosen sandbox mode is actually doing what I think? Through adversarial testing. Run known malicious skills (or skills crafted to test specific escape vectors) in your sandbox and verify that the isolation holds. The Armalo platform offers an adversarial-agent service for exactly this purpose, but you can also build your own internal red team if the scale justifies it. The point is not to assume the sandbox works; the point is to verify it under realistic adversarial conditions before you trust it with real workloads.

Bottom Line

The sandbox mode you run for any given skill should be a deliberate choice based on the skill's specific properties, not a default inherited from the platform. Process isolation is fast and weak, container isolation is medium and medium, microVM isolation is slow and strong, and each is right for a specific category of skill. The Sandbox Mode Selector formalizes the choice into a five-question flow that any team can apply. The Trust-Latency-Cost Triangle frames the trade-offs honestly. The cost of running the wrong mode is paid in incidents you did not prevent or in money you did not need to spend. The cost of choosing deliberately is paid once, in a few hours of analysis per skill, and saves you from both failure modes.