Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes
Yiming Yan, Hongyang Wang and colleagues (Amazon) describe Sandman, a sandbox service built and operated for agentic reinforcement learning on shared Kubernetes, with production data from four SWE-bench Verified training campaigns that created 3.1 million sandboxes in October 2026.
Ask this paper
Design. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified task image, and files and processes persist across calls until the framework releases it.
Scale. A scale test held 11,858 concurrent tiny sandboxes. Production ran 122 runs on up to 432 B200 GPUs.
Image pulls. A create that had to pull an image took 17.4 seconds at the median with a p90 of 74.3 seconds, four times the median create.
Where time goes. Rollout acquisition took at most 0.9% of step time on the critical path. Tests took 8-11% with one suite and 38% with two, and sampled rollout sandboxes sat idle 74-91% of their lifetime.
Failure case. With Service creates unpaced, 241 of 1,000 sandboxes failed in a Service-routing test, 237 of them on throttled creates.
Abstract
Agentic reinforcement learning (RL) often requires repeated tool calls in task-specific environments. A training job may create thousands of these environments in bursts while sharing a cluster with other jobs. We present Sandman, a sandbox service built and operated at Amazon for agent tool execution and evaluation. It serves multiple production agentic RL workloads on shared Kubernetes. A training framework supplies a task image and uses commands and file operations in the returned sandbox. Files and processes persist across calls until the framework releases the environment. Sandman manages creation, readiness and cleanup. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified image. In a scale test, Sandman held 11,858 concurrent tiny sandboxes. Four production SWE-bench Verified training campaigns created 3.1 million sandboxes in October 2026. We report the design trade-offs and operational lessons from handling creation bursts, preparing task environments, reclaiming abandoned sandboxes, and running evaluation under controlled conditions. We will open-source Sandman's daemon, provider and pool.