Skip to content
Changelog

Sandboxed agents scale across replicas with S3 checkpoints

October 19, 20261 min read

Agno's S3CheckpointStore(bucket, prefix=..., endpoint_url=...) stores each sandbox turn as one object named <prefix>/<sha256>.tar.gz, with the digest and size saved in the object's metadata. FileCheckpointStore was the only store in core before this, and the queue worker refuses its default path because another replica cannot read it.

Diagram: Replica A hashes the archive, uploads it unless the key already exists, and confirms size and digest with a HEAD request before put returns. The S3-compatible bucket holds the object sandbox/checkpoints/0a9686a9….tar.gz with sha256 and size 406 as metadata. Replica B downloads it to a temporary file, checks the bytes against size and sha256, and restores the workspace and native roots. Key and size are real output from a local S3 server.

Hand the store to the sandbox the same way you would a file store. Install agno[s3] for boto3.

from agno.agents.claude import ClaudeAgent
from agno.db.postgres import PostgresDb
from agno.sandbox.docker import DockerSandbox
from agno.sandbox.s3 import S3CheckpointStore
 
checkpoints = S3CheckpointStore("my-agno-checkpoints", prefix="sandbox/checkpoints")
 
agent = ClaudeAgent(
    id="coder",
    db=PostgresDb(db_url="postgresql+psycopg://ai:ai@localhost:5532/ai"),
    sandbox=DockerSandbox("agno-sandbox:local", checkpoints=checkpoints, network="isolated"),
)

put hashes the archive first and skips the upload when that object already exists. Archives above part_size (64 MiB by default) go up in parts, and a failed multipart upload is aborted. The archive limit of 1 GiB still applies. On the way back, open_archive downloads to a temporary file and hashes the bytes before handing over the path.

We ran the store against moto's local S3 server, with two store instances playing two replicas that share nothing but the bucket. Replica A captured a small workspace and replica B restored it:

put -> CheckpointRef(sha256='0a9686a94759f8f9dfbb5ad10bb46dc0a942e7a8b5be962cd760967faee18937', size=406)
put again -> True
object: sandbox/checkpoints/0a9686a94759f8f9dfbb5ad10bb46dc0a942e7a8b5be962cd760967faee18937.tar.gz 406 bytes {'sha256': '0a9686a94759f8f9dfbb5ad10bb46dc0a942e7a8b5be962cd760967faee18937', 'size': '406'}
restore on B -> {'turn': 1}
restored hello.py: print("hello")

Damaging the object three ways made replica B's verify fail each time:

# same size, different bytes, no digest in the metadata
verify -> SandboxError: Checkpoint checksum mismatch
# first 100 bytes only, digest metadata kept
verify -> SandboxError: Checkpoint size mismatch
# object deleted
verify -> CheckpointObjectNotFound: Checkpoint object is missing

Credentials come from the AWS environment or the deployment's role, so no secret goes into your code. For MinIO, R2 or a local test server, set endpoint_url.

See the cookbook, and learn more about deploying Claude and Codex services in the documentation.

Frequently asked questions

Yes. Pass the service's address as endpoint_url. Leave it unset for AWS S3. A local test server works the same way.

From boto3's standard resolution: AWS_PROFILE, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or the instance role. The constructor takes no keys. You can also hand it a boto_session.

Nothing readable is left behind. put returns only after a HEAD request shows the object with the expected size and digest, and a failed multipart upload is aborted so its parts are not kept.

s3:GetObject, s3:PutObject and s3:DeleteObject on the prefix, plus s3:AbortMultipartUpload for archives larger than part_size. Without s3:ListBucket, S3 answers a missing key with 403, which verify reports as an error.

Shipped around the same time