Sandbox sessions clean up after themselves and stop cleanly when a checkpoint is lost

Deleting a sandboxed Agno session used to leave its warm container and checkpoint objects behind, because the delete also dropped the session's sandbox bindings row, the only pointer to both. PostgresDb.delete_session and delete_sessions now destroy the container and delete the checkpoint objects once the delete commits.

All the output below is real, from runs on Docker and an embedded Postgres. The agent is Agno's FixtureAgent, a credential-free runtime that echoes its input, standing in for Claude or Codex.

Deleting a session cleans up its sandbox

Two turns, then a delete scoped to another user, then a delete by the owner:

# Earlier releases
after two turns: containers 1 checkpoints stored [True, True]
delete_session as another user -> False
  containers 1 checkpoints stored [True, True]
delete_session as owner -> True
  containers 1 checkpoints stored [True, True]
 
# This release
after two turns: containers 1 checkpoints stored [True, True]
delete_session as another user -> False
  containers 1 checkpoints stored [True, True]
delete_session as owner -> True
  containers 0 checkpoints stored [False, False]

The other user's delete removed no session, so it touched nothing in the sandbox. Cleanup covers single and bulk deletes on PostgresDb and AsyncPostgresDb, and reaches every checkpoint the session's runs reference. It relies on a coordinator registered on the db, which AgentOS builds at startup and a script builds on its first turn.

AgentOS checks the checkpoint store at startup

An AgentOS with QueueConfig(durable=True) now raises while it is constructed if a sandboxed agent has no checkpoints= store. Earlier releases started up and failed the first request:

# Earlier releases: AgentOS starts, then the first POST fails
POST /agents/engineer/runs -> 500 {"detail":"Run acceptance aborted: the run row could not be prepared (ValueError); the queued job was cancelled and will not execute. Retry the submission."}
 
# This release (home directory shortened to ~)
AgentOS(...) raised ValueError: Sandbox checkpoints default to ~/.agno/sandbox/checkpoints, which other queue replicas cannot read and which does not survive losing this host; pass checkpoints= to the sandbox

AgentOS accepts any store you pass explicitly as your word that every replica can read it, so use PostgresCheckpointStore or S3CheckpointStore.

A missing checkpoint ends the turn

Retention, a replica on the wrong store or a deleted bucket can all leave a session pointing at a checkpoint that no longer exists. To reproduce it, we ran one turn, removed its warm container, deleted the committed checkpoint object and ran two more turns:

# Earlier releases
turn 2: raised CheckpointObjectNotFound: Checkpoint object is missing
turn 3: raised SandboxSessionBusy: A sandbox session already has an active run; wait or reconcile that run
  run row: COMPLETED
  run row: RUNNING
 
# This release
turn 2: status=ERROR retryable=False
  content: This session's sandbox checkpoint is missing from checkpoint storage, so its workspace and conversation cannot be restored; start a new session, and check that every replica uses the same checkpoints= store
turn 3: status=ERROR retryable=False
  run row: COMPLETED
  run row: ERROR
  run row: ERROR

The coordinator now stores the failed turn as an ERROR that the queue won't retry, retires the computer it created for the restore, and releases the session. Other store errors, such as an S3 outage, still raise so a retry can recover.

Each session keeps its three newest checkpoints

Sandboxes take retain_checkpoints, which defaults to 3, on DockerSandbox, AgentCoreSandbox and the Sandbox base class. After a turn's outcome is stored, the coordinator deletes older checkpoints and any staged checkpoint that was never published, and retries a failed delete after the next turn. Five turns on one session:

# Earlier releases
checkpoints of turns 1-5 still stored: [True, True, True, True, True]
open_run_checkpoint(turn 1) -> 620 bytes
 
# This release
checkpoints of turns 1-5 still stored: [False, False, True, True, True]
CheckpointObjectNotFound Checkpoint object is missing

Pass retain_checkpoints=None to keep every checkpoint, as earlier releases did.

See the cookbook, and learn more about sandboxes and checkpoints in the documentation.