Your agent's own best runs are the training data

New in Agno: a straight path from evaluation to fine-tuning data, built from a few pieces that snap together. agno.scorer grades attempts, agno.environments runs an agent over a set of tasks in full isolation, and run_rollouts runs each task K times so you can see how often the agent actually succeeds instead of whether it managed it once.

Running a task once and watching it pass tells you almost nothing. Run it eight times and you get a real pass rate, which is what separates reliable behavior from a lucky draw. That number is pass@k, and you can now measure it directly instead of building a harness for it.

from agno.environments import Environment, Task, run_rollouts

env = Environment(
    agent=agent,
    tasks=[Task(input="...", scorer=my_scorer)],
)

results = run_rollouts(env, k=8)         # every task, eight isolated attempts
print(results.pass_rate)                 # how often it actually worked
results.to_sft_jsonl("training.jsonl")   # export the attempts that passed

‍

From there it turns into a training tool. to_sft_jsonl takes the attempts that passed and writes them out as conversational SFT data, with a provenance sidecar so you can trace every example back to the exact run it came from. You point a strong agent at a task, let it try many times, keep what worked, and hand that straight to fine-tuning. It's rejection sampling, minus the pipeline you'd normally have to stand up around it.

The isolation is what makes that exported data worth trusting. Each attempt runs on a fresh db, session, and user, with no memory or learning writes bleeding between runs, so neither your pass rate nor your training set gets contaminated by state left over from an earlier attempt.

And if all you want is scoring inside your existing eval suite, Case.scorer drops any scorer into a Case and checks it exactly, with no extra model call.

See the environments cookbooks for reference.