Grade multi-turn conversations in eval suites

An Agno eval Case now takes session_id, user_id, session_state, dependencies, metadata and knowledge_filters, hands them to agent.arun or team.arun, and copies session_state and dependencies for each run so a case you reuse starts from the same values every time. Cases that share a session_id run as consecutive turns, and a later turn reads the earlier ones when the agent has a db and add_history_to_context=True.

A booking agent that reads its calendar from session_state and greets a returning user can now be graded on the call it really receives:

from agno.eval import Case
 
Case(
    name="offers_open_slots",
    agent=agent,
    input="Hi, I'm Priya. What times do you have?",
    criteria="Offers Tuesday 10:00 and Wednesday 15:00.",
    expected_tool_calls=("get_open_slots",),
    user_id="lead-42",
    session_state={"calendar_id": "cal-1", "timezone": "UTC"},
)

Pass a Message as input to attach an image, audio, video or file to the graded turn. The judge reads the message text and the reply, so a criterion should name what the reply says, such as "Names the Golden Gate Bridge".

We ran two cases on one session_id against an agent with SqliteDb and history turned on. A stub model stood in for a real one and replied with every user turn it could see, and a small scorer checked the reply for the name:

# this release
turn_1  session=lead-42-session  passed=True
  output: user turns I can see: ["Hi, I'm Priya."]
turn_2  session=lead-42-session  passed=True
  output: user turns I can see: ["Hi, I'm Priya.", 'What was my name?']
 
# earlier releases
TypeError: Case.__init__() got an unexpected keyword argument 'session_id'

Every new field defaults to None, so a case that sets none of them runs as it always did. The suite grades only the case's own turn, and it never clears a session, so a case with a fixed session_id on a lasting database should delete it in teardown.

See the cookbook, and learn more about eval suites in the documentation.