Scenario Testing

This example demonstrates how to use the Scenario testing library to test an agent with defined success and failure criteria. It shows how to implement automated testing for agent behavior and responses.

Code

scenario_testing.py

"""
This is an example that uses the [scenario](https://github.com/langwatch/scenario) testing library to test an agent.

Prerequisites:
- Install scenario: `pip install scenario`
"""

import pytest
from scenario import Scenario, TestingAgent, scenario_cache

Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-5-nano"))


@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent():
    agent = VegetarianRecipeAgent()

    def vegetarian_recipe_agent(message, context):
        # Call your agent here
        return agent.run(message)

    # Define the scenario
    scenario = Scenario(
        "User is looking for a dinner idea",
        agent=vegetarian_recipe_agent,
        success_criteria=[
            "Recipe agent generates a vegetarian recipe",
            "Recipe includes a list of ingredients",
            "Recipe includes step-by-step cooking instructions",
        ],
        failure_criteria=[
            "The recipe is not vegetarian or includes meat",
            "The agent asks more than two follow-up questions",
        ],
    )

    # Run the scenario and get results
    result = await scenario.run()

    # Assert for pytest to know whether the test passed
    assert result.success


# Example agent implementation
from agno.agent import Agent  # noqa: E402
from agno.models.openai import OpenAIChat  # noqa: E402


class VegetarianRecipeAgent:
    def __init__(self):
        self.history = []

    @scenario_cache()
    def run(self, message: str):
        self.history.append({"role": "user", "content": message})

        agent = Agent(
            model=OpenAIChat(id="gpt-5-mini"),
            markdown=True,
            instructions="You are a vegetarian recipe agent",
        )

        response = agent.run(message)
        result = response.content
        print(result)
        self.history.append(result)

        return {"message": result}

Usage

1. Create a virtual environment

   Open the Terminal and create a python virtual environment.

   Mac

   python3 -m venv .venv
   source .venv/bin/activate
   
2. Install libraries

   pip install -U agno openai scenario pytest
   
3. Export your OpenAI API key

   Mac/Linux

     export OPENAI_API_KEY="your_openai_api_key_here"
   
4. Create a Python file

   Create a Python file and add the above code.

   touch scenario_testing.py
   
5. Run Test

   Mac

   pytest scenario_testing.py -v
   
6. Find All Cookbooks

   Explore all the available cookbooks in the Agno repository. Click the link below to view the code on GitHub:

   Agno Cookbooks on GitHub