Web Search Reader (Async)

The Web Search Reader searches and reads web search results, converting them into vector embeddings for your knowledge base.

Code

examples/concepts/knowledge/readers/web_search_reader_async.py

import asyncio

from agno.agent import Agent
from agno.db.postgres.postgres import PostgresDb
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.web_search_reader import WebSearchReader
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

db = PostgresDb(id="web-search-db", db_url=db_url)

vector_db = PgVector(
    db_url=db_url,
    table_name="web_search_documents",
)
knowledge = Knowledge(
    name="Web Search Documents",
    contents_db=db,
    vector_db=vector_db,
)


# Initialize the Agent with the knowledge
agent = Agent(
    knowledge=knowledge,
    search_knowledge=True,
)


if __name__ == "__main__":
    # Comment out after first run
    asyncio.run(
        knowledge.add_content_async(
            topics=["web3 latest trends 2025"],
            reader=WebSearchReader(
                max_results=3,
                search_engine="duckduckgo",
                chunk=True,
            ),
        )
    )

    # Create and use the agent
    asyncio.run(
        agent.aprint_response(
            "What are the latest AI trends according to the search results?",
            markdown=True,
        )
    )

Usage

1. Create a virtual environment

   Open the Terminal and create a python virtual environment.

   Mac

   python3 -m venv .venv
   source .venv/bin/activate
   
2. Install libraries

   pip install -U requests beautifulsoup4 agno openai
   
3. Run PgVector

   docker run -d \
     -e POSTGRES_DB=ai \
     -e POSTGRES_USER=ai \
     -e POSTGRES_PASSWORD=ai \
     -e PGDATA=/var/lib/postgresql/data/pgdata \
     -v pgvolume:/var/lib/postgresql/data \
     -p 5532:5432 \
     --name pgvector \
     agno/pgvector:16
   
4. Set environment variables

   export OPENAI_API_KEY=xxx
   
5. Run Agent

   Mac

   python examples/concepts/knowledge/readers/web_search_reader_async.py
   

Params

Parameter	Type	Default	Description
search_timeout	int	10	Timeout for search operations in seconds
request_timeout	int	30	Timeout for HTTP requests in seconds
delay_between_requests	float	2.0	Delay between requests in seconds
max_retries	int	3	Maximum number of retries for failed requests
user_agent	str	"Mozilla/5.0..."	User agent string for HTTP requests
search_engine	Literal["duckduckgo", "google"]	"duckduckgo"	Search engine to use
search_delay	float	3.0	Delay between search requests in seconds
max_search_retries	int	2	Maximum retries for search operations
rate_limit_delay	float	5.0	Delay when rate limited in seconds
exponential_backoff	bool	True	Whether to use exponential backoff for retries
chunking_strategy	Optional[ChunkingStrategy]	SemanticChunking()	Strategy for chunking content