Firecrawl Reader

The Firecrawl Reader uses the Firecrawl API to scrape and crawl web content, converting it into documents for your knowledge base.

Code

examples/concepts/knowledge/readers/firecrawl_reader.py

import os

from agno.knowledge.reader.firecrawl_reader import FirecrawlReader

api_key = os.getenv("FIRECRAWL_API_KEY")

reader = FirecrawlReader(
    api_key=api_key,
    mode="scrape",
    chunk=True,
    # for crawling
    # params={
    #     'limit': 5,
    #     'scrapeOptions': {'formats': ['markdown']}
    # }
    # for scraping
    params={"formats": ["markdown"]},
)

try:
    print("Starting scrape...")
    documents = reader.read("https://github.com/agno-agi/agno")

    if documents:
        for doc in documents:
            print(doc.name)
            print(doc.content)
            print(f"Content length: {len(doc.content)}")
            print("-" * 80)
    else:
        print("No documents were returned")

except Exception as e:
    print(f"Error type: {type(e)}")
    print(f"Error occurred: {str(e)}")

Usage

1. Create a virtual environment

   Open the Terminal and create a python virtual environment.

   Mac

   python3 -m venv .venv
   source .venv/bin/activate
   
2. Install libraries

   pip install -U firecrawl-py agno
   
3. Set API Key

   export FIRECRAWL_API_KEY="your-firecrawl-api-key"
   
4. Run Agent

   Mac

   python examples/concepts/knowledge/readers/firecrawl_reader.py
   

Params

Parameter	Type	Default	Description
api_key	Optional[str]	None	Firecrawl API key for authentication
params	Optional[Dict]	None	Additional parameters to pass to the Firecrawl API
mode	Literal["scrape", "crawl"]	"scrape"	Mode of operation - "scrape" for single page, "crawl" for multiple pages
url	str	Required	URL of the website to scrape or crawl