LLMs.txt Reader

The LLMs.txt Reader reads an llms.txt file, follows the linked documentation pages, and turns them into documents for your knowledge base.

Code

examples/basics/knowledge/concepts/readers/overview/llms_txt_reader.py

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    name="LLMs.txt Docs",
    vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)

knowledge.insert(
    url="https://docs.agno.com/llms.txt",
    reader=LLMsTxtReader(max_urls=10),
)

agent = Agent(
    knowledge=knowledge,
    search_knowledge=True,
)

agent.print_response("What is Agno?", markdown=True)

Usage

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U beautifulsoup4 sqlalchemy psycopg pgvector agno openai
   
3. Set environment variables

   export OPENAI_API_KEY=xxx
   
4. Run PgVector

   docker run -d \
     -e POSTGRES_DB=ai \
     -e POSTGRES_USER=ai \
     -e POSTGRES_PASSWORD=ai \
     -e PGDATA=/var/lib/postgresql/data/pgdata \
     -v pgvolume:/var/lib/postgresql/data \
     -p 5532:5432 \
     --name pgvector \
     agno/pgvector:16
   
5. Run Agent

   python examples/basics/knowledge/concepts/readers/overview/llms_txt_reader.py
   

Params

Parameter	Type	Default	Description
url	str	Required	URL of the llms.txt file to read
max_urls	int	20	Maximum number of linked URLs to fetch from the file
timeout	int	60	HTTP timeout in seconds
proxy	Optional[str]	None	Optional HTTP proxy URL
skip_optional	bool	False	Skip entries under the ## Optional section
chunking_strategy	Optional[ChunkingStrategy]	FixedSizeChunking()	Strategy for chunking content
allowed_hosts	Optional[List[str]]	None	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.