Create the llms.txt for your website:
- Provide the URL of the homepage
- Select what types of links you want to extract (header, nav, footer, or all)
- Click Submit and get your llms.txt in seconds
- Copy the generated content and save it as
llms.txtin your server’s root directory
It’s that simple – and completely free!
Want unlimited crawling and advanced filtering options?
Contact us! We’ll build a Knowledge Graph and generate your llms.txt effortlessly.
Frequently Asked Question
How does the llms.txt generator work behind the scenes?
The generator automatically crawls your website and creates a standardized llms.txt file following the official specification. The process includes:
- Crawling your website pages starting from the homepage
- Extracting key metadata (titles, descriptions, content)
- Categorizing content into sections like Main, Documentation, API, etc.
- Filtering out duplicate and non-essential content
- Generating a properly formatted llms.txt file with the most important information about your site
What if I need to scan more pages?
For larger websites requiring comprehensive scanning, we recommend our enterprise solution that includes:
– Full site ingestion into a Knowledge Graph
– Advanced content categorization and analysis
– Automated llms.txt generation with custom rules
– Regular updates and maintenance
Book a demo with us to learn more about this solution.
What is llms.txt and why is it important?
llms.txt is a proposed web standard (similar to robots.txt) that helps AI systems better understand your website. It provides:
– A standardized way to present key website information to AI models
– A concise, markdown-formatted summary of your site’s most important content
– Curated links to detailed documentation and resources
– Better context for AI tools when they interact with your site
The proposal originated from the need to make websites more AI-friendly, as context windows are too small to handle entire websites and HTML pages are often too complex for efficient AI processing.
A special thanks to Elias Dabbas for supporting the initiative and sharing ideas.