Getting Started with llms.txt

Learn how to implement llms.txt in your website or documentation

What is llms.txt?

The llms.txt file is a proposed standard that helps AI models better understand and interact with your website's content. Unlike robots.txt or sitemap.xml, it's specifically designed to enhance AI interactions by providing structured content overviews and navigation paths.

Purpose and Benefits

• Enhanced AI Understanding: Provides a structured overview of your website content, helping AI systems understand your site quickly and accurately
• Efficient Information Retrieval: Enables AI systems to locate and retrieve relevant information efficiently
• Improved Contextual Understanding: Reduces misinterpretation by providing context and relationships between content
• Better User Experience: Leads to more accurate, context-aware responses when users interact with AI about your content
• Overcoming Context Limitations: Helps AI systems work within their context window limitations by providing streamlined content access

How llms.txt Differs from robots.txt and sitemap.xml

While these files might seem similar, they serve distinct purposes:

• robots.txt: Focuses on controlling search engine crawler access but doesn't help with content understanding
• sitemap.xml: Lists all indexable pages without providing context or helping with content processing
• llms.txt: Specifically addresses AI-related challenges by providing content structure in a format optimized for AI processing

The llms.txt Specification

The specification defines two distinct files:

• /llms.txt: A streamlined view of documentation navigation to help AI systems quickly understand a site's structure
• /llms-full.txt: A comprehensive file containing all documentation in one place

Both files use Markdown formatting, which provides a natural hierarchy that AI models can easily parse.

Basic Implementation Guide

1. Create the File Structure

The llms.txt file uses Markdown with a specific structure:

# Your Website/Project Name

> A brief description of your website or project

## Documentation

- [Getting Started](/docs/getting-started) - Guide for new users
- [API Reference](/docs/api) - Complete API documentation
- [Tutorials](/docs/tutorials) - Step-by-step guides

## Examples

- [Basic Implementation](/examples/basic) - Simple integration example
- [Advanced Features](/examples/advanced) - Using advanced capabilities

## Optional Resources

- [Community Forum](/community) - Get help from other users
- [Change Log](/changelog) - Track updates and changes

2. Place the File in the Correct Location

Save the file as llms.txt in your website's root directory, ensuring it's accessible at yourwebsite.com/llms.txt. If you're creating a comprehensive version with all your documentation, save it as llms-full.txt.

3. Add HTTP Headers (Optional but Recommended)

Add the following HTTP header to your server configuration:

X-Robots-Tag: llms-txt

4. Verify Implementation

Test your implementation by:

1. Accessing yourwebsite.com/llms.txt
2. Checking HTTP headers
3. Validating the file format

Using llms.txt with AI Systems

Currently, most AI models don't automatically discover and index llms.txt files. To use your llms.txt file with AI systems:

1. Direct Link: Provide the AI with a link to your llms.txt file
2. Manual Copy: Copy the contents of your llms.txt file directly into your prompt
3. File Upload: Use the AI tool's file upload feature if available

As adoption increases, more AI systems will likely integrate automatic discovery of llms.txt files.

Where To Find a list of all llms.txt files?

llms.txt hub is rapidly becoming a popular open-source hub tracking all the websites that have implemented the llms.txt and llms-full.txt standards. Updated regularly, this comprehensive directory allows you to:

• Discover real-world implementations across various industries and platforms
• Study how leading organizations structure their llms.txt files
• Access the latest news and developments related to the standard
• Find open-source tools, plugins, and resources for implementing llms.txt on your own website

By exploring this directory, developers and content creators can observe implementation patterns, stay informed about best practices, and connect with the growing community of websites optimizing their content for AI systems through the llms.txt standard.

Generation Tools

Several tools can help you generate llms.txt files:

• llmstxt by dotenv: Open source CLI tool that generates llms.txt based on a site's sitemap.xml file.
• llmstxt by Firecrawl: Uses Firecrawl to generate a llms.txt file.
• Mintlify: Documentation platform with llms.txt generation

Real-World Examples

Many organizations have already adopted the llms.txt proposed standard:

• Cloudflare: Impressive and extensive documentation about all the Cloudflare services
• Anthropic's llms.txt: All their documentation and prompt library
• Perplexity's llms-full.txt: A comprehensive implementation all their documentation and content website
• ElevenLabs: All the documenation about their API and product guides

Best Practices

1. Keep it Updated: Regularly update your llms.txt file as your website structure changes to ensure AI systems have the most current information.

2. Use Clear Markdown Structure:

   • Start with an H1 project name
   • Include a blockquote summary
   • Use H2 headers to organize documentation links
   • Provide brief descriptions for each link

3. Be Selective: Focus on the most important resources in llms.txt, using the Optional section for less critical content.

4. Test with AI Systems: Verify that AI models correctly interpret and navigate your content based on your llms.txt file.

5. Optimize for AI Processing: Remove non-essential markup and scripts in llms-full.txt to help AI models focus on the important content.

Resources

• llms.txt Standard Proposal
• GitHub Repository
• Community Forum