What Is llms.txt and Does Your Website Need One in 2026?
Abhishek Dwivedi
Team Lead, SEO
In September 2024, Jeremy Howard, the co-founder of fast.ai and Answer.AI, proposed a new file format called llms.txt. The idea was simple. Large language models struggle to process full HTML pages the way a browser does, so give them a clean, structured summary instead. Two years on, adoption has grown across SaaS companies, documentation sites, and content-heavy publishers, even though no major AI lab has confirmed it officially crawls the file yet. If you're building a GEO or AEO strategy for your brand, it's worth understanding exactly what this file does before you add one.
What Is llms.txt, in Plain Terms?
llms.txt is a markdown file you place at the root of your domain, at yourdomain.com/llms.txt. It lists your most important pages in plain language, grouped under simple headers, with a one-line description of what each page covers. Some sites also publish an llms-full.txt, which includes the entire text content of key pages in one file so a model can ingest it without crawling the live site page by page.
Where Did This Idea Come From, and Is It an Official Standard?
It isn't an official web standard the way robots.txt is, and it hasn't been ratified by the W3C or adopted as documented policy by OpenAI, Anthropic, or Google. It started as a proposal on llmstxt.org and spread the way a lot of useful web conventions do: through voluntary adoption. Companies like Anthropic, Mintlify, and Cursor have published their own llms.txt files. That's a meaningful adoption signal even without formal backing, but it also means you shouldn't treat it as a guaranteed ranking or citation lever.
How Is llms.txt Different From robots.txt and sitemap.xml?
robots.txt is a permissions file. It tells crawlers what they're allowed and not allowed to fetch, and it says nothing about content quality or importance. sitemap.xml is a complete index, often thousands of URLs long, meant for exhaustive discovery. llms.txt does a third, different job. It's a short, curated, human-readable brief pointing to maybe 10 to 30 pages that matter most, written in a format a language model can parse without rendering JavaScript or navigating a menu. None of the three files replace each other.
Does Adding llms.txt Actually Improve AI Citations?
Here's the honest answer: the evidence right now is mixed and mostly anecdotal. Some documentation-heavy sites have reported showing up more often in AI-generated answers after publishing one, but there's no independent, large-scale study confirming causation as of this writing. What we do know is that clean, well-structured content with clear headings and direct answers tends to get cited more often in tools like ChatGPT, Perplexity, and Google's AI Overviews, and llms.txt is one more way to present that structure. This mirrors a broader shift we've covered before, where content graphs are replacing social graphs as the backbone of discovery. Treat it as a small, low-cost addition to a broader content strategy, not a replacement for one.
Which Websites Get the Most Value From It?
Sites with deep, evergreen content libraries benefit the most: SaaS products with documentation, EdTech platforms with course catalogs (our complete guide to EdTech SEO covers this in more depth), agencies with detailed service pages and guides, and publishers with large archives. If your site is a five-page brochure site with no blog or documentation, an llms.txt file won't do much for you. Fix your on-page content and technical SEO first. Structure follows substance, not the other way around.
How Do You Create Your First llms.txt File?
Start by listing the 10 to 20 pages you'd want an AI assistant to read if it could only read a handful. Group them under markdown H2 headers such as "Docs," "Guides," "Product," or "Case Studies." Write one plain sentence under each link describing what it covers and who it's for; skip marketing language and get straight to the point. Save the file as llms.txt and upload it to your site's root directory so it's reachable at yourdomain.com/llms.txt. Once it's live, keep an eye on referral traffic from ChatGPT, Perplexity, and Gemini inside GA4 over the following month to see if anything shifts.
What Should You Do Next?
Don't treat llms.txt as a checkbox you tick once and forget. Update it whenever you publish major new guides or retire outdated pages, the same way you'd maintain a sitemap. Pair it with solid technical SEO, clear heading structure, and content that answers real questions directly near the top of the page. That combination, not any single file, is what actually improves your odds of being read and cited accurately by AI systems in 2026.
Frequently Asked Questions
Do I need to remove or change my robots.txt file if I add llms.txt?
No. llms.txt and robots.txt serve different purposes and work together. robots.txt controls what crawlers can access, while llms.txt highlights your most useful pages for AI models to read. Keep both files active and update them independently as your site changes.
Will Google use my llms.txt file for search rankings?
Google has not confirmed that it reads or ranks pages based on llms.txt. It is aimed at AI assistants and chat tools rather than traditional search crawlers. Adding one is unlikely to hurt your SEO, but treat it as a supplement to strong technical SEO, not a ranking factor on its own.
How long should an llms.txt file be?
Most effective llms.txt files list 10 to 30 links, each with a one-line description. Keep it focused on your most valuable pages, such as core service pages, documentation, or flagship guides, rather than trying to include your entire site.
Can I automate the creation of an llms.txt file?
Yes. Several open-source scripts and CMS plugins can generate an llms.txt file automatically from your sitemap or content management system. For small sites, writing it by hand often produces a more useful, curated result since you can choose which pages truly matter.
Does WordPress or Shopify support llms.txt out of the box?
Not natively as of 2026. You will need to manually upload an llms.txt file to your site's root directory, or use a plugin or app built for this purpose. Check with your hosting provider or developer to confirm root-level file access.
What happens if I don't create an llms.txt file?
Nothing breaks. Your site will continue to be crawled and indexed normally by search engines and AI tools. You may simply miss a small, low-cost opportunity to guide AI assistants toward your best content, so it's worth doing once your core SEO and content are solid.
Like what you're reading?
Get a free marketing plan tailored to your brand
Get Your Free Planyes, actually free. we're not kidding.
Key Takeaways
- llms.txt is a curated, plain-language map of your best pages for AI models, not a crawler permissions file like robots.txt.
- It was proposed by Jeremy Howard of Answer.AI in September 2024 and has voluntary adoption, but it isn't an official web standard yet.
- No major AI lab has publicly confirmed it crawls the file, so results are a signal, not a guarantee.
- It works best for content-rich sites like SaaS, EdTech, agencies, and publishers with real documentation or guides.
- Fix technical SEO and content quality first; llms.txt supplements a strong content strategy, it doesn't replace one.
- Update the file whenever your key pages change, and track AI-referral traffic in GA4 to see if it's making a difference.
the TL;DR your boss will love
This Isn't Just Theory. We Do This Daily.
Let us show you exactly what we'd do for your brand - strategy, channels, budget, timeline. Free. No pitch deck.
Get Smarter Every Week
Ideas, data, and the occasional hot take. Biweekly, no spam.
we promise not to sell your email to robots
You Might Also Dig These
^ we picked these just for you (okay, the algorithm did)
Why 90% of Indian D2C Brands Fail at Performance Marketing
We have audited 120+ D2C ad accounts over the last two years. The pattern is eerily consistent - brands burn through their seed funding on Meta and Google Ads, hit a plateau around ₹15-20L monthly spend, and then blame the platform. The problem is almost never the platform. It is five fundamental mistakes that keep repeating across Indian D2C.
ReadThe Complete Guide to SEO for EdTech Companies in 2026
Google AI Overviews have reshaped the EdTech search landscape overnight. Our data across 14 EdTech clients shows that 68% of education-related informational queries now trigger an AI Overview, and organic click-through rates on those queries have dropped by 23% year-over-year. If your SEO strategy still revolves around ranking for "best data science course" with a listicle, you are fighting a losing battle. Here is what works in 2026.
ReadHow We Built CopyLoop - Our AI Ad Copy Tool
In early 2024, our copywriters were spending 60% of their time writing variations of the same ad - different hooks, different CTAs, different lengths for different placements. Good copywriters should spend their time on strategy and big ideas, not grinding out the 47th variation of a Google RSA headline. That frustration is what led us to build CopyLoop, and 18 months later, it has generated over 10,000 ads with a 23% higher CTR than our human-written baseline. Here is the full story of how we built it.
Read