Do AI agents read llms.txt and Markdown?
AI crawlers fetched our Markdown pages 1,428 times in one week and our llms.txt 7 times. AI assistants working for a person asked for Markdown 15 times, and for llms.txt not once. If you want agents to read your docs, serve them as Markdown.
Who fetched what.
4,504 Markdown fetches and 272 llms.txt fetches came from outside in the week. AI crawlers made 1,428 of the Markdown fetches and other named bots and tools 1,825. Unnamed programs made 164 of the llms.txt fetches.
| Who asked | .md pages | Markdown on request | llms.txt | llms-full.txt |
|---|---|---|---|---|
| AI assistants | 0 | 15 | 0 | 0 |
| AI crawlers | 1,428 | 0 | 7 | 5 |
| Search engines | 15 | 0 | 1 | 0 |
| Named bots and tools | 776 | 1,049 | 46 | 2 |
| Unnamed programs | 8 | 0 | 164 | 0 |
| Failed browser checks | 430 | 6 | 40 | 3 |
| Browsers | 775 | 2 | 14 | 2 |
| All | 3,432 | 1,072 | 272 | 12 |
By name.
The programs that read either file most in the week. Browsers aren't listed: their name doesn't say who drives them.
| Program | Markdown | llms.txt | Who it is |
|---|---|---|---|
| ExaSearchBot | 743 | 0 | Named bot |
| QlyzeBot | 625 | 0 | Named bot |
| meta-externalagent | 570 | 4 | AI crawler |
| Amazonbot | 386 | 3 | AI crawler |
| KeenableBot | 244 | 3 | Named bot |
| GPTBot | 222 | 0 | AI crawler |
| ClaudeBot | 190 | 0 | AI crawler |
| node | 0 | 164 | Unnamed program |
| ElevenlabsCrawler | 64 | 1 | Named bot |
| ShapBot | 57 | 0 | Named bot |
| AhrefsBot | 53 | 0 | Named bot |
| GoogleOther | 39 | 2 | AI crawler |
| Reflectionbot | 22 | 2 | Named bot |
| OAI-SearchBot | 21 | 1 | AI crawler |
| AgenstryBot | 0 | 18 | Named bot |
| Googlebot | 15 | 1 | Search engine |
What it means for your docs.
Serve every page as Markdown.
AI crawlers fetched 1,428 of our .md pages in the week, against 7 fetches of llms.txt. Each .md page is linked from its page.
Answer a request for Markdown.
Every one of the 15 Markdown fetches by an AI assistant asked the page's own address for Markdown (Accept: text/markdown). None of them opened llms.txt.
Keep llms.txt, but don't rely on it.
A program named only "node" made 164 of the 272 llms.txt fetches, from 107 clients. Named bots and tools made 46, AI crawlers 7.
Method and limits.
- Every request to agentdrives.com from 19 to 25 Oct 2026, UTC, read from our server's access log: a GET or HEAD of /llms.txt, /llms-full.txt or a .md address, or a page that answered as Markdown because the client asked for it. Only answered requests count, not errors.
- Who asked comes from the user agent and the address. AI crawlers and search engines go by the names their companies publish. Where a company publishes its addresses, we check them.
- AI assistants: assistants and coding agents fetching for a person (ChatGPT-User, Claude-User, Perplexity-User) and programs that name themselves as working for a user.
- Unnamed programs: only an HTTP library's default name, such as node or curl, or none.
- Failed browser checks: a browser's name with an old version, missing browser headers or a data centre's address.
- Our own agent, checks and tests are left out.
- A fetch is one request. One program can fetch the same file many times.
- This is one site, a directory that publishes a Markdown version of every page and links it from the page. A site with other files or links may see other readers.
- A program can claim any name. A name we can't check against its company's addresses counts as given.
See what an agent finds on your site.
Free: what an agent finds on your site, any wall in its way and how many things to fix.