State of agent access, October 2026
We asked the 5,000 most-linked software sites what an AI agent meets on the way in. 602 stopped our agent at the home page. 204 of 1,153 sign-up pages carry a CAPTCHA.
The numbers
Each counts the sites we could ask, out of 5,000.
602 of 5,000 stop an agent at the home page.
Their first answer to our agent was a bot check (372) or a refusal (230). Cloudflare's challenge was 321 of the 372.
Our survey of 5,000 sites, 12–13 Oct 2026 · The data
Data table
| The home page | A bot check or a refusal | No answer or no page | Total | |
|---|---|---|---|---|
| What our agent got at the home page of 5,000 sites | 3,973 | 602 | 425 | 5,000 |
Our survey of 5,000 sites, 12–13 Oct 2026 · The data
Data table
| Sites | Of | |
|---|---|---|
| Consumer services (1,121 sites) | 252 | 1,121 |
| AI (39 sites) | 8 | 39 |
| Security (66 sites) | 7 | 66 |
| Developer and cloud (512 sites) | 53 | 512 |
| Other software (3,018 sites) | 271 | 3,018 |
| Business software (88 sites) | 6 | 88 |
| Collaboration and office (156 sites) | 5 | 156 |
At sign-up, the wall is a CAPTCHA.
Of the 1,153 sign-up pages a home page links to, 204 carry a CAPTCHA, reCAPTCHA in 160 of them. A bot check in front of the sign-up page itself was rare: 43 of 1,598.
Our survey of 5,000 sites, 12–13 Oct 2026 · The data
Data table
| Sign-up pages | Of | |
|---|---|---|
| A CAPTCHA in the sign-up page | 204 | 1,153 |
| robots.txt closes the sign-up page to our agent | 136 | 1,445 |
| A bot check or a refusal at the sign-up page | 43 | 1,598 |
264 robots.txt files shut out AI assistants.
Of the 3,689 robots.txt files we read, 863 name an AI agent or crawler and 492 shut at least one out of the whole site. An assistant or agent is what a person sends to open your page for them, so a rule against it turns that person away too.
Our survey of 5,000 sites, 12–13 Oct 2026 · The data
Data table
| robots.txt files | Of | |
|---|---|---|
| Name an AI agent or crawler | 863 | 3,689 |
| Shut at least one AI agent or crawler out of the whole site | 492 | 3,689 |
| Shut an AI assistant or agent out of the whole site | 264 | 3,689 |
| Carry a Content-Signal line | 149 | 3,689 |
| Shut out every bot they don't name | 91 | 3,689 |
Few sites publish what an agent reads.
859 of 3,973 sites serve an llms.txt. 158 of 5,000 offer an MCP server and 82 an OpenAPI document where we looked.
Our survey of 5,000 sites, 12–13 Oct 2026 · The data
Data table
| Sites | Of | |
|---|---|---|
| An llms.txt, on the site or its docs | 859 | 3,973 |
| An llms-full.txt | 208 | 3,973 |
| An MCP server (in the official registry, or a server card) | 158 | 5,000 |
| An OpenAPI document at the usual paths | 82 | 3,973 |
| An A2A agent card | 13 | 3,973 |
The 329 products we drive or plan to drive publish these far more often, and more of them put a CAPTCHA on the sign-up page.
| Top 5,000 | Our 329 | |
|---|---|---|
| llms.txt | 22% 859 of 3,973 | 72% 226 of 314 |
| MCP server | 3% 158 of 5,000 | 30% 100 of 329 |
| OpenAPI | 2% 82 of 3,973 | 21% 67 of 314 |
| Bot check at the home page | 12% 602 of 5,000 | 3% 10 of 329 |
| CAPTCHA at sign-up | 18% 204 of 1,153 | 21% 45 of 211 |
| robots.txt shuts out AI assistants | 7% 264 of 3,689 | 2% 6 of 315 |
No site asked our agent to pay.
We called one documented API operation at 57 sites: none answered 402 Payment Required. 1 of 82 API documents describes paid calls, and 18 sites are listed on mpp.dev.
Fix what stops agents on your site.
One guide per wall, with the fix in steps and a check you can run yourself.
- Bot checkLet signed AI agents past your bot checkThe setting for Cloudflare, Vercel, Akamai, DataDome and HUMAN, and what to do about a check on the form.Read the fix 602 of 5,000stopped our agent at the home page
- Browser-only sign-upLet AI agents sign up and get an API keyAn API call that signs up and returns a limited key, and a claim step for the person.Read the fix 204 of 1,153sign-up pages with a CAPTCHA
- DocsDocs AI agents can read: llms.txt, Markdown, OpenAPIAn llms.txt, Markdown twins of your pages, an OpenAPI spec and an MCP server card.Read the fix 3,114 of 3,973sites with no llms.txt
- Card-only checkoutLet AI agents pay you per call: x402 and MPPx402 or MPP on a paid endpoint, your prices where agents read them, and plans an agent can buy by API or MCP.Read the fix 57 of 57API calls that didn't offer a 402 payment
- Read your robots.txt as an agent would.
- Open /robots.txt on your site and find each User-agent line that names an AI assistant or agent.
- A Disallow: / under it shuts that agent out of every page, sign-up and docs included.
- If you want assistants to open pages for your users, allow them, at least on your docs and sign-up pages.
How sure we are.
These numbers are what an AI agent meets when it runs on a server and reads pages without a browser, like ours.
We checked our reading. On 13 Oct 2026 we drew 100 at random from the 606 sites that first stopped our agent, asked them again and read every answer by eye. 97 of 100 were a real bot check or refusal. 2 were the site's own page that we had read wrong, and 1 let our agent in this time. We fixed the reading and asked all 606 again: 602 still stop our agent, the number on this page. All 50 we drew from the sites that let our agent in served their home page again.
Signed agents that Cloudflare knows get in more often. 71 of 92 sites that stopped our agent let Cloudflare's own signed agent in. Let signed agents past your bot check, and sign your own agent.
Details
- Each of the 150 sites got one more request the same way, and one through Cloudflare Browser Rendering with JavaScript off. The draw used a fixed seed.
- The second look also counts 49 sites as refusals that we had first counted as challenges.
- An agent that works in a person's own browser, on their home connection, is not part of these numbers.
How we asked.
What. The 5,000 most-linked sites in the Majestic Million that Wikidata names as the website of software, an online service or a software company. On each: robots.txt, the home page, llms.txt, the MCP and A2A files under /.well-known/, OpenAPI at the usual paths, the sign-up page the home page links to, and one documented API call. Also the official MCP registry and mpp.dev.
How politely. GET requests only, from our server in Germany, signed as our agent. robots.txt obeyed for every URL, one request every 3 seconds per site. A site that answered with a bot check, a refusal or a 429 was asked nothing more there.
What we didn't do. Sign up, send a form, solve a CAPTCHA or open a browser.
Details
- Each number counts the sites we could ask. A bot check at the home page ended every other question there, a robots.txt we couldn't read has no robots.txt numbers, and a sign-up number needs a sign-up page we found.
- A request from a server doesn't see a check that runs in the browser after the page loads, a CAPTCHA a script adds, files at other paths, or an MCP server outside the registry without a server card. What sites publish and the checks they run are at least these numbers.
- The CAPTCHA numbers count the sign-up pages a home page links to. Where it linked none we tried /signup, and some of those pages are the home page again. Counting every sign-up page that loaded, 241 of 1,555 carry a CAPTCHA.
- 252 sites answered a check partway, after the home page loaded. What we hadn't asked them by then counts as not there.
- AI assistants and agents are the names ai.robots.txt lists as assistants or agents (181 names in all, read during the survey); the rest are crawlers.
- Kinds of site come from Wikidata's types. Our directory is the 329 products we drive or plan to drive, counted apart; 118 of them are also in the top 5,000. The MCP registry was read on 12 Oct 2026.
- Sources: Majestic Million (CC BY 3.0), Wikidata (CC0), ai.robots.txt (MIT).
Use the data.
Every count on this page as JSON, free to reuse with a credit and a link (CC BY 4.0).
Cite it. AgentDrives, State of agent access, October 2026, https://agentdrives.com/research/state-of-agent-access-2026-10
See what an agent meets on yours.
The free check shows what an agent finds on your site and the wall it would likely hit. A Drive ($49) shows how far it gets.