Enterprise AI in Numbers · Risk and regulation
Top websites that block AI crawlers in robots.txt
394
394 as of October 19, 2026, according to Cloudflare Radar, AI Insights. Number of scanned domains whose robots.txt disallows OpenAI's GPTBot on every path; parts give the other major AI crawlers.
What it means for leaders
Many leading publishers block AI training crawlers while allowing search crawlers. Make a deliberate choice for your own sites, distinguishing training, search and agent access, rather than inheriting a default.
What would change the reading: Blocks shifting from AI search crawlers to training-only crawlers.
The breakdown
Every reading
| Date | Value | Change | Source |
|---|---|---|---|
| 394 | — | Source |
Definition and method
- What it measures
- Number of scanned domains whose robots.txt disallows OpenAI's GPTBot on every path; parts give the other major AI crawlers.
- Population
- Robots.txt files of the top domains Cloudflare Radar scans each week
- Updated
- Weekly
- Source
- Cloudflare Radar, AI Insights · CC BY-NC 4.0
Caveats
- Robots.txt is a request, not an enforcement: some crawlers ignore it.
- Counts are of domains in Radar's weekly scan of top sites.
Cite this
Enterprise AI in Numbers, "Top websites that block AI crawlers in robots.txt", https://lucianoayres.com/ai-brief/data/sites-blocking-ai-crawlers/, retrieved October 25, 2026. Data: Cloudflare Radar, AI Insights.