Free toolNo account needed
Can AI agents see your website?
Test your robots.txt against 16 AI crawlers and assistants, and check for an llms.txt.
The 16 AI agents we check
| Agent | Company | Type |
|---|---|---|
| GPTBot | OpenAIAI training | AI training |
| OAI-SearchBot | OpenAIAI search | AI search |
| ChatGPT-User | OpenAIOn request | On request |
| ClaudeBot | AnthropicAI training | AI training |
| Claude-SearchBot | AnthropicAI search | AI search |
| Claude-User | AnthropicOn request | On request |
| PerplexityBot | PerplexityAI search | AI search |
| Perplexity-User | PerplexityOn request | On request |
| Google-Extended | GoogleAI training | AI training |
| Applebot-Extended | AppleAI training | AI training |
| Meta-ExternalAgent | MetaAI training | AI training |
| Amazonbot | AmazonAI training | AI training |
| CCBot | Common CrawlAI training | AI training |
| Bytespider | ByteDanceAI training | AI training |
| DuckAssistBot | DuckDuckGoAI search | AI search |
| MistralAI-User | Mistral AIOn request | On request |
AI training: collects pages for AI models. AI search: indexes pages for AI answers. On request: fetches a page when a person asks an assistant. Each token and what it does was checked against its company's own documentation on .
How the check works
Read robots.txt
We fetch robots.txt from your site once, as PlainrivalBot, and nothing else from your pages.
Test each agent
Each agent gets the rules of its own group, or the * group when none names it. The longest matching rule wins, as in RFC 9309.
Look for llms.txt
We check whether /llms.txt exists: a short Markdown file that tells AI assistants what your site is and where its pages are.
Plainrival reads your competitors' public pages the same polite way, every day. See website change tracking, pricing tracking and a sample weekly brief.
Questions
What does blocking an AI crawler change?
It asks that company not to fetch your pages for that purpose. Blocking a training crawler like GPTBot keeps your pages out of future model training, but not out of AI search; the search bots, like OAI-SearchBot, have their own tokens.
Why are Google-Extended and Applebot-Extended on the list?
They aren't crawlers. Google and Apple read these tokens to decide whether pages their normal crawlers fetch may be used for AI. Blocking them doesn't remove you from Google or Apple search.
Do assistants follow robots.txt when a person asks for a page?
Not always. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. The result shows what your robots.txt asks for.
What is llms.txt?
A Markdown file at /llms.txt that sums up your site for AI assistants and links to your main pages. It's a proposal from llmstxt.org, not a standard, and it doesn't block anything.
What if my site has no robots.txt?
Then every agent may read everything. A robots.txt that returns a server error is the opposite: crawlers treat the whole site as blocked until it works.
Do you keep the address I check?
We don't store results. Our server log records the host name, to stop abuse. Our analytics count checks without the address.
Now see what your competitors change.
Plainrival checks their websites every day and sends you a weekly brief. Start free, no card needed.