Can AI see your website?
Free robots.txt checker for AI search
Paste your website. In seconds, see if ChatGPT, Claude, Perplexity, Google and Bing can read it, and how to fix any block.
What it checks
Three kinds of AI bots. Only some should worry you.
AI search bots
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Block one and that AI can’t find or cite your pages.
Live visit bots
ChatGPT-User, Claude-User and Perplexity-User open your page when someone asks about it. Block them and the AI answers without reading it.
AI training bots
GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent and CCBot collect pages to train models. Blocking them is your call. It doesn’t hide you from AI search.
FAQ
Robots.txt and AI, answered.
What is a robots.txt file?
It’s a plain text file at yoursite.com/robots.txt. It tells bots which parts of your site they may read.
Each group starts with a User-agent line naming a bot, then Allow or Disallow lines. “Disallow: /” blocks the whole site.
Does blocking GPTBot hide my site from ChatGPT?
No. GPTBot collects pages to train OpenAI’s models. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when someone asks.
You can block GPTBot and still show up in ChatGPT, as long as OAI-SearchBot is allowed.
Should I block AI training bots?
It’s up to you. Blocking them keeps your pages out of future training data. It doesn’t remove you from AI search, so many sites block training and allow search.
How do I unblock an AI bot?
Open your robots.txt and find the bot’s name. If its group says “Disallow: /”, change it to “Allow: /”.
If the block comes from the group for all bots (User-agent: *), add a group for the bot with “Allow: /”. A bot follows its own group over the one for all bots.
Do I need an llms.txt file?
No. It’s an optional Markdown guide to your site that some AI tools read. Google says you don’t need special AI files to show up in its AI features.
My robots.txt allows every AI bot. Why doesn’t AI mention me?
Being allowed in only means AI can read your site. Whether it recommends you depends on what it finds about you across the web.
LinkGPT asks ChatGPT, Claude, Perplexity, Gemini and Grok about your business, shows which sites they cite instead, and tells you what to fix first.
Can my host block AI bots even if robots.txt allows them?
Yes. Some hosts and CDNs, like Cloudflare, can block AI bots at the firewall. This checker reads your robots.txt rules, so check your host’s bot settings too.
AI can see you. Does it recommend you?
LinkGPT shows what ChatGPT, Claude, Perplexity, Gemini and Grok say about your business, and what to fix first. 3-day free trial.
Start with my website