AI Crawler Access Checker
Are AI Crawlers blocked from your site?
Instantly find out
Use this tool to check if your website’s robots.txt is configured to allow or block popular AI crawlers.
Quickly verify your site’s current AI access settings like GPTBot, ClaudeBot, and others.
Check robots.txt for AI Bot Blocks
Enter your domain (with or without http/https):
What is robots.txt?
robots.txt is a standard file used to tell crawlers which parts of your site they’re allowed to access. It’s placed at the root of your domain—like https://yoursite.com/robots.txt. Originally designed for search engines, it’s now also used to block AI bots like GPTBot, ClaudeBot and others. Whether or not you want AI tools reading your content, robots.txt gives you a say. You can allow some, block others, or shut them all out entirely.
How do you block AI bots?
To block AI bots, you just add rules to your robots.txt file. For example:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
This tells those bots to stay out. You can block one, many, or all at once.
Do you really want to block them?
That depends. Blocking AI bots protects your content from being used to train models or served elsewhere without credit. But it also means losing visibility in tools that might surface your site in AI-driven answers or summaries.
There’s no single right answer — it’s about what matters more to you: control or reach.
Read more on our blog:
To Block or Bot to Block… Should You Block AI Bots from Crawling Your Website?
As a wise playwright once said, to block or bot to block, that is the question. If Shakespeare were alive today, would he have been proud that Google’s AI chatbot began life named after a term he is commonly referred to as? Sadly, we shall never know. However, what we do know is that AI
What are some common User-agents
Common AI-related User Agents
ClaudeBot – AI bot from Anthropic.
Claude-Web – AI web crawler from Anthropic.
anthropic-ai – Anthropic’s agent.
cohere-ai – used by Cohere for model training.
Bytespider – used by ByteDance for AI training.
GPTBot – OpenAI’s web crawler for training.
ChatGPT-User – denotes ChatGPT browsing activity.
Google-Extended – used by Google to expand AI training data.
PerplexityBot – bot from Perplexity.ai.
Perplexity-User – user browsing via Perplexity.
Google-CloudVertexBot – part of Google’s Vertex AI services.
meta-externalagent – Meta’s data collection for AI.
OAI-SearchBot – OpenAI’s search crawling bot.
CCBot – used by Common Crawl, supports AI training.
Common Non-AI User Agents
TurnitinBot – Not AI; plagiarism detection bot.
Amazonbot – Not primarily AI; used by Amazon for web indexing.
magpie-crawler – Likely scraping bot, unclear AI use.
omgili – Not directly AI; forums/thread indexing bot.
omgilibot – Same as omgili; not explicitly AI.
Scrapy – Not an AI bot; open-source scraping framework.
Applebot-Extended – Unclear AI use; used for Siri and Spotlight.
YandexAdditional – Not primarily AI; indexing-related.
YandexAdditionalBot – Same as above; indexing.
Find Out More:
Are Title Tags Still Important for SEO?
We all want to know how effective areas of optimisation can be, and with many articles floating around online often contradicting one another it can be challenging to comprehend.
Schema Markup for Dummies: An MRS Guide
Schema markup is one of the most under-utilised SEO techniques out there. This is surprising, because it is one of the most valuable and powerful tools available to search engine marketers. So, why are people missing out?
CPA Calculator
Our CPA Calculator is designed to help you understand how you can change the metrics that make up your CPA in order to hit your targets.