Skip to content
SBMM Cloud
Sign inGet started free

Free AI Crawler Access Check

Test which AI crawlers your robots.txt allows.

What a run costs
One free run a day, no account needed
How it runs
Straight away
Who it is for
Free: 2 a day. Silver: 5 a day. Gold: 10 a day. Ultimate: unlimited.

A free account raises the daily allowance and keeps your results. Create one — no card needed.

Loading…

Overview

AI crawlers are the bots that ChatGPT, Claude, Gemini, Perplexity, and other generative engines send to your website. They read your robots.txt to decide what they can fetch for training, for live search, and for on demand user queries. This free tool checks any URL against all 22 known AI crawlers.

Which Bots Get Tested

The checker covers the full current set: GPTBot, OAI SearchBot, and ChatGPT User from OpenAI; ClaudeBot, Claude Web, anthropic ai, and Claude SearchBot from Anthropic; Google Extended for Gemini training; PerplexityBot for live AI search; Applebot Extended; Meta ExternalAgent; Bytespider; CCBot; and several more.

How To Read The Result

For each bot you get a clear allowed or blocked verdict, the specific robots.txt rule that produced it, the bot vendor and purpose (training, search, on demand), and a link to the vendor documentation page so you can verify behaviour. Use the Robots.txt Generator tool to build a policy from scratch with the right AI crawler blocks in place.

About the tool

What Is The SBMM AI Crawler Access Checker?

The SBMM AI Crawler Access Checker is a free online tool that tests any URL against the 22 plus known AI crawlers and returns a clear allowed-or-blocked verdict per bot. It tells you exactly which AI training datasets are ingesting your content, which AI search engines can cite your pages, and which crawlers your robots.txt is silently letting through when you thought it was blocking them.

AI crawlers are the bots ChatGPT, Claude, Gemini, Perplexity, and other generative engines send to your website. They read your robots.txt to decide what they can fetch for training, for live search, and for on-demand answers to user queries. Most site owners do not realise the same vendor often runs multiple bots (one for training, one for live search) that need separate access rules to handle correctly.

This checker covers the full current set of AI crawlers across every vendor, classifies each bot by purpose (training versus search versus on-demand), reads your robots.txt, and tells you for each bot whether it is allowed or blocked along with the exact rule that produced the verdict. You finish the audit knowing exactly what your AI access policy actually is, not what you assumed it was.

Step by step

How To Use This Tool In 3 Steps

  1. Enter The URL You Want To Check

    Drop any public URL into the form. The checker fetches the robots.txt for that domain and parses every User-agent group to see how each known AI crawler is treated.

  2. Tested Against 22+ AI Crawlers

    Each AI bot in the matrix is matched against the robots.txt rules in priority order (most specific User-agent first, then wildcard). The checker records the matched rule and the resulting allow / block verdict.

  3. Read The Per-Bot Verdict Matrix

    See a clear table: bot name, vendor, purpose (training / search / on-demand), allowed or blocked verdict, the exact robots.txt rule that decided it, and a link to the vendor documentation page so you can verify the bot identity.

Why this tool

Why Use This Tool

  • 22+ AI Crawlers Covered

    GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, Claude-SearchBot, Google-Extended, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, cohere-ai, Omgilibot, FacebookBot, and more.

  • Training Vs Search Bot Classification

    Each bot is tagged by purpose. Training bots ingest content for future model training. Search bots fetch live for AI Overviews. On-demand bots fetch per-user-query. The right policy depends on which jobs you want to allow.

  • Per-Rule Diagnostic

    Every verdict shows the exact robots.txt rule that produced it (User-agent group plus matched Disallow / Allow line) so you can fix the rule, not guess. No more "I thought I blocked GPTBot" surprises.

  • Vendor Doc Links Per Bot

    Every bot in the matrix links back to the vendor documentation page that defines its user agent string, purpose, and intended behaviour, so you can verify the bot identity is current with the vendor's latest disclosure.

  • Live Robots.txt Fetch

    The checker fetches the live robots.txt at audit time, not from a stale cache, so the result reflects exactly what the AI crawler sees on its current visit to your domain.

  • Free, One Run A Day

    Ten URLs in one run without an account, 25 signed in, on your own site or a competitor's. One free run a day without an account. A free account gives you two runs a day of this tool, keeps your results and lets you download them. Paid plans raise the allowance further.

FAQ

Frequently Asked Questions

What is an AI crawler?

An AI crawler is an automated bot that visits your website on behalf of an AI vendor (OpenAI, Anthropic, Google, Perplexity, Meta, and others) to ingest your content for model training, live AI search, or on-demand answers to user queries. Each vendor runs one or more named crawlers with distinct user agent strings and purposes.

How are AI crawlers different from search engine crawlers?

Search engine crawlers like Googlebot exist to build a search index. AI crawlers exist to ingest content for training or for live model retrieval. They use different user agent strings, follow different politeness rules, and respond to separate robots.txt directives, so a site owner needs explicit rules for each.

Should I block AI crawlers from my site?

It depends on your content rights strategy. Blocking GPTBot, ClaudeBot, and Google-Extended prevents your content being used to train future model versions but does not affect live AI search citations from PerplexityBot or OAI-SearchBot. Most publishers block training crawlers but allow live search crawlers to keep getting cited.

What is the difference between GPTBot and OAI-SearchBot?

Both are OpenAI bots. GPTBot crawls your site to gather training data for future ChatGPT versions. OAI-SearchBot crawls for live AI search results inside ChatGPT. Blocking GPTBot keeps your content out of training; blocking OAI-SearchBot means ChatGPT cannot cite you when users ask questions in your niche.

What is Google-Extended?

Google-Extended is the separate user agent Google uses to fetch content specifically for Gemini training and Bard / Search Generative Experience answers. Blocking Google-Extended in robots.txt opts your content out of Gemini training and AI Overviews without affecting normal Googlebot crawling for regular search ranking.

Do AI crawlers respect robots.txt?

The major vendors (OpenAI, Anthropic, Google, Perplexity, Apple, Meta) publicly commit to respecting robots.txt. A small set of lower-quality scrapers ignore it. If you need stronger enforcement than robots.txt, add server-side IP blocks or user-agent firewall rules on top.

Why does the checker need to read my robots.txt?

Robots.txt is the file every AI crawler reads to decide what it can fetch. The verdict for each bot depends on the matched rule in that file. Without reading the live robots.txt, the checker cannot tell you which bots can actually see your site.

How often should I run this audit?

Run it after any robots.txt change, after a major site migration, and at least once a quarter as a maintenance pass. New AI crawlers launch frequently, and existing rules sometimes break in unexpected ways after a CMS upgrade or theme switch, so quarterly is a sane cadence. After verifying access, publish an llms.txt manifest so the bots you allow get a curated reading list.