A New Class of Visitor on Your Practice Website
Alongside Googlebot, Bingbot, and the standard search crawlers, a new generation of bots has been visiting practice websites since 2023. These are the crawlers operated by AI companies: OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot, Perplexity’s PerplexityBot, Meta’s FacebookBot (in its AI-training mode), and several others.
They are not indexing your site for a search results page. They are reading your content to train large language models, to build knowledge bases for retrieval-augmented generation (RAG), or both. The distinction matters for how you think about visibility, control, and what you want these systems to know about your practice.
This article explains how each major AI crawler works, what they take, how to control access, and how to optimize for the outcome you actually want: being cited accurately when patients ask AI systems about surgical care in your market.
The Two Purposes of AI Crawling
Not all AI crawlers do the same thing. The fundamental split is between training crawlers and retrieval crawlers.
Training crawlers collect content to include in the dataset used to train a language model. The content becomes part of the model’s weights. If your practice website content is included, the model may “know” facts about your practice, but it cannot cite a specific URL. It has absorbed the content, not stored a reference to it.
Retrieval crawlers index content into a live knowledge base that the AI system queries at inference time. When a user asks a question, the system fetches relevant pages from its index and incorporates them into the response, with attribution. This is what produces citations in Perplexity, ChatGPT’s web search mode, and Google AI Overviews.
Most AI companies now operate both types. A practice that optimizes only for training exposure gets absorbed into generic model knowledge with no citation. A practice that also optimizes for retrieval gets named as a source when a patient asks a relevant question.
The Major AI Crawlers and What They Do
GPTBot: OpenAI training crawler
User agent: GPTBot. Used exclusively for collecting publicly available content that may be used to train OpenAI’s generative AI models. GPTBot is a training crawler, not a search crawler. Blocking it prevents your content from being used in OpenAI model training, but it has no effect on whether your site appears in ChatGPT Search results.
OAI-SearchBot: OpenAI ChatGPT Search crawler
User agent: OAI-SearchBot. This is the crawler that indexes content for ChatGPT Search (formerly SearchGPT). If you want your practice to appear in ChatGPT Search results, OAI-SearchBot needs access. Blocking GPTBot while allowing OAI-SearchBot is a valid configuration: your content is excluded from training but remains eligible for search citation.
One practical nuance from OpenAI’s documentation: if a site permits both GPTBot and OAI-SearchBot, OpenAI may consolidate the crawl internally to avoid redundant visits. The permissions remain separate regardless: GPTBot controls training eligibility, OAI-SearchBot controls search eligibility, and the two do not depend on each other.
ChatGPT-User: OpenAI user-triggered agent
User agent: ChatGPT-User. This agent fetches a specific page only when a user explicitly asks ChatGPT to retrieve it, or when a GPT Action is triggered that requires accessing a URL. It is not a scheduled crawler and does not build an index. You will see it in access logs when a ChatGPT user pastes a link and asks the model to read it.
ClaudeBot (Anthropic)
User agent: ClaudeBot. Used for training data collection for Claude models. Anthropic also operates anthropic-ai as an agent string in some contexts.
What it collects: public web content for model training. As of 2025, ClaudeBot’s retrieval pipeline (for Claude’s real-time search) goes through a separate partnership layer rather than direct crawling.
PerplexityBot
User agent: PerplexityBot. Perplexity is primarily a retrieval system; it queries a live web index rather than a static training corpus. PerplexityBot crawls pages to build and refresh that index.
This is the crawler most directly tied to citation outcomes. A page that PerplexityBot has indexed well is more likely to appear as a cited source when a patient asks Perplexity about surgical procedures in your area.
Google Extended / Google-Extended
User agent: Google-Extended. Google’s separate crawler for Gemini model training and the Vertex AI API, distinct from Googlebot. Pages can block Google-Extended without affecting standard Google Search indexing.
Meta-ExternalAgent
User agent: Meta-ExternalAgent. Used by Meta for AI training data collection. Respects robots.txt directives.
Common Crawl
User agent: CCBot. Common Crawl is a nonprofit that publishes open web crawl datasets widely used in AI training. Its dataset has been incorporated into the training of GPT-3, LLaMA, and numerous other models. Unlike the commercial crawlers above, Common Crawl operates on a slower schedule and does not support real-time retrieval.
robots.txt: Your Control Layer for AI Crawlers
robots.txt remains the primary mechanism for communicating crawl permissions to bots. All the major AI crawlers above respect robots.txt directives for their specific user agents.
A practice that wants to block AI training crawlers while allowing retrieval crawlers (to preserve citation eligibility) can configure this granularly:
# Block training crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow search/retrieval crawlers (ChatGPT Search, Perplexity)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Standard search crawlers always allowed
User-agent: Googlebot
Allow: /This is a reasonable starting position for most practices: block GPTBot and the bulk training harvesters that absorb content without producing citations, while allowing OAI-SearchBot and PerplexityBot that are directly tied to search results where your practice can be named. Note that blocking GPTBot here does not affect ChatGPT Search eligibility, which is controlled by OAI-SearchBot independently.
A practice that wants no AI crawling at all can blanket-block with User-agent: * combined with specific Allow directives for Googlebot and Bingbot. This is a valid choice, though it forfeits AI citation eligibility entirely.
What AI Crawlers Actually Prioritize
Understanding what AI retrieval systems rank highly helps in structuring content for citation. The patterns are consistent across Perplexity, ChatGPT search, and Google AI Overviews.
Direct, specific answers. A page that opens with a clear statement (“TLIF is typically recommended for patients with single-level lumbar instability who have not responded to six months of conservative treatment”) is far more citation-eligible than a page that approaches the same topic through narrative or marketing language. AI systems extract the specific claim, not the surrounding context.
Factual claims with implicit sourcing. AI retrieval prefers content that reads as authoritative: definitive statements, specific numbers, named procedures, named conditions. Hedging language and vague qualifications reduce citation confidence.
Structured content with clear sections. Heading structure is how AI systems navigate to the relevant section of a long page. A page with well-labeled H2 and H3 sections enables the AI to pull a specific answer from a specific section without ingesting the whole page.
Fast, clean pages. Retrieval crawlers do not wait for JavaScript to render. Page weight, render time, and JavaScript dependency all affect whether a crawler successfully indexes a page’s content.
Schema.org markup. Structured data is a direct signal to AI systems about what a page contains. A page with MedicalProcedure schema is more likely to be retrieved when a user asks about that procedure than an identical page without schema.
The Citation Gap: Why Some Practices Get Named and Others Don’t
In any metro area with competitive surgical practices, AI systems are answering patient questions about local surgeons. The practices that get named are not always the largest or most established. The pattern is content-driven.
Practices that get cited reliably share a few traits: individual procedure pages (not a single services list), surgeon profile pages with structured credentials, FAQ content that answers specific patient questions, fast pages with no JavaScript dependency on primary content, and Schema.org markup covering MedicalOrganization, Physician, and MedicalProcedure.
Practices that are invisible in AI responses tend to have: a single “Services” page, no surgeon-specific content, slow WordPress builds with third-party JavaScript rendering primary content, and no structured data beyond a basic business listing.
The gap is not primarily a function of domain authority or backlink count, which are the dominant factors in traditional SEO. It is a function of content structure and machine readability.
Practical Steps for AI Visibility
If you want your practice to appear in AI-generated answers when patients in your market are researching surgical options:
- Audit your robots.txt. Confirm that OAI-SearchBot and PerplexityBot are not blocked. These are the crawlers tied to ChatGPT Search and Perplexity citation respectively. GPTBot (training only) can be blocked without affecting search visibility. If you have an older WordPress site, a plugin may have added blanket bot blocking that was intended for spam crawlers but also blocks AI retrieval systems.
- Check whether your primary content renders without JavaScript. Navigate to your site, disable JavaScript in the browser (DevTools → Settings → Disable JavaScript), and reload. If your procedure descriptions and surgeon profiles disappear, a retrieval crawler likely cannot read them.
- Add individual procedure pages. Each procedure you offer should have its own URL with a dedicated, substantive page. A single “Services” dropdown is invisible to AI citation systems.
- Implement MedicalProcedure and Physician schema. This is the single highest-leverage structured data investment for surgical practice AI visibility. Our guide to GEO and structured data for surgical practices covers each schema type in detail.
- Add FAQ blocks to procedure pages. Three to five specific questions with direct answers on each procedure page is the most reliable path to appearing in AI overview responses.
The DataCentury infrastructure build implements all of the above natively. Every site we deliver has robots.txt configured for selective AI crawler access, static rendering of all primary content, Schema.org markup across MedicalOrganization, Physician, and MedicalProcedure types, and a FAQ layer on procedure pages.