What AI Bots Do on Your Site — 24h Logs: 24 Training Crawls vs 424 Citation Fetches
RanketAI measured 24 hours of production server logs: most AI bot traffic was live citation fetches — pages read the moment a user asked — not training crawls (GPTBot 4 vs ChatGPT-User 390). How the three bot types differ and how robots.txt applies to each.
This blog content may use AI tools for drafting and structuring, and is published after editorial review by the RanketAI Editorial Team.
Key takeaway: When people say "AI bots are scraping my site," they usually picture training crawls. Measuring 24 hours of logs on the RanketAI production site showed the opposite — 24 training-crawl hits (GPTBot·ClaudeBot) versus 424 live citation fetches (ChatGPT-User·Claude-User): pages opened at the very moment a user asked a question. AI bot traffic is less about "being absorbed into a model" and more about "being read at the moment an answer is generated." That distinction changes how you configure robots.txt and how you think about AI visibility.
Three-line summary
- Out of 17,712 requests in 24 hours, 588 came from AI bots — 72% (424) were live citation fetches, 24% (140) AI search indexing, and only 4% (24) training crawls.
- The three bot types serve different purposes, and robots.txt works differently for each — blocking a training bot is an opt-out signal for training, blocking an indexing bot removes you from AI search answers, and citation-fetch agents are user-initiated so robots.txt may not apply at all.
- This is a single site over a single day — take the method of splitting your own logs into the three types, not the exact ratios.
The measurement — 24 hours of AI bot traffic
RanketAI classified AI bot User-Agents in 17,712 requests logged on ranketai.com across July 22–23, 2026 (24 hours).
| Purpose | Bot (vendor) | Hits in 24h |
|---|---|---|
| Live citation fetch | ChatGPT-User (OpenAI) | 390 |
| Claude-User (Anthropic) | 34 | |
| AI search indexing | OAI-SearchBot (OpenAI) | 80 |
| PerplexityBot (Perplexity) | 60 | |
| Training crawl | ClaudeBot (Anthropic) | 20 |
| GPTBot (OpenAI) | 4 |
Classic search-engine bots (Googlebot 148 · bingbot 317 · Naver Yeti 66) logged a separate 531 hits in the same window. Looking at AI bots alone, the structure is stark — 424 citation fetches : 140 indexing : 24 training. Both OpenAI and Anthropic showed user-initiated agents far outrunning their training crawlers (OpenAI 390 vs 4, Anthropic 34 vs 20).
The three bot types — in the vendors' own words
This three-way split is not RanketAI's interpretation; it is how the vendors themselves define their crawlers. OpenAI's crawler documentation states the purpose of each bot (OpenAI Crawlers).
"When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." — OpenAI Crawlers documentation
- GPTBot — crawls content that may be used to train foundation models. Disallowing it in robots.txt signals "exclude my content from training."
- OAI-SearchBot — indexes sites for ChatGPT's search features. Block it and your site will not appear in ChatGPT search answers.
- ChatGPT-User — opens your page at the moment a user's question needs it. Because these are user-initiated actions rather than automatic crawling, OpenAI states that robots.txt rules may not apply.
Anthropic runs the same structure — training-collection ClaudeBot separated from the user-initiated Claude-User agent. Perplexity's PerplexityBot is an indexing bot for search exposure.
Interpretation — AI visibility is less "trained in" than "read live"
Citation fetches at 72% corrects a common assumption about how AI visibility works. If training collection were the main channel, your content's fate would be locked to training cycles months apart. What dominated the RanketAI measurement instead was live reading at answer-generation time. Three implications:
- Fix it now, and it can show up now. A citation fetch reads the page as it exists when the question arrives. Content updates and structural fixes have a path into the very next answer — no waiting for a training cycle.
- If you cannot be read at fetch time, you drop out of that answer. A slow page, a JS-dependent body the bot cannot parse, or a firewall blocking the agent can knock you out of a citation even when the model knows your brand.
- Blocking training bots and AI exposure are separate questions. A site that blocks GPTBot can still receive ChatGPT-User citation fetches (user-initiated, so robots.txt may not apply — per the official documentation). Conversely, blocking OAI-SearchBot removes you from ChatGPT search answers entirely. Treating "block AI bots" as a single decision produces results you may not intend.
In practice — reading the three types in your own logs
- Tally User-Agents in three buckets. In your hosting console's request logs or server access logs, count
GPTBot·ClaudeBot(training) /OAI-SearchBot·PerplexityBot(indexing) /ChatGPT-User·Claude-User(citation fetch) separately. If citation fetches are showing up, your site is already in the candidate pool for . - Split robots.txt by intent, not by vendor. If your goal is training opt-out, disallowing GPTBot and ClaudeBot is enough. If AI search exposure matters, OAI-SearchBot and PerplexityBot need to stay open. Check whether a firewall or bot-blocking layer is silently blocking these agents wholesale.
- Verify you are readable at fetch time. Citation-fetch agents often read without JS rendering, so your body content needs to be present in the initial HTML. A page structure check reviews AI crawler accessibility and structural signals for free, and AI brand visibility analysis measures repeatedly how your brand actually appears in AI answers.
One more reminder — these numbers are a single site's single day. Ratios will vary with content type and traffic; take the classification method and its implications rather than the exact figures.
FAQ
If I block GPTBot, do I disappear from ChatGPT answers?▾
No. Blocking GPTBot is a training opt-out signal; ChatGPT search exposure (OAI-SearchBot) and citation fetches at question time (ChatGPT-User) are separate paths. Block all three, however, and your routes into AI answers effectively vanish.
Why were there only 4 training-crawl hits?▾
Training crawls run as low-frequency batch collection, and there is little reason to re-crawl an already-collected site daily. Citation fetches, by contrast, fire every time a user question needs your page — which is how we interpret the wide gap in the RanketAI sample.
What if my citation fetches are zero?▾
First check whether bots can reach you at all (firewall, robots.txt, JS-dependent structure), then whether you have content worth citing — comparison and reference material that answers questions head-on. Accessibility is checkable with a page structure check; actual brand mentions with AI brand visibility analysis.
Related reading
Execution Summary
| Item | Practical guideline |
|---|---|
| Core topic | What AI Bots Do on Your Site — 24h Logs: 24 Training Crawls vs 424 Citation Fetches |
| Best fit | Prioritize for geo workflows |
| Primary action | Standardize an input contract (objective, audience, sources, output format) |
| Risk check | Validate unsupported claims, policy violations, and format compliance |
| Next step | Store failures as reusable patterns to reduce repeat issues |
Data Basis
- First-party measurement: a 24-hour sample of server request logs on the RanketAI production site (ranketai.com), 2026-07-22 16:54 to 07-23 16:52 KST, 17,712 requests in total. AI bots classified by User-Agent — OpenAI 474 hits (GPTBot 4 · OAI-SearchBot 80 · ChatGPT-User 390), Anthropic 54 (ClaudeBot 20 · Claude-User 34), Perplexity 60 (PerplexityBot). A single site and a single 24-hour window — presented as a log-reading methodology, not a generalizable ratio.
- Bot definitions: official crawler documentation from OpenAI, Anthropic, and Perplexity, checked on 2026-07-23, quoted for each bot's purpose (training / search indexing / user-initiated fetching) and how robots.txt applies.
- Classification limits: User-Agent strings can be forged, so this tally counts official UA strings only. Forged-UA traffic was classified separately as browser-like and excluded from AI bot counts.
Key Claims and Sources
This section maps key claims to their supporting sources one by one for fast verification. Review each claim together with its original reference link below.
Claim:GPTBot crawls content that may be used in training generative AI foundation models; disallowing it in robots.txt signals that the site should be excluded from training
Source:OpenAI Crawlers documentationClaim:OAI-SearchBot surfaces websites in ChatGPT search results; sites that opt out will not appear in ChatGPT search answers
Source:OpenAI Crawlers documentationClaim:ChatGPT-User visits web pages when users ask questions; because these are user-initiated actions rather than automatic crawling, robots.txt rules may not apply
Source:OpenAI Crawlers documentationClaim:Anthropic likewise separates ClaudeBot for training collection from the user-initiated Claude-User agent
Source:Anthropic crawler documentation
External References
The links below are original sources directly used for the claims and numbers in this post. Checking source context reduces interpretation gaps and speeds up re-validation.
Is your site visible in AI search?
See for free how ChatGPT, Perplexity, and Gemini describe your brand.
Start Free Diagnosis →Related Posts
These related posts are selected to help validate the same decision criteria in different contexts. Read them in order below to broaden comparison perspectives.
The 40% GEO Gain Has a Condition — Inside the 2026 Critical Survey of 45 Studies
The famous claim that GEO lifts AI-answer visibility by up to 40% was measured with the source already inside the answer context. A 2026 critical survey grades 45 GEO studies (2023-2026) and shows what reproduces, what backfires, and what to measure first.
The Source Gap — Finding the Third-Party Sites That Cite Only Your Competitors
Community platforms capture 52.5% of AI citations vs 47.5% for brand domains (OtterlyAI, 1M+ citations). Half the game happens off your site — how to find and classify the third-party sources citing only your competitors, with a field case.
AI Search Answers Without Referring: 9.4% Search Displacement, 5.2% Outbound Clicks
A new arXiv study finds wider ChatGPT Search access cut traditional search use by 9.4%, while only 5.2% of ChatGPT sessions produced an outbound click versus 31.1% for Google. Why brands should measure presence inside AI answers, not just site visits.
Don't Measure AI Visibility Once: Why You Need 7 Runs a Day and 2–4 Week Windows
University of St. Gallen researchers measured ChatGPT, Perplexity, Gemini, and Google AI Mode for 45 days: cited sources turned over roughly 65% day to day, and trusting brand metrics took 7 runs per prompt per day plus 2–4 week rolling windows.
ChatGPT Ads Land in Korea — What Advertising Can and Cannot Buy in AI Answers
OpenAI now runs ChatGPT ads in Japan and South Korea — its first non-English markets. We break down the ad format, plan-tier exposure, and targeting, and why the answer body — the spot ads cannot buy — is where Korean brands should invest first.