AI bot traffic in WordPress — see it, measure it, decide what to do
Summary. I opened my server logs one morning expecting the usual Googlebot entries and found strangers instead — GPTBot, CCBot, names I had never invited. WordPress had shown me none of it. Not a counter, not a notice, nothing. This post is what I wish I had found that morning: how to see AI bot traffic on a WordPress site, what each kind of visit actually means, and how I learned to decide per bot instead of copying somebody's blocklist.
Two kinds of AI visitors, and why they matter differently
Here is the confusion I see everywhere, and I had it too: people say "AI bot traffic" like it is one thing. It is two, and they want opposite things from you.
First come the crawlers. They fetch your pages to learn from them. GPTBot gathers training data for OpenAI. CCBot feeds Common Crawl, the nonprofit dataset half the industry trains on. Then there is ClaudeBot and anthropic-ai for Anthropic, Bytespider for ByteDance, PerplexityBot fetching pages so it can cite them in answers. None of them click anything. None of them buy anything. They take your words and leave.
Then there are the referrals, and these walk in the other direction. Someone asks an AI assistant a question, the assistant links to your page, and a living human being lands on your site. In analytics they look like ordinary referrals from chatgpt.com or perplexity.ai. They read. They click. Some of them convert. This is the traffic you actually want — which is why the block-or-allow question is trickier than the angry threads make it sound.
Seeing the crawlers in your logs
WordPress keeps no record of who reads your site. None. No visitor log, no bot list, nothing you can open in the admin. The record lives one floor below WordPress, in the server's access log. Every hosting panel has one — look under Statistics or Metrics — and the file is usually called access.log.
I want to be honest about this part: the first time you open that file, it is ugly. Thousands of lines, IPs, timestamps, noise. Do not read it. Search it. Open find-in-page and type the names. The regulars:
- GPTBot / ChatGPT-User / OAI-SearchBot — OpenAI (training, user actions, search)
- CCBot — Common Crawl, the nonprofit dataset behind many models
- anthropic-ai / ClaudeBot — Anthropic
- PerplexityBot — Perplexity, fetching pages to cite
- Bytespider — ByteDance
- Google-Extended / Applebot-Extended — Google's and Apple's opt-out training crawlers, separate from their search crawlers
- Meta-ExternalAgent — Meta
If your site has been up for a year, every one of these has visited. That was the part that surprised me — not that they came, but how long they had been coming while I looked at nothing but the WordPress dashboard. The question was never whether. It was whether anyone was watching.
Reading one log line
A log line looks scary exactly once. Then somebody shows you where to look, and you cannot unsee it. Here is a typical one:
203.0.113.40 - - [12/Sep/2026:10:14:02 +0000] "GET /pricing/ HTTP/1.1" 200 48210
"-" "Mozilla/5.0 ... compatible; GPTBot/1.2; +https://openai.com/gptbot"Read it left to right. An IP address. The date. The page it asked for — /pricing/, in this case, which already tells you something, because bots that only read your homepage behave differently from bots reading your pricing page. Then 200, meaning your server handed the page over without complaint. The bytes it carried home. And at the very end, the user-agent introducing itself as GPTBot, with a link to OpenAI's own page about it.
One line is trivia. A thousand lines is a story: which bot keeps coming back, which pages it likes, whether it visits at 3am like a polite guest or hammers fifty pages a minute. And here is the catch I keep coming back to — the bots that never introduce themselves never appear in this story at all. Logs show you the honest visitors. They say nothing about the quiet ones.
Should you block them?
Most guides go quiet right here and hand you a snippet to paste. I think that is backwards. The snippet takes ten seconds; the decision deserves ten minutes. So here is how I actually think about it, both sides, no cheerleading.
- The case for blocking. Your content becomes training data for products that compete with you for the same readers. Crawlers cost bandwidth — on a small VPS, an aggressive crawl is a bill and a slowdown. And there is a principle in it: your work, your permission.
- The case against. Blocking does not remove anything already trained on; that data is baked in. Assistants that cannot read your site cite your competitors instead, and the referral visitors in the previous section go to them. For most small sites, obscurity is a bigger threat than being quoted.
My own answer, for what it is worth: I block the bandwidth hogs and the crawlers that give me nothing back, and I welcome the ones that cite pages and send actual humans. PerplexityBot can stay; it links to things. A nameless scraper pulling a thousand pages at midnight cannot. And I revisit the list about once a year, because the cast keeps changing and a decision from 2024 is not a decision, it is a habit.
robots.txt, honestly
So much for the thinking. The mechanism is robots.txt — and before you trust it, you should know exactly what it is. A text file. A request. That is all. Here is what asking the major AI crawlers to stay out looks like:
User-agent: GPTBot
User-agent: CCBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Bytespider
Disallow: /The polite ones obey. GPTBot, CCBot, Google-Extended, most crawlers with real names on them — they check robots.txt before fetching, because ignoring it publicly would be embarrassing for a company that size. But understand what you just did: you put up a sign. There is no lock behind it and no one enforcing it. A crawler that hides its name never reads your sign, and a few named ones have been caught fetching pages they were told not to. Keep the sign up — it is correct and necessary — and then check the logs to see who listened. The log check is the part everyone forgets.
One thing people always ask me at this point, usually worried: will this hurt my Google rankings? No. Googlebot crawls for search under its own name, and Google-Extended — the training opt-out — is a separate user-agent entirely. Refuse one, welcome the other. Your search traffic will not move. I have done exactly this and watched Search Console stay flat.
What llms.txt is actually for
If robots.txt is the keep-out sign, llms.txt is the note you leave for the guests you invited in. It is a young convention, not a standard anyone enforces: a Markdown file at /llms.txt that describes your site the way language models digest best. What the site is. What each section covers. Which pages actually matter.
People misunderstand the purpose, so let me say it plainly: this is not about feeding the models. They already have your pages — that battle is over. It is about accuracy. A model working from your own summary quotes you correctly. A model reconstructing you from scraped fragments invents things, confidently, in your name. I have seen my own plugin described wrong by an assistant, and it is a strange feeling.
The file costs an hour to write well. Whether every assistant reads it is still unsettled — conventions take years, and some may never bother. But there is no downside to having one, and the upside is being cited correctly instead of creatively.
The visitors the assistants send you
Now the second kind of visitor, because I suspect this number will surprise you the way it surprised me. Open your analytics, find referrals, and pull out chatgpt.com, perplexity.ai, copilot.microsoft.com and their friends. Whatever number comes out — that is your AI referral traffic. Real humans, arriving because a machine recommended you. There is something funny about that, and something worth money.
Two things follow from knowing the number. First, it becomes the scoreboard for everything above: if it climbs after you publish an llms.txt and keep your content quotable, the welcoming strategy is paying off. Second, these visitors arrive strange — they land mid-question, half-answered, impatient. The pages they land on should answer fast and link onward. Watch the landing pages, not just the totals. Totals flatter; landing pages teach.
If you want a plugin to watch this
I will say what I always say about this category: nobody opens access logs every morning. Not you, not me, nobody with a site to run. So eventually a tool watches them for you. When you go looking for one, ignore the screenshots and ask these instead:
- Does it show you which bot, which page, and how often — or just a total count? A total tells you nothing to act on.
- Does it separate crawlers from AI referrals? A tool that mixes the two will have you blocking the visitors you wanted.
- Does it manage robots.txt rules per bot, instead of one all-or-nothing switch? Your decision for PerplexityBot and your decision for Bytespider are not the same decision.
- Does it write the llms.txt for you? The file is small but fiddly; generating it from your actual content beats a blank template.
- Does everything happen on your own server? A bot-watching tool that phones your visitor data home is solving one privacy problem by creating another.
Disclosure. I build one of these — CrawlWatch — AI Bot Insights — because after writing all of the above for myself, I wanted it in one place: the log-watching, the per-bot robots.txt control, the llms.txt generator, the referral tracking, all on my own server. Everything I tried had three of the five. It is free and open source. Use it or do not. The checks above work either way, and honestly I would rather you understood what you are looking at than trusted anybody's dashboard. Mine included.
Common mistakes
I have made three of these myself. The rest I have watched other people make in forum threads, which is almost as instructive:
- Copying a blocklist without deciding. Someone else's robots.txt encodes someone else's trade-offs. Decide first, paste second.
- Blocking everything including the citers. The nuclear option feels decisive and costs you every AI referral you would have had.
- Trusting robots.txt as enforcement. It is a request. Check the logs afterwards to see who listened.
- Blocking Google-Extended by accident. It sits next to Googlebot in every example snippet. Read before you paste, or your search crawler stays welcome while you meant to block training — or the reverse.
- Forgetting the referrals exist. Half the story is humans arriving from assistants. If you never segment them, you will make the whole decision looking at half the data.
- Writing llms.txt once and never updating it. A stale summary misquotes you with confidence. Regenerate it when the site changes.
FAQ
- How do I know if AI bots are visiting my WordPress site?
- Check your server access logs for crawler user-agents like GPTBot, CCBot, ClaudeBot or PerplexityBot. WordPress itself records nothing about them, so if you never look at the logs, the visits are invisible. A plugin that watches the logs can show you the same thing inside the admin.
- Should I block GPTBot and other AI crawlers?
- It depends what you want. Blocking keeps your content out of future training data, but it does not remove what is already trained on, and it can cost you citations and visitors from AI answers. Block when bandwidth or principle demands it, not by default.
- Does robots.txt stop AI crawlers?
- The well-behaved ones obey it: GPTBot, CCBot, Google-Extended and most named crawlers check robots.txt before fetching. But robots.txt is a request, not a lock. Crawlers that do not identify themselves never read it, and a few named ones have been accused of fetching anyway.
- What is llms.txt and do I need one?
- llms.txt is a proposed convention: a Markdown file at /llms.txt that summarises your site for large language models, so they quote you accurately instead of guessing from scraped pages. It costs one small file. Whether assistants will universally read it is still unsettled, but there is no downside to having one.
- Do visitors arriving from ChatGPT show up in my analytics?
- Usually yes, as referral traffic from chatgpt.com, perplexity.ai and similar domains. They look like ordinary referrals, so unless you segment them you will never know how many of your visitors an AI assistant sent you.
- Will blocking AI bots hurt my Google rankings?
- No. Googlebot and Google-Extended are separate crawlers with separate user-agents. Blocking GPTBot, CCBot or ClaudeBot in robots.txt touches none of Google's search crawling, and your rankings do not move.
Conclusion
So that is the whole picture, and it fits in one paragraph: AI bot traffic on a WordPress site is two stories — crawlers taking your words, assistants sending you readers. Open the logs so both become visible. Decide per bot, not all at once. Write the decision into robots.txt knowing it is a request, then verify in the logs like someone who does not trust signs. And leave an llms.txt out for the honest ones, so they quote you right.
That morning I found strangers in my logs, I felt behind. Now I think being behind is the normal state — the cast of crawlers changes every year, and nobody gets to decide once and relax. Do the looking, make the decision, and check again next year. The worst case is a quiet log file. The alternative is letting every crawler's defaults decide for you, on your content, every day, while your dashboard shows you nothing.
Related guides
- What the WordPress.org plugin review actually flagged in my plugin
- How to find unused images in WordPress — and delete them safely
- The plugins I build
