CrawlWatch — see which AI bots read your site
Summary. CrawlWatch is a free WordPress plugin that logs AI-crawler visits where you can see them: which bots came, which URLs they took, and whether anyone arrived from an AI assistant afterwards. It also scores how ready your site is for AI readers, generates llms.txt, and lets you allow or block each bot — all without anything leaving your server.
The traffic your analytics never mentions
GPTBot, ClaudeBot, PerplexityBot and Google-Extended read sites every day, and none of it appears in a normal analytics dashboard. You cannot tell which pages they took, whether your content is worth reading to them in the first place, or whether a single visitor ever arrived from an AI assistant afterwards.
CrawlWatch puts that traffic on a screen in your own dashboard. It recognises more than forty AI crawlers, stores every visit in one table on your server, and asks for nothing in return — no API key, no account, no external service.
A score with a to-do list attached
The readiness score runs from 0 to 100, but a number on its own never fixed anything. Underneath it sits a breakdown of what was measured — llms.txt catalogue, excerpts as AI descriptions, schema, image alt text, robots.txt, active tracking — and a short list of quick wins ordered by impact, each linking to the exact screen that resolves it. The score is the headline; the list is the point.
Two files, no code
On the Files screen, llms.txt generates in one click — a catalogue version plus a full-content one — and each AI bot gets its own allow-or-block checkbox for robots.txt, with a preview before anything is saved. Blocked bots are told Disallow; the plugin only appends, so the WordPress core rules are never overwritten. And blocking a training bot touches search traffic not at all, because regular Googlebot is never managed here.
One honest limitation
Tracking runs in PHP, so a full-page cache that serves a page without running PHP will hide those visits from the log. The plugin detects the common caching plugins and says so on its own screens instead of quietly showing an empty chart. I would rather read that warning than wonder why a busy site shows no bots at all.
FAQ
- Which AI bots does it recognise?
- More than forty, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended. A crawler the list does not know is one the plugin cannot report, so the signature list is the first place to look when traffic goes missing.
- Will blocking a bot hurt my Google rankings?
- No. The screen only manages AI training and search bots. Regular Googlebot is never touched here, so search traffic is unaffected.
- Does any data leave my server?
- No. There is no API key and no external service. Every visit is stored in one table on your own server, IPs are hashed and truncated, and logs older than 30 days are deleted on a schedule.
- Will it slow down my site?
- No. It adds no CSS or JavaScript to the front end, and a normal human visit costs zero extra queries.
- Is there a Pro version?
- No. Bot logging, referrals, the readiness score, llms.txt and robots.txt — everything is in the free plugin, reviewed by the WordPress.org team before release.
Install it and give it a few days
Install it from your dashboard like any other plugin, open the CrawlWatch menu, and give it a few days — bot traffic arrives on its own schedule, not yours. The readiness score tells you what to fix first.
Download on WordPress.org Source on GitHub
