CrawlWatch — see which AI bots read your site
Summary. CrawlWatch is a free WordPress plugin that logs AI-crawler visits where you can see them: which bots came, which URLs they took, and whether anyone arrived from an AI assistant afterwards. It also scores how ready your site is for AI readers, generates llms.txt, and lets you allow or block each bot — all without anything leaving your server.
| Version | 1.1.2 |
|---|---|
| Last updated | |
| Requires | WordPress 6.2 or later, PHP 7.4 or later |
| License | GPLv2 or later |
| Price | Free, no Pro version |
Full version history: CrawlWatch changelog on WordPress.org.
AI bot traffic your analytics misses
GPTBot, ClaudeBot, PerplexityBot and Google-Extended read sites every day, and none of it appears in a normal analytics dashboard. You cannot tell which pages they took, whether your content is worth reading to them in the first place, or whether a single visitor ever arrived from an AI assistant afterwards.
CrawlWatch puts that traffic on a screen in your own dashboard. It recognises more than forty AI crawlers, stores every visit in one table on your server, and asks for nothing in return — no API key, no account, no external service.
AI readiness score with quick wins
The readiness score runs from 0 to 100, but a number on its own never fixed anything. Underneath it sits a breakdown of what was measured — llms.txt catalogue, excerpts as AI descriptions, schema, image alt text, robots.txt, active tracking — and a short list of quick wins ordered by impact, each linking to the exact screen that resolves it. The score is the headline; the list is the point.
llms.txt and robots.txt without code
On the Files screen, llms.txt generates in one click — a catalogue version plus a full-content one — and each AI bot gets its own allow-or-block checkbox for robots.txt, with a preview before anything is saved. Blocked bots are told Disallow; the plugin only appends, so the WordPress core rules are never overwritten. And blocking a training bot touches search traffic not at all, because regular Googlebot is never managed here.
Limitations of AI bot tracking
Tracking runs in PHP, so a full-page cache that serves a page without running PHP will hide those visits from the log. The plugin detects the common caching plugins and says so on its own screens instead of quietly showing an empty chart. I would rather read that warning than wonder why a busy site shows no bots at all.
FAQ
- Which AI bots does it recognise?
- More than forty, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended. A crawler the list does not know is one the plugin cannot report, so the signature list is the first place to look when traffic goes missing.
- Will blocking a bot hurt my Google rankings?
- No. The screen only manages AI training and search bots. Regular Googlebot is never touched here, so search traffic is unaffected.
- Does any data leave my server?
- No. There is no API key and no external service. Every visit is stored in one table on your own server, IPs are hashed and truncated, and logs older than 30 days are deleted on a schedule.
- Will it slow down my site?
- No. It adds no CSS or JavaScript to the front end, and a normal human visit costs zero extra queries.
- Is there a Pro version?
- No. Bot logging, referrals, the readiness score, llms.txt and robots.txt — everything is in the free plugin, reviewed by the WordPress.org team before release.
Get started with CrawlWatch
Install it from your dashboard like any other plugin, open the CrawlWatch menu, and give it a few days — bot traffic arrives on its own schedule, not yours. The readiness score tells you what to fix first.
Download on WordPress.org Source on GitHub
