Forge AI Bot Log – Block AI Bots, Crawler Detection & Verification

Block GPTBot to stop OpenAI collecting your pages as training data for its models, and block OAI-SearchBot and ChatGPT-User if you also want to keep your content out of ChatGPT’s search and live page-fetching. The plugin lists all three separately, because most people want to opt out of training without disappearing from AI search. Blocking writes the relevant robots.txt directives for you.

Yes — that is the main reason this plugin exists. A user-agent string is trivial to forge, so any scraper can send GPTBot and appear in your logs as OpenAI. On each tracked visit the plugin checks the visiting IP against whichever proof that crawler’s owner publishes. For OpenAI, Anthropic and Perplexity that is their official IP-range list. For Google, Apple, Amazon and Meta it is a reverse DNS lookup on the visiting IP, re-resolving the hostname it gets back to confirm it maps to the same address and to the crawler’s published domain. Genuine crawlers pass; spoofing does not. Hits are labelled Verified, Spoofed or Unverified so you can see the difference.

Tick the bots you want to block on the plugin’s Access page. The directives are merged into the robots.txt WordPress serves, so you never edit a file by hand. One caveat worth knowing: if a real robots.txt file exists in your site root, your web server serves that file directly and WordPress never runs — so the plugin’s rules will not appear. Delete or fold in that physical file if you want the plugin to manage your AI crawler directives.

robots.txt is advisory. Well-behaved bots obey it; some (Bytespider, Perplexity-User and Grok’s fetchers, in the bundled list) are reported to ignore it. For those the plugin generates an .htaccess hard-block snippet that rejects them at the server level, on Apache.

No. Google-Extended controls whether your content is used to train and ground Google’s Gemini models. Normal Google Search crawling and indexing is done by Googlebot, which this plugin does not touch, so your search rankings are unaffected. Note also that blocking Google-Extended does not remove you from AI Overviews, which are generated from Google’s regular search index.

26 bots from 16 companies — OpenAI, Anthropic, Google, Perplexity, xAI, Amazon, Apple, Meta, ByteDance, Cohere, Mistral AI, DuckDuckGo, You.com, Diffbot, Timpi and Common Crawl — split into AI training crawlers, AI search indexers and live retrieval agents. Each has its own entry explaining what it does with your pages. The crawlers behind the big assistants are all covered, so ChatGPT, Claude, Gemini, Grok and Perplexity traffic is identified by name rather than lumped together.

xAI is the least transparent of the major AI companies. Its documented user agents (GrokBot, xAI-Grok, Grok-DeepSearch) are in the bundled directory and are logged when they appear, but independent monitoring consistently finds that much of Grok’s real fetching arrives under generic Chrome or Safari user-agent strings from rotating IP addresses, which no user-agent-based tracker can attribute. Grok can also answer questions from its search partner’s existing index without visiting your site at all. So treat Grok’s numbers in any tool — this one included — as a minimum.

No. Logging is limited to matched AI bots and runs on an early hook, and verification is throttled and cached — vendor IP-range lists are fetched at most once a day and stored locally. LLMs are not consulted at any point; detection is local pattern matching against the bundled list.

Full-page caches can serve bots without running PHP. Use the included must-use early logger and/or the access-log importer to capture those hits.