All-IN Crawler & AI Agent Manager – Block AI Bots, Bad Bots & Rate-Limit Crawlers

Installing it does not change how crawlers are treated: a new install starts in Watch-only mode. With the default Balanced rules, search engines stay allowed; verification only catches scrapers pretending to be a known crawler. Rate limits and load shedding answer with 429 or 503 and Retry-After, which major search engines treat as “try again later”. It can affect SEO if you choose that: blocking a search engine, the Lockdown preset, or robots.txt Disallow lines for a blocked bot tell that bot to stay out.

Apply the “Protect content from AI training” preset on Bot Rules, or set the AI training category to Block, then switch to Enforce. AI search and assistant agents (ChatGPT search, Claude, Perplexity) stay allowed, so you stay in AI answers.

Set vulnerability scanners, scripts & HTTP libraries, unidentified bots and empty User-Agents to Block or Rate-limit, block abusive IPs from the Dashboard, and keep impostor protection on so fake Googlebots are rejected.

Use Rate-limit instead of Block. Over-limit requests get 429 with Retry-After, which tells crawlers to slow down rather than drop your pages. Verified search engines are not limited by the default rules.

Yes. Turn it on in the llms.txt tab. It lists your published content and leaves out pages marked noindex in Yoast SEO, Rank Math or SEOPress.

Not without blocking Google Search. AI Overviews use regular Googlebot. Google-Extended only controls Gemini training/grounding use.

No. It complements them. Large DDoS attacks must be stopped before they reach PHP.

No. Search engine bots are verified with DNS lookups through your own server’s resolver. The official crawler IP lists and a bot-list feed URL of your own are off until you turn them on. See “External services”.

OpenAI and Perplexity bots have no reverse DNS, so without their published IP lists a real GPTBot can’t be told apart from a scraper using the same name. With the lists on, a request claiming to be one of those bots from an unlisted IP is treated as an impostor. A list is only used while under 7 days old, so a failed download never blocks a genuine bot.

Cached pages don’t run WordPress, so the plugin doesn’t see them, but they are cheap. Denials are sent with Cache-Control: no-store and never cached.

Found a bug or have a question? Use the support forum. If the plugin helps you, a review helps other site owners find it.