Known Agents – Track AI Bots and Crawlers, Block Scrapers, Analyze LLM Referral Traffic

More than you think. On average, 40% or more of website visits come from AI agents and bots, a volume that is completely invisible to traditional analytics tools like Google Analytics. This has been observed consistently across websites of all sizes and industries.

AI agent traffic has a direct and growing impact on your business. It determines the following:

• Whether bad actors and unauthorized scrapers are stealing your content
• Whether your content is included in AI model training datasets
• How accurately AI assistants like ChatGPT and Claude summarize and recommend your content to users
• How many humans are clicking through to your website from AI citations
• Whether AI shopping agents can successfully complete purchases on your site
• How your pages rank in traditional and AI-powered search engine results
• How much server bandwidth and cost bot traffic consumes
• Which AI agents have accessed your content, for licensing and negotiation purposes

With AI adoption accelerating, bot traffic is becoming one of the most important segments to understand and optimize for.

Known Agents is a freemium service. The free tier gives you immediate visibility into your hidden bot traffic with no credit card required. You can upgrade to unlock higher event volumes, additional projects for multiple websites, CSV export, API access, and priority support. Full details are on the pricing page.

Yes. Creating an account is free and only takes a few seconds. This is where you’ll be able to see dashboards and insights into agent activity on your website.

Known Agents makes it easy. Once the plugin is installed and connected, go to the plugin settings and check the categories of bots you want to block, such as AI Data Scrapers, AI Assistants, or AI Search Crawlers. Known Agents will automatically generate and maintain robots.txt rules covering every agent in those categories, including GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (TikTok), and many more. The rules update automatically as new bots are discovered, so you never have to manage them manually.

GEO and AEO refer to the practice of optimizing your website to appear in AI-generated responses, such as answers from ChatGPT, Claude, Gemini, and Perplexity, rather than traditional search engine results pages. Known Agents supports GEO and AEO by showing you which pages AI platforms are citing most often and measuring how many humans click through to your website from those citations. This lets you double down on content that’s already working and identify gaps where AI platforms aren’t recommending you.

Known Agents tracks thousands of known AI agents, web crawlers, and scrapers by inspecting the user agent string and other signals on each request. This includes GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Googlebot, Bytespider (TikTok), CCBot (Common Crawl), and many more. See the full agent directory for the complete list.

Known Agents categorizes agents into distinct types to help you understand and control different kinds of bot traffic. These categories include AI Agents, AI Assistants, AI Data Providers, AI Data Scrapers, AI Search Crawlers, Archivers, Developer Helpers, Fetchers, Automated Agents, Intelligence Gatherers, Scrapers, SEO Crawlers, Search Engine Crawlers, Security Scanners, and Undocumented AI Agents. Each agent in our directory is classified into one of these categories.

Yes. Known Agents can work as a useful complement to content monetization platforms and standards like RSL. It gives you visibility into which agents are accessing your content and whether they’re complying with your licensing terms.