ParseLess

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and a dozen others crawl WordPress sites constantly. Each request renders your full theme, runs widget queries, loads page-builder assets, and ships hundreds of KB of HTML per page — most of which the bot discards before extracting the actual text.

ParseLess intercepts these crawls and serves a tiny Markdown payload instead. You keep the AEO/SEO benefit of being indexed by AI search, without paying the server cost of rendering your full theme for every bot hit.

Typical impact on AI bot traffic (measured on real WordPress pages):

• 95%+ less bandwidth per crawled page (typically 15–30x smaller; e.g. a 79 KB page → 4 KB. Heavy page-builder sites see 100x+)
• ~60–80% fewer database queries per request
• ~60% lower peak PHP memory per request
• ~50–80% faster Time To First Byte (measured 54% on a content-heavy page, 67% on a short one)
• On a site with 500 posts crawled monthly by major AI bots, that’s roughly gigabytes → tens of megabytes of monthly bandwidth and a fraction of the cumulative PHP execution time

Numbers vary by theme and content. Heavier setups (Elementor, Divi, Avada) see the biggest savings; lightweight themes see less dramatic but still meaningful gains. The conversion is cached as a transient, so repeated bot hits cost almost nothing.

How it works

• AI crawlers are detected by User-Agent and served Markdown automatically — no configuration required.
• Humans, search engines, and unknown bots receive your normal HTML output. ParseLess never affects what real visitors see.
• The ?format=md query parameter works on any post URL for manual preview or CLI piping.
• /llms.txt is published at your site root with a list of available content for AI indexers.
• Conversion happens once per post and is cached. Subsequent requests serve a single transient read.

Features

• Automatic Markdown for known AI bots (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, and more), plus Accept: text/markdown , .md addresses and ?format=md
• Respects noindex flags from Yoast SEO, Rank Math, and Genesis; skips private, draft, password-protected and trashed posts; per-post “Exclude from AI” checkbox
• Works with all public post types (configurable); WooCommerce products include price, SKU, stock, attributes and variations
• Optional YAML frontmatter and schema.org Article block
• Analytics: requests per bot, company and purpose; chat impressions; top URLs; unknown bot-like User-Agents with one-click “add to list”; per-bot reachability with last seen; CSV export. Optional site-wide crawler logging to see bots that get HTML too.
• Controls: Blocked Bots (403), per-bot rate limit (429 + Retry-After ), Content Signals, AI bot rules and RSL licence in robots.txt — all off by default
• Abilities API registration for MCP clients on WordPress 6.9+
• /llms.txt , /llms-full.txt , an AI sitemap advertised in robots.txt, and an experimental OKF bundle
• Settings page at Tools → ParseLess ; per-post meta box with Markdown preview, copy-to-clipboard and the exclude toggle
• Zero dependencies, no build step, nothing phones home

For developers

The public API below is frozen for the 1.x series: names, arguments and return types will not change without a major version bump.

Filters

• md4ai_should_serve_markdown( bool $serve ) — final say on whether the current request gets Markdown.
• md4ai_bot_list( string[] $bots ) — User-Agent substrings treated as AI crawlers.
• md4ai_blocked_bots( string[] $bots ) — User-Agent substrings refused with a 403.
• md4ai_bot_taxonomy( array $map ) — bot_name => [ company, purpose ] ; purposes are training , search , live , agent .
• md4ai_rate_limit( int $per_minute, string $bot ) — per-bot request cap when rate limiting is on; return 0 to exempt a bot.
• md4ai_unknown_ua_tokens( string[] $tokens ) — substrings that make an unlisted User-Agent “bot-like” for the unknown-UA log.
• md4ai_supported_post_types( string[] $types ) — post types served as Markdown.
• md4ai_excluded_post_types( string[] $types ) — public post types never served (page-builder templates and the like).
• md4ai_excluded_post_ids( int[] $ids ) — individual posts never served on any endpoint.
• md4ai_markdown_output( string $markdown, WP_Post $post ) — the finished Markdown document.
• md4ai_product_details( string $table, WC_Product $product ) — the WooCommerce product details section.
• md4ai_max_variations( int $max ) — cap on listed variations (default 25).
• md4ai_cache_ttl( int $seconds, int $post_id ) — cache lifetime; $post_id is 0 for site-wide documents.
• md4ai_cache_bypass( bool $bypass ) — skip the cache for this request (default: logged-in users).
• md4ai_llmstxt_max_posts( int $max ) , md4ai_llmsfull_max_bytes( int $bytes ) , md4ai_sitemap_max_posts( int $max ) , md4ai_okf_max_posts( int $max ) — endpoint size caps.
• md4ai_okf_endpoint( string $slug ) — path segment of the OKF bundle.
• md4ai_okf_concept( string $markdown, WP_Post $post ) — one OKF concept file.
• md4ai_rsl_terms_url( string $url ) , md4ai_rsl_xml( string $xml ) — the RSL licence document.
• md4ai_log_request( bool $log, array $context ) — veto a log row before it is written.

Actions

• md4ai_logged_request( int $row_id, array $context ) — after a request row is written.

Abilities (WordPress 6.9+, via wp_register_ability() )

• parseless/get-markdown — input post_id or url ; returns post_id , url , title , markdown , tokens .
• parseless/list-markdown-urls — input limit (capped by the llms.txt Max Posts setting); returns post_id , title , url , markdown_url , modified per post.
• parseless/get-stats — input days (1–365); returns totals, cache hit rate, chat impressions, distinct crawlers and top bots. Requires manage_options .

Storage : one option, md4ai_settings ; one table, {prefix}md4ai_requests (only when logging is on); post meta _md4ai_exclude ; transients prefixed md4ai_ . Uninstall removes all of them.

Privacy and data collection

When logging is enabled (off by default), ParseLess records the following for each Markdown request:

• Post ID, URL, and request timestamp
• The full User-Agent header
• A salted SHA-256 hash of the requester’s IP address (the raw IP is never stored)
• Matched bot identifier, bytes served, and whether the response came from cache
• What the request received: Markdown, the normal HTML page, a block (403) or a throttle (429)

HTML page views are only recorded when “Log All AI Crawler Traffic” is also enabled (off by default), and only for User-Agents on your bot list. Human visitors are never logged.

IP hashes cannot be reversed from an email address, so the WordPress privacy exporter/eraser tools will report no personal data on demand.

Logs are pruned daily according to the configured retention window (7/30/90/365 days, default 30). Site owners can disable logging or click “Delete all logged requests” at any time from Tools → ParseLess.