Crawlune audits how easily your published content can be extracted, controls AI crawler access, and publishes machine-readable summaries. These features run locally and do not require an account or license. It does not measure or promise citations by answer engines. Hosted rewriting is available wit
PricingFree downloadThe WordPress.org download is free. Pro versions, add-ons or external services may cost extra.
CompatibilityRequires WordPress 6.4 and PHP 8.1.
SourceWordPress plugins · Nov 1, 2026
01 / Source details
Aboutthistool.
Crawlune audits how easily your published content can be extracted, controls AI crawler access, and publishes machine-readable summaries. These features run locally and do not require an account or license. It does not measure or promise citations by answer engines.
Hosted rewriting is available with a Crawlune license from WP Shelf. Verify your account email to start a one-time trial of 25 items on one site; no credit card is required. One rewritten opening, one heading rewrite or one grounded FAQ set uses one item. Additional one-time packs are available in your account. Local audits, llms.txt and crawler controls remain free without an account. Open WP Shelf → Crawlune → Settings for the trial and sign-in links, then paste your Crawlune license key into the License card.
Start the 25-item trial Plans and item rules
Implemented:
Per-agent AI crawler controls for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, distinguishing training crawlers from retrieval crawlers
Three named positions to start from — Open, “Answers yes, training no”, Closed — each setting every crawler and the matching content signals in one go
robots.txt directives generated from those rules
Optional refusal (403) for a crawler you blocked that requests pages anyway — off by default, and the requests keep being counted
A count of people arriving from ChatGPT, Claude, Perplexity, Gemini and Copilot — click-throughs, which is a floor under citations rather than a measure of them
Content signals (search, ai-input, ai-train) in robots.txt, which state what a page may be used for rather than who may fetch it — off until you set them, and a declaration rather than an enforcement
The same refusal of AI training repeated as a noai response header, a robots meta tag and a W3C TDM reservation at /.well-known/tdmrep.json
Markdown for agents: every page also available at its own address with .md on the end, and on request via Accept: text/markdown — off by default, and honouring the same exclusions as llms.txt
IndexNow submission when you publish or update a page — off by default, and honouring the same exclusions as llms.txt
An extractability audit scoring content on liftable answer blocks, question-shaped headings, external citations, quotable statistics, modification dates and thin content
A site-wide visibility report aggregating that across all published content, worst pages first
/llms.txt and /llms-full.txt, generated from your real content and refreshed when you publish
Admin screens for the report, a per-page work-list behind every number, crawler controls, llms.txt and licensing, plus REST routes for the same
Export the searched and sorted loaded Pages list across table pages as a private CSV, with saved measurements, manual steps and explicit freshness and scope limits. Search wording is included only when you request this download; the separate filter-wide export remains available without JavaScript.
Background analysis of your content, so none of the above is computed while you wait for a page to load
Structured data for your pages when no SEO plugin is already providing it
AI rewriting of a page’s opening into a liftable answer block, reviewed as a word-level diff before anything is applied
AI rewriting of section headings toward the question they answer, keeping each heading’s level and its existing link address
Generated FAQ data for pages that answer questions they never ask, where every proposed answer has to be text already on the page — anything else is discarded before you see it
Applying a rewrite without disturbing surrounding block markup, and reverting a run from a journal
License activation against the WP Shelf licensing service
Reviewing or applying an existing result does not generate another item. A service-review pause preserves unused items and existing results. Contact WP Shelf support, then explicitly retry after the review is complete.
Not included:
Citation tracking — whether a given engine actually quoted you. Counting people who click through from an assistant is implemented and is a different thing: it is a floor under citations, not a measure of them
Crawlune does not claim it will get your site cited by ChatGPT, Perplexity, Claude or Google AI Overviews. That is not measurable from inside WordPress and will not be claimed. What it reports is what it measured about your content.
External services
The content audit, crawler controls, structured-data checks and generated llms.txt files run on your site. External requests occur only for the optional features described below. Activating the plugin alone does not opt you into usage reporting or start generation.
1. IndexNow. With “Submit a page to IndexNow” on, publishing or updating a page sends that page’s address to https://api.indexnow.org/indexnow , together with your site’s host name and a verification key this site publishes at its own root. Nothing else goes with it — no content, no e-mail address, no license key. IndexNow is a shared endpoint operated by Microsoft on behalf of Bing, Yandex, Seznam and Naver; the same URL is submitted to all of them once. At most one submission per address per hour. Service information and terms: https://www.indexnow.org/faq Privacy information: https://www.microsoft.com/privacy/privacystatement
2. WP Shelf licensing and generation. A WP Shelf account and a product-specific license are required to use the hosted generation service; the local features above remain free without one.
Service host: https://gloty-api.wp-shelf.com , operated by WP Shelf. A developer can override it with CITELY_SERVICE_URL for a self-hosted service.
Licensing requests send the license key, site URL, product identifier and, on activation, plugin version. They occur when you activate or deactivate a license and when cached license status is refreshed. These requests are separate from starting a rewrite.
Starting a rewrite sends the original passage or heading, the page title, URL and post type, language, other headings, bounded body or section text, available audit signals, rewrite constraints, item identifiers and a batch identifier, together with the site URL and license key. A developer can change context through the citely_rewrite_context filter.
The plugin sends batches after you start a run and polls for results while it is running. If enabled on the service, results may also arrive through an authenticated callback. Merely reading an audit or serving a page does not send its content for generation.
The generation implementation uses OpenAI to process content. The plugin calls WP Shelf, not OpenAI directly. License keys are for WP Shelf authentication, not part of the provider’s generation prompt.
3. Optional usage reporting. Only after you enable usage reporting in Settings, a weekly background request sends the payload below to https://gloty-api.wp-shelf.com/v1/telemetry (or your configured service host). It is operated by WP Shelf under the same terms and privacy policy. Turning it off stops future reports. It is independent of license activation and rewriting.
What the optional report sends
No usage report is sent unless you enable it in Settings. Its JSON payload contains the product identifier and the following data:
How many AI crawlers you block, and how many of those are search crawlers — as counts. Which crawlers you chose is your editorial decision and is never sent
Whether WordPress’s “discourage search engines” setting is on
Whether you publish an llms.txt
Once you have run an audit: how many pages were analysed, the median score, and the component counts behind it. A site that has never run one sends no scores at all rather than zeroes — an unmeasured site is not a badly structured one
Your WordPress, PHP and Crawlune versions, your site’s locale, and whether WordPress considers the site production, staging or local
A one-way hash identifying this installation, salted with a random value generated on your site and never sent. It lets one site’s weekly reports be counted as one site; it cannot be turned back into an address
Not included in the usage-report payload: your site address, any e-mail address, any user account, any page, heading, answer or other content, any URL of yours, which specific crawlers you block, and your license key. Turning the setting off stops the weekly job immediately; uninstalling deletes the salt, so a later reinstall is a different installation as far as we can tell. As with any HTTPS request, the receiving server can see your server’s network address; the payload does not make the connection anonymous.
E-mailed reports: the separate scheduled content report uses your site’s WordPress mail configuration and the recipient you choose. Its delivery may use your hosting or SMTP provider; Crawlune does not send that mail through WP Shelf.
Why we ask: how many real sites arrive with a search crawler blocked decides whether the crawler screen is the front door of this plugin or a footnote in it, and the median score in the wild is the only honest basis for saying what a typical site’s gap is. Both are currently guesses.
Full documentation, including every component of the extractability score and what this plugin cannot measure: https://wp-shelf.com/crawlune/docs
Keeping your data when you uninstall
Deleting the plugin removes everything it stored, which is the right default — a plugin you have deleted should not leave rows behind. But it also removes your score history, which is a measurement series that cannot be recreated for months already past.
If you are deleting the plugin to troubleshoot and intend to reinstall, add this to wp-config.php first:
define( 'CITELY_KEEP_DATA', true );
Uninstalling then leaves your data in place, and a reinstall picks it up where it left off. Rem
See it in action
Screenshots & previews.
Demo data — a real local audit of four invented pages, showing the median extractability score and the content signals behind it. These measurements do not predict rankings or citations.Demo data — hand-authored suggestions stored in the real rewrite review queue. Read or edit a suggestion, approve it, then use the separate publish control. No AI provider was called for this example.Demo data — the no-license Rewrite screen explains the optional hosted trial: 25 items once on one site, email verification and no credit card. The local audit, llms.txt and crawler controls remain free.
Website LLMs.txt and LLMs-Full.txt Generator is an LLMs.txt generator and llms-full.txt generator for WordPress. It helps site owners generate, manage, preview, validate, and serve AI discovery files from published public content. Use it to create structured /llms.txt and optional /llms-full.txt end
EngageDock AI is a customizable floating contact widget for WordPress websites that want more leads and faster visitor communication. It helps small businesses turn website traffic into calls, texts, emails, and callback requests from one clean contact panel. Key Features: Click-to-Call – One tap ca
Altify AI is an auto ALT text and AI ALT text plugin for WordPress images. It helps generate descriptive image ALT text for accessibility and image SEO. Use bulk image ALT text tools for your Media Library, generate featured image ALT text automatically, and scale WooCommerce image ALT updates with
The Accepted Payment Methods plugin is a straightforward, lightweight solution that enhances your website by displaying visually appealing icons of the payment methods you accept. This plugin is ideal for a variety of websites, including e-commerce platforms, service-oriented websites, and informati
