🕷️ Free · no signup · no app install · read-only
Is your robots.txt telling AI shopping agents to go away?
A theme edit, an SEO app or a leftover pre-launch rule can quietly add a Disallow for the crawlers behind AI shopping answers. We read your robots.txt and your served HTML and tell you, user-agent by user-agent, which ones your site asks to stay away — and whether a crawler that never runs JavaScript can see your products at all.
Based on publicly available data only. Informational, not legal advice.
What this checks
- A named list of AI and search crawler user-agents, each evaluated against your home page and one real product URL taken from it
- A site-wide Disallow: / — the one line that asks every crawler to stay out of the whole store
- Whether /products/ is disallowed for the default User-agent: * group
- Whether /llms.txt exists, and whether it is a real text file rather than a 404 page
- Whether robots.txt declares a sitemap, and whether that address returns actual sitemap entries
- Whether product links and a price are in the HTML the server returns, or only appear after JavaScript runs
- A noindex in the home page meta tags or the X-Robots-Tag header
FAQ
If a crawler is allowed, does that mean ChatGPT will recommend my products?
No — and it is worth being precise about what we even measured. We read a text file that asks crawlers to behave a certain way. "Not disallowed" is the entry ticket, not the outcome, and not proof that any crawler came. This tool tells you whether you have disqualified yourself on a technicality; it cannot tell you how any model ranks, retrieves or cites you, and it never claims to speak for OpenAI, Anthropic, Perplexity, Google or anyone else.
Why do some lines say 'could not tell'?
We fetch pages the way a plain crawler does and never run JavaScript. If robots.txt does not load, or the storefront blocks our request, we mark the affected checks unknown with the reason attached instead of guessing a pass. robots.txt is also honour-system: it says what compliant crawlers are asked to do, not what every bot actually does.
Is blocking AI crawlers always wrong?
No, and we do not score it that way. Setting a training-only opt-out token such as Google-Extended can never fail this report: Google's own documentation says that token does not affect inclusion or ranking in Google Search, so it is a deliberate trade with a cost the vendor has stated. The rule we do weight heavily is one that disallows the crawlers documented as fetching a page because a shopper just asked about you — that one is rarely what anybody meant to do.
Want robots.txt unblocked, a real llms.txt shipped, and a monthly check that it stays that way?
We are a Shopify studio. The audit is free; the fix is yours to make — or ours to ship.
Other free tools
- HreflangAudit — Are your language versions helping each other, or competing in the same search results?
- LiquidSweep — Is your storefront printing a Liquid error at a customer right now?
- SEORegress — Did the redesign quietly break your SEO?
- MotionPause — Can a visitor stop the things on your store that move by themselves?
- See all 29 free tools