Ahrefs and Backlinko Have No llms.txt: An AI Crawler Audit of the SEO Industry
Everyone sells GEO advice; almost nobody audits the layer that decides whether AI engines can read the site at all. I pointed a new zero-dependency CLI at three famous SEO sites and my own projects. Findings: ahrefs.com and backlinko.com both return 404 on llms.txt, subpath disallows get misread as full blocks by naive checks, and robots.txt allows do not survive Cloudflare's edge bot toggles. Real CLI output for every claim.
TL;DR — zens-ink v1.4.8 ships ai_crawler_audit: it reads your robots.txt, classifies 17 AI crawlers (training vs AI search, per vendor), tells site-wide blocks apart from subpath-only blocks, checks Content Signals, verifies the llms.txt / llms-full.txt layer, and scores AI readiness 0-100. First real runs: ahrefs.com scores 88 with no llms.txt at all, backlinko.com the same, searchenginejournal.com 96 with a 515-link llms.txt. The industry that sells GEO readiness is missing the cheapest layer of it. One command, no API key, no dependencies.
Ahrefs returns 404 on /llms.txt. So does Backlinko. I checked because I needed test targets for a new audit tool, and the two most famous names in SEO turned out to be missing the cheapest AI-discovery layer there is.
Search Engine Journal has one — 515 links, 225 KB. So this is not an industry-wide impossibility. It is an omission, and it is measurable in one command.
What the tool checks
zens-ink v1.4.8 ships ai_crawler_audit. It exists because “is my site ready for AI search” has a boring, checkable substrate that most GEO content skips over:
pip install --upgrade zens-ink
zens-ink ai_crawler_audit https://ahrefs.comIt fetches robots.txt, classifies 17 AI crawlers by vendor and purpose, reads Content Signals, probes /llms.txt and /llms-full.txt, and prints a score with plain-language notes. Zero dependencies, no API key.
The part naive checks get wrong
The interesting bug I hit while dogfooding: my first version read Disallow: /api/ as “GPTBot blocked”. It is not. A Disallow rule covers only the paths it names. Disallow: /admin leaves the rest of the site fully crawlable; only Disallow: / is a site-wide block. The shipped version distinguishes four states per crawler — blocked, partial, allowed, unlisted — because conflating them turns every real robots.txt into an alarm.
That distinction is the difference between “your CMS blocks AI crawlers, fix it now” and “your CMS hides /admin from crawlers, which is correct”.
Two layers that do not know about each other
The second thing the audit surfaces: robots.txt allows are not promises. One of my own sites allows every AI crawler in robots.txt but sits behind Cloudflare, whose “AI Scrapers and Crawlers” toggle 403s most of them at the edge. Both states are legitimate; believing you have the first while running the second is how sites lose AI citations for months without noticing.
The audit prints a note for exactly this case — robots.txt allows everything, verify the edge layer separately.
Real runs
$ zens-ink ai_crawler_audit https://ahrefs.com
robots.txt: HTTP 200 (908 bytes) · sitemap declared: no · signals: none
GPTBot partial * OpenAI training
OAI-SearchBot partial * OpenAI AI search grounding
PerplexityBot partial * Perplexity AI search answers
llms.txt: HTTP 404 · 0 bytes · 0 links
AI-readiness score: 88/100
- No llms.txt — AI engines discover content depth from it first.
$ zens-ink ai_crawler_audit https://searchenginejournal.com
robots.txt: HTTP 200 (2020 bytes) · sitemap declared: yes · signals: none
llms.txt: HTTP 200 · 225584 bytes · 515 links
AI-readiness score: 96/100The 88 vs 96 gap is almost entirely the llms.txt layer. Ahrefs’ robots.txt is fine — subpath rules only, every AI crawler can reach the content. The missing catalog is the whole difference.
What I would do with this
Run it against your own site today, then against the three competitors you care about. If a competitor has signals set, llms.txt live, and search crawlers allowed while you sit at 404 on all three, that is the cheapest gap you will close this quarter. Generate the missing layer with zens-ink llms_gen --sitemap https://yoursite/sitemap.xml, then re-run the audit to confirm.
The honest caveat: an llms.txt and an open robots.txt do not make AI engines cite you. They make you eligible. The 88-score sites are eligible too — they just made the discovery layer harder than it needs to be, and for an industry that sells this advice, that is the finding.
FAQ
Does Disallow: /admin in robots.txt block AI crawlers from my whole site?
No. A Disallow rule only covers the paths it names. Disallow: /admin blocks /admin/* while the rest of the site stays crawlable. Only Disallow: / is a site-wide block. Many naive audit tools treat any Disallow line as a full block and report false alarms — ai_crawler_audit distinguishes blocked (site-wide), partial (subpaths only), allowed, and unlisted per crawler.
If robots.txt allows GPTBot, does that mean OpenAI can crawl my site?
Not necessarily. robots.txt is one of two independent layers. Edge bot management — for example Cloudflare's AI Scrapers and Crawlers toggle — can return 403 to AI crawlers before your origin ever sees the request, regardless of what robots.txt says. If you want AI search to cite you, check both layers: the robots policy and the edge firewall.
What is llms.txt and do I need llms-full.txt too?
llms.txt is a markdown catalog at your site root that tells AI engines what your content is and where the important pages live. A two-tier setup works best: a curated llms.txt with core pages (tens of links) and llms-full.txt with the complete catalog grouped by section. The zens-ink CLI generates both from a static build directory or a live sitemap with zens-ink llms_gen, and zens-ink ai_crawler_audit verifies the layer after deployment.
Want to run this analysis on your own site?
ZensInk Pro automates this pipeline. One command, from seed keywords to content plan.
Get Pro →