The AI-Readability Index

The gap is not access — almost nobody blocks the crawlers. The gap is substance. Only 1 of these 45 companies serves a plain-text version of its own homepage, so what a crawler takes away is a page of scripts and components with very little quotable text in it.

That is the whole opportunity. A machine-readable page describing a company costs nothing to produce and almost no one has one — while publishers like TIME are rebuilding entire sites in markdown to sell that exact surface at premium rates.

Three HTTP requests per domain: /llms.txt, /robots.txt and the homepage.

• llms.txt counts only when the path returns plain text. Several companies redirect it to their HTML homepage — that is not an llms.txt, and an earlier version of this scan wrongly counted four of them.
• Plain-text page counts when the homepage advertises a markdown alternate, or when /index.md returns non-HTML content.
• Crawler access is read from robots.txt: a root-level Disallow: / against the named agent, or against * when the agent is not named.

Anyone can repeat this in a few minutes. If a company here has since fixed its setup, tell us and we will re-run it — corrections are published, not quietly edited.

These companies are named because we measured their public files. None of them is a customer of this site, and nothing here implies any relationship with them.