/llms.txt Validator, AI Crawler Registry & Web-to-Token Observatory

Live HTTP/TLS socket diagnostics, HTML-to-LLM token footprint analysis, and crawler governance across production domains.

Hosts: 32 /llms.txt: 22/32 Mean TTFB: 325 ms

Agentic Web & Token Telemetry Summary

Live DOM, /llms.txt & Crawler Probes
Indexed Production Hosts 32 verified domains
/llms.txt Manifest Adoption 22 of 32 hosts (69%)
Mean HTML Signal-to-Noise 6.4% readable text vs. markup/script tokens
Flagship Research Report 2026 Web-to-Token Empirical Study (32 Hosts) →
Agentic Endpoint Specs HTML Token Optimization · /.well-known/mcp.json

Specification Linter & FinOps Utilities

All Engineering Guides →
What Is /llms.txt? (Spec) Markdown grammar, llms-full.txt vs robots.txt, and linter checklist
CMS /llms.txt Setup Guide 1-line <head> & 301 redirect guide for Shopify, Webflow, Framer, Wix & WP
/llms.txt Spec Builder Crawl any domain’s HTML head, nav links & sitemap to build an llms.txt file
AI robots.txt Builder Configure rules for 10 AI crawlers (OAI-SearchBot vs GPTBot & ClaudeBot)
Cloud & Token Estimator Compare Cloudflare Workers, Vercel Pro, AWS Lambda & LLM API rates

Observed Host Registry (Web-to-Token Footprint · /llms.txt · Crawler Posture · TTFB)

Click any host to inspect full token economics & /llms.txt linter dossier
# Host Grade Score /llms.txt Raw HTML Tokens Clean Text Tokens Signal % AI Crawler Posture Edge / Stack TTFB
1 sitecube.com A+ 98/100 2,099 tok 16,563 1,347 8.1% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 96 ms
2 cloudflare.com A+ 96/100 4,294 tok 333,631 471 0.1% Open to Search & Training Cloudflare · Astro 294 ms
3 trendyflash.com A+ 95/100 3,256 tok 12,230 1,739 14.2% Open to Search & Training Hostinger Edge CDN · Next.js 85 ms
4 multifactor.com A+ 94/100 656 tok 98,275 673 0.7% Open to Search & Training Cloudflare · Next.js 109 ms
5 tierones.io A+ 93/100 481 tok 3,744 693 18.5% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 30 ms
6 dosefacts.org A+ 93/100 10,613 tok 29,181 7,309 25% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 107 ms
7 aiimpacted.org A+ 93/100 1,355 tok 17,701 3,599 20.3% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 119 ms
8 github.com A+ 92/100 7,165 tok 131,479 1,185 0.9% Open to Search & Training GitHub Pages · Custom SSR / Static 26 ms
9 vercel.com A+ 92/100 1,385 tok 149,553 1,193 0.8% Open to Search & Training Vercel · Next.js 66 ms
10 render.com A+ 92/100 13,304 tok 190,754 2,018 1.1% Open to Search & Training Cloudflare · Next.js 306 ms
11 cloq.com A 91/100 1,169 tok 2,109 690 32.7% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 34 ms
12 postman.com A 91/100 5,375 tok 118,892 11,040 9.3% Open to Search & Training Cloudflare · Next.js 769 ms
13 neon.tech A 90/100 7,418 tok 110,734 1,629 1.5% Open to Search & Training Cloudflare · Next.js 106 ms
14 supabase.com A 89/100 879 tok 337,794 1,905 0.6% Open to Search & Training Vercel · Next.js 45 ms
15 shopify.com A 89/100 3,161 tok 172,970 513 0.3% Open to Search & Training Cloudflare · Shopify 84 ms
16 redis.io A 88/100 11,721 tok 427,364 11,796 2.8% Open to Search & Training Cloudflare · Next.js 130 ms
17 linear.app A 87/100 2,903 tok 316,944 313 0.1% Open to Search & Training Cloudflare · Next.js 769 ms
18 gitlab.com A 85/100 — 56,566 1,903 3.4% Open to Search & Training Cloudflare · Nuxt 404 ms
19 stripe.com A 85/100 17,454 tok 172,347 3,523 2% Open to Search & Training Direct Origin · Next.js 520 ms
20 fly.io A 85/100 1,463 tok 25,089 1,387 5.5% Open to Search & Training Fly.io Edge · Custom SSR / Static 876 ms
21 railway.app B+ 84/100 1,758 tok 49,064 1,728 3.5% Open to Search & Training Cloudflare · Custom SSR / Static 697 ms
22 huggingface.co B+ 82/100 — 43,545 1,148 2.6% Open to Search & Training AWS CloudFront · Custom SSR / Static 42 ms
23 anthropic.com B+ 81/100 — 43,572 494 1.1% Open to Search & Training Cloudflare · Next.js 356 ms
24 datadoghq.com B+ 81/100 150,542 tok 70,990 3,506 4.9% Open to Search & Training AWS CloudFront · Custom SSR / Static 464 ms
25 mozilla.org B+ 79/100 — 11,971 656 5.5% Open to Search & Training Fastly · Custom SSR / Static 252 ms
26 figma.com B 77/100 — 466,946 8,864 1.9% Search Allowed · Training Restricted Netlify Edge · Next.js 1002 ms
27 mongodb.com B 75/100 2,775 tok 319,238 2,450 0.8% Open to Search & Training AWS CloudFront · Next.js 1633 ms
28 openai.com B 74/100 — 2,444 1 0% Open to Search & Training Cloudflare · Custom SSR / Static 27 ms
29 tailwindcss.com B 74/100 — 219,497 1,678 0.8% Open to Search & Training Cloudflare · Next.js 50 ms
30 microsoft.com B 74/100 — 44,542 859 1.9% Open to Search & Training Direct Origin · Custom SSR / Static 351 ms
31 astro.build B 73/100 — 78,970 1,617 2% Open to Search & Training Cloudflare · Astro 112 ms
32 google.com B 70/100 — 791 258 32.6% Open to Search & Training Direct Origin · Custom SSR / Static 426 ms