SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory
Recent: cloudflare.com vercel.com stripe.com github.com openai.com trendyflash.com

/llms.txt Validator, AI Crawler Registry & Web-to-Token Observatory

Live HTTP/TLS socket diagnostics, HTML-to-LLM token footprint analysis, and crawler governance across production domains.

Hosts: 26 /llms.txt: 15/26 Mean TTFB: 249 ms

Agentic Web & Token Telemetry Summary

Live DOM, /llms.txt & Crawler Probes
Indexed Production Hosts 26 verified domains
/llms.txt Manifest Adoption 15 of 26 hosts (58%)
Mean HTML Signal-to-Noise 7.4% readable text vs. markup/script tokens
Flagship Research Report 2026 Web-to-Token Empirical Study (26 Hosts) →

Specification Linter & FinOps Utilities

Direct Tools
CMS /llms.txt Setup Guide 1-line <head> & 301 redirect guide for Shopify, Webflow, Framer, Wix & WP
/llms.txt Spec Builder Crawl any domain’s HTML head, nav links & sitemap to build an llms.txt file
AI robots.txt Builder Configure User-agent rules for OAI-SearchBot, GPTBot, ClaudeBot & CCBot
Cloud & Token Estimator Compare Cloudflare Workers, Vercel Pro, AWS Lambda & LLM API rates

Observed Host Registry (Web-to-Token Footprint · /llms.txt · Crawler Posture · TTFB)

Click any host to inspect full token economics & /llms.txt linter dossier
# Host Grade Score /llms.txt Raw HTML Tokens Clean Text Tokens Signal % AI Crawler Posture Edge / Stack TTFB
1 sitecube.com A+ 98/100 1,600 tok 11,832 1,132 9.6% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 99 ms
2 cloudflare.com A+ 96/100 4,294 tok 333,631 471 0.1% Open to Search & Training Cloudflare · Astro 291 ms
3 trendyflash.com A+ 95/100 2,987 tok 12,123 1,660 13.7% Open to Search & Training Hostinger Edge CDN · Next.js 112 ms
4 multifactor.com A+ 94/100 656 tok 98,275 673 0.7% Open to Search & Training Cloudflare · Next.js 97 ms
5 github.com A+ 92/100 7,165 tok 131,504 1,159 0.9% Open to Search & Training GitHub Pages · Custom SSR / Static 35 ms
6 vercel.com A+ 92/100 1,385 tok 149,859 889 0.6% Open to Search & Training Vercel · Next.js 53 ms
7 aiimpacted.org A+ 92/100 971 tok 17,598 3,568 20.3% Open to Search & Training Direct Origin · Custom SSR / Static 55 ms
8 tierones.io A+ 92/100 481 tok 3,744 693 18.5% Open to Search & Training Direct Origin · Custom SSR / Static 95 ms
9 cloq.com A 91/100 1,169 tok 2,109 690 32.7% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 51 ms
10 supabase.com A 89/100 879 tok 337,513 1,905 0.6% Open to Search & Training Vercel · Next.js 48 ms
11 dosefacts.org A 89/100 10,936 tok 29,008 7,291 25.1% Open to Search & Training Direct Origin · Custom SSR / Static 335 ms
12 postman.com A 89/100 5,375 tok 118,892 11,040 9.3% Open to Search & Training Cloudflare · Next.js 947 ms
13 linear.app A 87/100 2,903 tok 317,016 313 0.1% Open to Search & Training Cloudflare · Next.js 452 ms
14 gitlab.com A 85/100 — 56,566 1,903 3.4% Open to Search & Training Cloudflare · Nuxt 399 ms
15 stripe.com A 85/100 17,454 tok 172,347 3,523 2% Open to Search & Training Direct Origin · Next.js 503 ms
16 datadoghq.com B+ 83/100 150,542 tok 70,990 3,506 4.9% Open to Search & Training AWS CloudFront · Custom SSR / Static 276 ms
17 huggingface.co B+ 82/100 — 43,957 1,148 2.6% Open to Search & Training AWS CloudFront · Custom SSR / Static 69 ms
18 anthropic.com B+ 81/100 — 43,572 494 1.1% Open to Search & Training Cloudflare · Next.js 352 ms
19 mozilla.org B+ 79/100 — 11,971 656 5.5% Open to Search & Training Fastly · Custom SSR / Static 259 ms
20 microsoft.com B 77/100 — 44,551 859 1.9% Open to Search & Training Direct Origin · Custom SSR / Static 120 ms
21 figma.com B 77/100 — 466,946 8,864 1.9% Search Allowed · Training Restricted AWS CloudFront · Next.js 1210 ms
22 openai.com B 74/100 — 2,444 1 0% Open to Search & Training Cloudflare · Custom SSR / Static 27 ms
23 tailwindcss.com B 74/100 — 219,494 1,678 0.8% Open to Search & Training Cloudflare · Next.js 51 ms
24 shopify.com B 73/100 — 2,167 22 1% Open to Search & Training Cloudflare · Custom SSR / Static 20 ms
25 astro.build B 73/100 — 78,970 1,617 2% Open to Search & Training Cloudflare · Astro 80 ms
26 google.com B 70/100 — 791 258 32.6% Open to Search & Training Direct Origin · Custom SSR / Static 426 ms