First CH Apps / AI Crawler Access Checker
Enter a URL and we fetch its robots.txt, then rule on each major AI crawler one by one for that exact path. Matching follows Google's rules (longest match wins; ties go to Allow). We also report whether llms.txt exists and what X-Robots-Tag the page returns.
This app drills into one axis of GEO (AI search optimisation): crawler access. For a score that also covers structured data, llms.txt, E-E-A-T and page structure, use the GEO Checker.
Go to GEO Checker (six axes, 100 points) →Sent: only the URL you type. Our server (Cloudflare Workers) fetches three public resources — that URL, /robots.txt and /llms.txt on the same site (at most five outbound requests per check).
Stored: nothing. The check lives entirely inside one request and writes neither the URL nor the result to any database (no KV, no D1). It leaves memory when the request ends.
Returned: allow/deny verdicts, the Disallow pattern that matched, whether each file exists plus its byte size, and the X-Robots-Tag value (first 300 characters). We never re-serve the fetched robots.txt or page.
Note: public GET only. URLs pointing at internal networks, localhost or private IPs are rejected, and pages behind a login cannot be checked.
POST /ai-access/api/check with JSON ({"url": "https://example.com/"}) or a form body and you get the same verdicts back as JSON.
robots.txt is a voluntary convention: a Disallow asks a crawler to stay away, it does not stop the request. Blocking for real needs server-side user-agent rules.
UA ledger updated: 2026-09-11 · AI crawler tokens change a few times a year. The ledger lives in its own JSON file (ai-crawlers.json) and is swapped out on a yearly review.