First CH Apps / robots.txt Tester
Enter a URL and a crawler name. We fetch that site's /robots.txt and decide whether the URL may be crawled, following Google's matching rules (longest match wins; ties go to Allow), showing the winning rule and the rules it beat with line numbers. Syntax warnings included.
Pick one from the list or paste a full user-agent string — we extract the token used for matching.
Once you know what to change, the robots.txt generator on tools.first-ch.com builds the file from a form — entirely in your browser, nothing uploaded — including AI-crawler directives.
Go to the robots.txt Generator (tools.first-ch.com) →To see GPTBot, ClaudeBot, PerplexityBot and the rest bot by bot in one table, use the AI Crawler Access Checker. This app is the drill-down for a single crawler.
Go to the AI Crawler Access Checker →Sent: only the URL and the user-agent name you type. Our server (Cloudflare Workers) fetches one public resource — /robots.txt on that site — using at most three outbound requests per test (two DNS lookups plus the file). The URL you are testing is never fetched itself.
Stored: nothing. The test lives entirely inside one request and writes neither the URL, the user-agent nor the result to any database (no KV, no D1). It leaves memory when the request ends.
Returned: the verdict, the winning rule's pattern and line number, syntax warnings (first 60 characters of the offending line) and up to five Sitemap lines. We never re-serve the fetched robots.txt.
Note: public GET only. URLs pointing at internal networks, localhost or private IPs are rejected, and sites behind a login cannot be tested.
POST /robots-tester/api/test with JSON ({"url": "https://example.com/admin/", "userAgent": "Googlebot"}) or a form body and you get the same verdict back as JSON.
robots.txt is a voluntary convention. A Disallow asks a crawler to stay away; it does not stop the request. Blocking for real needs server-side user-agent rules or authentication.
Path matching is case-sensitive (`/Admin/` and `/admin/` are different). Only `*` (any sequence) and a trailing `$` (end of URL) are treated as wildcards.
This tester follows Google's matching rules. Other crawlers may differ in the details, such as how they read `$` or pick a group.