Agentwatch

Fetch a website the way ChatGPT, Claude, Perplexity, Google and Copilot fetch it, and report what stops them. Checks robots.txt per crawler with the deciding line number, makes real requests as each crawler to catch firewall and CDN rules that robots.txt cannot express, and measures whether the page carries readable text without JavaScript. Use this before advising anyone on why their site does not appear in AI answers: the usual cause is one of these three and none of them are visible from the outside.

The robots.txt tokens used by the major assistants, separated by purpose: the ones that build the index answers come from, the ones that fetch a page live when a person asks, and the ones that only collect training data. Blocking the last group costs no visibility; blocking the first two removes a site from the answers. Use this before writing or editing a robots.txt.

Given the IP address and User-Agent from a log line, say whether that visit really came from the crawler it claims to be. Checks the address against the vendor's own published ranges where they exist, and falls back to forward-confirmed reverse DNS. On one ordinary server's logs, 14% of requests claiming to be Googlebot were not Google. Use this before anyone writes a firewall rule based on a user agent string, because that string is a claim and not proof.

Verify a Web Bot Auth signature (RFC 9421, Ed25519) on a captured request. Unlike an address check, this proves who sent that exact request: it fetches the signer's published key directory, rebuilds the signature base from the covered components, and checks the signature. Use it when a request carries Signature-Input and Signature headers. Returns 'unsigned' when they are absent, and 'cannot_verify' rather than 'invalid' whenever the limitation is on our side.