stillindexed SEO checkers

Indexability check for up to 20 URLs in one call. Order preserved. Invalid URLs become per-row errors; HTTP status stays 200. Counts against the 60 fetches/hour budget, one per URL.

Check robots.txt for the 13 tracked AI crawlers (ChatGPT search, Claude, Perplexity, Gemini grounding, and the training crawlers) and report, per crawler, whether the given URL's path is allowed or blocked, naming the deciding group and rule. Also probes the edge once as each of the three AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) against a browser request. Costs up to 5 fetches. Same check as https://stillindexed.com/tools/ai-crawler-checker.

Domain registration expiry via RDAP. Many ccTLDs have no RDAP; the tool says so instead of inventing an expiry.

Validate a robots.txt with RFC 9309 and Google's parsing rules. Give url to fetch the live file, or content to test a pasted one, plus proposed to diff a second version. For each path or URL (up to 100, or use_sitemap to read them from the file's Sitemap line) it reports allowed or blocked for Googlebot, Googlebot-Image, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot, with the deciding rule and its line number, plus lints (unsupported fields, 500 KiB limit, BOM, HTTP status semantics). With proposed, changes lists only the verdicts that flip. Same check as https://stillindexed.com/tools/robots-txt-tester.

Fetch a sitemap (or the host's /sitemap.xml) and report status, XML validity, and URL count.

TLS certificate for the host: days remaining, chain validity, hostname match.

Fetch one URL and report HTTP status, redirect chain, meta robots, X-Robots-Tag, canonical, title, H1, and whether it is indexable. Same check as https://stillindexed.com/tools/indexability-checker.

Fetch the same URL as Googlebot, Bingbot, GPTBot and a browser, then compare status, indexability directives and content. Costs 4 fetches.

Triage a domain that disappeared from Google. Checks nine causes in crawl-then-index order (domain registration via RDAP, TLS certificate, robots.txt status, robots.txt rule for the homepage, homepage status as Googlebot vs a browser, redirects off the domain, noindex in meta or X-Robots-Tag, canonical to another host, sitemap served and parseable) and returns each as ruled_out, suspect, confirmed or unknown with evidence. Costs 6 fetches. Same check as https://stillindexed.com/tools/deindex-triage.

Fetch the URL, robots.txt, sitemap (with index expansion), host variants, a four-persona crawler diff and TLS, then return a SKILL.md an agent can install for that host. Up to 12 fetches. Same generator as https://stillindexed.com/tools/seo-skill-generator.

Audit a sitemap: pass a sitemap URL, or a hostname to use the sitemap robots.txt declares. Samples up to 200 listed URLs (evenly across the file; up to 5 children of a sitemap index), fetches each with the stillindexedbot user agent, and flags non-200, redirected, noindex, blocked by robots.txt for Googlebot, and canonicalized elsewhere. Also reports identical and future-dated lastmod values. Takes up to 90 seconds. Separate allowance of 2 audits per IP per hour, not the 60-fetch budget. Same audit as https://stillindexed.com/tools/sitemap-audit.