Rat Radar

Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.

Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps and risks, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces (sitemap, llms.txt, JSON-LD), and eu_ai_training: the signals the EDPB draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data (robots.txt blocks on AI training crawlers, a Content-Signal of ai-train=no, ai.txt, CAPTCHAs, login walls), as none, some or strong. Signals, not legal advice. Uses a weekly-probed dataset of several hundred sites; for any other domain it probes live from a datacenter IP.

What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example "price alerts", "sales leads", "new jobs", "change log", "knowledge base"). Returns the best-matching workflow for that site: what it does, its cadence, a trigger, collect, shape, store, act pipeline, the fields to keep, and step-by-step build recipes for n8n, Make, Zapier and Apify (or just one with platform). Workflows marked found_on_this_site come from what the weekly probe saw on the site itself. Also returns the other workflows, the golden data, and the guardrails.

List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.

Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered before calling check_site.

Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.

Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page is made of. Returns a download link for the .excalidraw file (give it to the user; it opens at excalidraw.com or in Obsidian) and a file_name. With include_scene, also the scene itself (a few hundred KB).

How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated records with CSS selectors and field names, tables, feeds, hidden API calls. Each source has a reliability, proof rows pulled from the real page, and the plan notes pagination and blockers. Set include_code for a starter Python script. Works on the raw HTML only, so JavaScript-built pages report a blocker instead of data. Use check_site first to learn whether the site allows scraping at all.