MCP HeartbeatTHE MCP OBSERVATORY
Public workspace

Explore

Evidence for your next connection.

Independent measurements of remote MCP servers. Every observation has a timestamp.

Read-only probes

MCP server · Browser & web scraping

Robots.txt & AI Crawler Checker

By robots-check.quietworklabs.workers.dev Verified namespace

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

WebsiteSearch & web data
Connect to Robots.txt & AI Crawler Checker
Checks passed
13 / 22

1 failed · 8 not measured

Median response
81 ms

Median of successful probes over 7 days

30-day uptime
Not measured

Share of countable probes over 30 days, including daily rollups. Auth challenges are excluded.

Discovered tools
3

Tool definitions returned by this server

Independent monitoring

Health & checks

Status history

Responded

The server completed its last measured handshake. This is an observation, not a guarantee.

Last checked
Check summary by test group
Test areaPassedFailedUnmeasured
ConnectionCan a client connect and discover tools?
500
Reliability & speedDoes it respond consistently and quickly?
004
Access & authenticationWhat is required to use this server?
102
Tool qualityAre the tool definitions usable by an agent?
400
Publisher identityCan the publisher be traced to its source?
312

Unmeasured means there is not enough evidence to decide. How these checks work

Inspect individual checks 22 recorded

Core

  • Pass
    Reachable

    Latest probe returned HTTP 200 in 81 ms.

  • Pass
    Valid handshake

    protocolVersion 2025-11-25, serverInfo.name dev.workers.quietworklabs.robots-check/robots-check.

  • Pass
    Lists tools

    tools/list returned 3 tools.

  • Pass
    Current protocol

    Negotiated 2025-11-25. The bar is 2025-06-18 or newer.

  • Pass
    HTTPS

    The endpoint URL is https.

Performance

  • Not measured
    Fast (p50)

    Median latency 81 ms over 1 samples in 7 days. The bar is 1000 ms. Need at least 5 samples.

  • Not measured
    Consistent (p95)

    95th percentile 81 ms over 1 samples in 7 days. The bar is 3000 ms. Need at least 20 samples.

  • Not measured
    Reliable (uptime)

    1 of 1 attempts returned a tool list in 30 days (100%). The bar is 99%. Need at least 30 samples.

  • Not measured
    Stable tool list

    Fewer than two successful probes in 30 days, so tool-list stability is not measured.

Access

  • Pass
    Open access

    tools/list succeeded with no credential, and the record declares none.

  • Not measured
    Standard auth

    Auth is not required, so the OAuth metadata check does not apply.

  • Not measured
    Free tier cited

    No publisher quote has been checked.

Tools

  • Pass
    Tools described

    Every tool description is at least 20 characters.

  • Pass
    Valid input schemas

    Every inputSchema is a JSON Schema object.

  • Pass
    Parameters documented

    4 of 4 parameters have a description (100%). The bar is 80%.

  • Pass
    Safety hints

    Every tool sets readOnlyHint or destructiveHint.

Publisher

  • Pass
    Verified namespace

    Listed in the Official MCP Registry.

  • Pass
    Website matches publisher

    The website host matches the reverse-domain namespace.

  • Fail
    Source repo linked

    No repository URL is on file.

  • Not measured
    Repo owner matches

    This namespace is not an io.github login, so the owner comparison does not apply.

  • Pass
    Recently updated

    Updated 2026-10-18T03:32:52.238228Z, within 180 days.

  • Not measured
    Open-source licence

    No GitHub repository to ask about a licence.

Start using Robots.txt & AI Crawler Checker

Connect to this server

Choose your client, copy the configuration, then enable the server in your client’s MCP settings.

Run this command in your terminal, then restart Claude Code.

Claude Code

claude mcp add --transport http robots-txt-ai-crawler-checker https://robots-check.quietworklabs.workers.dev/mcp

Capabilities

Tools 3

ai_crawler_access

AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Discovered
robots_check

Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Discovered
sitemap_check

Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Discovered

Recent measurements

Probe history

View status
Latest stored probes. Raw probe rows are kept for 14 days.
CheckedResultResponseTools
Connected81 ms3

Related servers

All alternatives

Same function and industry, ranked by checks passed then median latency.

Explore alternatives to Robots.txt & AI Crawler Checker