Directory · Best Browser & web scraping
Browser & web scraping
Function. Browse, crawl, or scrape pages. Ordered by checks passed.
| Server / capability | Last probe | Median latency | Checks passed | Last checked |
|---|---|---|---|---|
CitedBy Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only. | Responded | 60 ms | 15 / 22 | |
upGrowth Growth Toolkit 29 free read-only growth tools: CAC, ROAS, PPC, schema, meta tags, robots.txt and AI crawler checks. | Responded | 113 ms | 15 / 22 | |
ActableSite AI Crawler Monitor Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes. | Responded | 251 ms | 15 / 22 | |
mcp-my-site Search and fetch tools for one website, served from an index you build with the mcpmysite crawler. | Responded | 771 ms | 15 / 22 | |
Spree Commerce developer docs for AI agents Spree Commerce docs for AI coding agents: build headless B2B and marketplace ecommerce on its APIs. | Responded | 1009 ms | 15 / 22 | |
Octocrawl Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record. | Responded | 1176 ms | 15 / 22 | |
Wincher MCP by usefulapi Read Wincher keyword rankings, SERPs, competitors and history; add keywords, groups and notes. | Responded | 56 ms | 14 / 22 | |
AIReady Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free. | Responded | 60 ms | 14 / 22 | |
Crawl Census Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? | Responded | 63 ms | 14 / 22 | |
WebsiteIQ Robots-aware website audits, pre-crawl quotes, and structured evidence for people and agents. | Responded | 100 ms | 14 / 22 | |
Proven Plaza Calculators & Open Data 29 free construction calculators plus Florida contractor, permit, federal award and AI-crawler data. | Responded | 102 ms | 14 / 22 | |
Site SEO & AI Crawler Audit Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion. | Responded | 210 ms | 14 / 22 | |
Business Site Audit Audit a business website for observed crawl and indexing issues with source-linked fixes. | Responded | 221 ms | 14 / 22 | |
agentbuilt Free AI-readiness audit of any URL: AI crawler rules, JS-free text, JSON-LD, llms.txt. Tool catalog. | Responded | 465 ms | 14 / 22 | |
Writ Cloud Read, crawl and act on websites, signed in as the user, and turn any site into an API. | Responded | 693 ms | 14 / 22 | |
com.see-geo/seegeo Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check. | Responded | 1159 ms | 14 / 22 | |
No Niner Data Tools No-code web scraping & data-extraction Apify actors: scrape and enrich web data at scale. | Responded | 477 ms | 14 / 22 | |
Browserless Browserless MCP — wraps the Browserless headless-Chromium REST API (browserless.io) | Responded | 6152 ms | 14 / 22 | |
Apify Apify MCP — run web-scraping Actors and fetch their dataset results. | Responded | 6162 ms | 14 / 22 | |
Decodo Decodo MCP — wraps the Decodo Web Scraping API (decodo.com, formerly Smartproxy). Rotating… | Responded | 6170 ms | 14 / 22 | |
WebLens Scrape, crawl, map and extract the web. Pay per call in USDC, no account or API key. | Responded | 60 ms | 13 / 22 | |
HolicAudit website checks Free SEO, AI visibility (AI crawlers, llms.txt, JSON-LD) and security-header checks for any URL. | Responded | 72 ms | 13 / 22 | |
Myworker — 20 paid crypto & AI micro-APIs (x402) 20 paid micro-APIs: token price, DeFi APY, FX rates, web scraping. Pay USDC per call. | Responded | 76 ms | 13 / 22 | |
Robots.txt & AI Crawler Checker robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check. | Responded | 81 ms | 13 / 22 |