MCP HeartbeatTHE MCP OBSERVATORY
Public workspace

Explore

Evidence for your next connection.

Independent measurements of remote MCP servers. Every observation has a timestamp.

Read-only probes

MCP server · Code & DevOps

IA-QA — 152 QA & Dev Tools for AI Agents

152 free QA tools for agents: injection scan, evals, RAG, web audits. Smaller packs: /mcp/{pack}.

Checks passed
13 / 22

1 failed · 8 not measured

Median response
249 ms

Median of successful probes over 7 days

30-day uptime
Not measured

Share of countable probes over 30 days, including daily rollups. Auth challenges are excluded.

Discovered tools
153

Tool definitions returned by this server

Independent monitoring

Health & checks

Status history

Responded

The server completed its last measured handshake. This is an observation, not a guarantee.

Last checked
Check summary by test group
Test areaPassedFailedUnmeasured
ConnectionCan a client connect and discover tools?
500
Reliability & speedDoes it respond consistently and quickly?
004
Access & authenticationWhat is required to use this server?
102
Tool qualityAre the tool definitions usable by an agent?
400
Publisher identityCan the publisher be traced to its source?
312

Unmeasured means there is not enough evidence to decide. How these checks work

Inspect individual checks 22 recorded

Core

  • Pass
    Reachable

    Latest probe returned HTTP 200 in 249 ms.

  • Pass
    Valid handshake

    protocolVersion 2025-11-25, serverInfo.name ia-qa-toolbox.

  • Pass
    Lists tools

    tools/list returned 153 tools.

  • Pass
    Current protocol

    Negotiated 2025-11-25. The bar is 2025-06-18 or newer.

  • Pass
    HTTPS

    The endpoint URL is https.

Performance

  • Not measured
    Fast (p50)

    Median latency 249 ms over 1 samples in 7 days. The bar is 1000 ms. Need at least 5 samples.

  • Not measured
    Consistent (p95)

    95th percentile 249 ms over 1 samples in 7 days. The bar is 3000 ms. Need at least 20 samples.

  • Not measured
    Reliable (uptime)

    1 of 1 attempts returned a tool list in 30 days (100%). The bar is 99%. Need at least 30 samples.

  • Not measured
    Stable tool list

    Fewer than two successful probes in 30 days, so tool-list stability is not measured.

Access

  • Pass
    Open access

    tools/list succeeded with no credential, and the record declares none.

  • Not measured
    Standard auth

    Auth is not required, so the OAuth metadata check does not apply.

  • Not measured
    Free tier cited

    No publisher quote has been checked.

Tools

  • Pass
    Tools described

    Every tool description is at least 20 characters.

  • Pass
    Valid input schemas

    Every inputSchema is a JSON Schema object.

  • Pass
    Parameters documented

    490 of 545 parameters have a description (90%). The bar is 80%.

  • Pass
    Safety hints

    Every tool sets readOnlyHint or destructiveHint.

Publisher

  • Pass
    Verified namespace

    Listed in the Official MCP Registry.

  • Not measured
    Website matches publisher

    This namespace is not a reverse domain, so the website comparison does not apply.

  • Pass
    Source repo linked

    A repository URL is on file.

  • Fail
    Repo owner matches

    The GitHub repository owner does not match the namespace login.

  • Pass
    Recently updated

    Updated 2026-10-26T21:30:38.392237Z, within 180 days.

  • Not measured
    Open-source licence

    The GitHub licence lookup has not returned an OSI result yet.

Start using IA-QA — 152 QA & Dev Tools for AI Agents

Connect to this server

Choose your client, copy the configuration, then enable the server in your client’s MCP settings.

Run this command in your terminal, then restart Claude Code.

Claude Code

claude mcp add --transport http ia-qa-152-qa-dev-tools-for-ai-agents https://www.ia-qa.com/mcp

Capabilities

Tools 153

Showing 100 of 153 tools.

ab_test_report

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

Discovered
analyze_diff_bugs

Pattern-based diff linter: flags a fixed set of risky shapes in changed code — query-string interpolation (SQL/Cypher/Mongo injection shape), shell interpolation, eval/new Function, empty catch blocks, regex built from a variable, fewer catch blocks than before, and named authorization guards that disappeared. Every finding cites the line that produced it. It does NOT do data-flow analysis: it cannot follow a value to a sink, across functions or files, and an empty result is not a safety verdict (the response lists what it did not analyse). Advisory triage — use a static analyser for a real security gate.

Discovered
analyze_responses

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.

Discovered
base64_decode

Decode a Base64 string back to UTF-8 text. Use for inspecting Base64-encoded API responses, JWT payload claims, config file values, or attachment data.

Discovered
base64_encode

Encode a UTF-8 string to Base64. Use when you need to embed binary data, multi-line text, or special characters safely inside JSON fields, HTTP headers, or data URIs.

Discovered
bias_detect

Analyse a set of LLM responses generated from the same prompt template but with different demographic variants (gender, origin, age, tone). Returns a bias score (0-100), sentiment analysis per variant, pairwise Jaccard similarity, and a human-readable verdict. No API key needed — runs entirely locally.

Discovered
bm25_score

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

Discovered
build_rag_prompt

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

Discovered
calculate_readability

Calculate readability scores: Flesch Reading Ease, Flesch-Kincaid Grade Level, Coleman-Liau Index, and Automated Readability Index. Useful for evaluating LLM output quality.

Discovered
case_convert

Convert a string between naming conventions: camelCase, PascalCase, snake_case, kebab-case, UPPER_SNAKE_CASE, dot.case, Title Case. Essential for code generation and refactoring.

Discovered
check_contrast_ratio

Calculate WCAG 2.1 contrast ratio between two colors. Returns ratio and compliance for AA/AAA normal and large text.

Discovered
color_convert

Convert a color between HEX, RGB, and HSL formats. Use when translating design tokens between CSS notations, verifying color accessibility, or normalizing color values from user input. Accepts #rrggbb, #rgb, rgb(r,g,b), or hsl(h,s%,l%).

Discovered
compare_models

Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.

Discovered
compare_responses

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

Discovered
consistency_check

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.

Discovered
context_window_check

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

Discovered
conversation_analyze

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

Discovered
cookie_security_audit

Audit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, missing Secure flag, SameSite=None without Secure, overly broad Domain, and excessive TTL. Returns per-cookie grades and an overall security score (0–100) — the score is the WEAKEST cookie, not the average, so one leaking session cookie cannot be averaged into a green result (average_score is reported separately).

Discovered
cors_checker

Would a browser on this origin be allowed to read this URL? Verdict allowed | blocked | unknown, the reason, and the header to add. Only GET and HEAD are really sent: other methods measure the preflight alone. Many origins at once: cors_test.

Discovered
cors_test

Audit a URL for CORS misconfigurations across several origins: wildcard with credentials, reflected origin, null origin, subdomain bypass, missing Vary. Returns risk level (safe/low/medium/high/critical) plus per-origin results. "unknown" means nothing was tested: every origin failed to connect, answered 5xx, or was refused with no CORS header (a 404, a firewall); never read it as "safe".

Discovered
cot_analyzer

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Discovered
count_code_lines

Count lines of code: total, code lines, comment lines, blank lines, and comment density. Supports JS/TS, Python, Java/C/C++, Ruby, Go, Shell, HTML/XML, and CSS.

Discovered
count_tokens

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.

Discovered
create_confluence_page

Create a new Confluence page from the output of jira_to_test_suite. Formats Gherkin, E2E steps, API tests, and test data as a properly structured Confluence page with code blocks and tables. STATEFUL — creates a new page in the specified space.

Discovered
cron_parse

Parse a cron expression into a human-readable schedule description. Supports standard 5-field cron (minute hour day month weekday).

Discovered
cron_validator

Validate a 5-field cron expression, explain the schedule, and preview the next execution times. Use this to debug cron jobs before they reach production. Returns parsed fields, a human-readable description, and upcoming ISO timestamps.

Discovered
decode_jwt

Decode a JWT (JSON Web Token) and return its header and payload without verifying the signature. Also reports whether the token is expired and the exact expiry date. Use to inspect claims (sub, iss, exp, roles) during debugging or when integrating with an auth provider.

Discovered
detect_language

Detect the natural language of a text using n-gram frequency analysis and common word markers. Supports 15 languages: English, French, Spanish, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, Korean, Arabic, Polish, Turkish, Swedish.

Discovered
detect_secrets

Scan code or config files for hardcoded secrets: AWS keys, GitHub tokens, OpenAI/Anthropic API keys, Stripe secrets, JWTs, Slack/Teams/Discord webhook URLs, session cookies, database connection strings, and generic passwords. Returns findings with severity.

Discovered
diff_mappings

Diff a baseline page mapping against a current one and return a CI-style verdict: PASS / FIX / BLOCK, plus per-element drift (ok, renamed, healable, ambiguous, lost, added, rebound). Pure and deterministic — provide two mappings as JSON with "elements" arrays of {role, name, selector, context?}. Use the companion @ia-qa/self-healing package (npm install -g @ia-qa/self-healing) to capture mappings from your app via its local MCP server ia-qa-heal-mcp, or paste the snippet from ia-qa.com/devtools/selector-drift into your browser console.

Discovered
diff_text

Compute a unified line-by-line diff between two text strings (LCS algorithm). Returns added/removed/unchanged line counts and formatted diff hunks with configurable context lines (0–20). Use to compare versions of prompts, configs, code snippets, or any text where you need to see exactly what changed.

Discovered
embedding_similarity

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

Discovered
env_parse

Parse a .env file content into a JSON object. Handles quoted values (single and double), inline comments, export prefix, and escaped sequences (\n, \t inside double quotes). Returns all key-value pairs. Use in CI/CD pipelines, agent config loaders, or when processing dotenv files programmatically.

Discovered
escape_html

Escape HTML special characters (&, <, >, ", ') to their safe HTML entities. Escaped text inserted into HTML cannot run as script (XSS).

Discovered
estimate_llm_cost

Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.

Discovered
extract_json_from_text

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.

Discovered
extract_json_path

Extract a value from a JSON string using dot-notation path (e.g., "user.address.city", "items.0.name", "meta.tags"). Supports array index access via numeric path segments.

Discovered
extract_links

Extract all URLs, email addresses, and domain names from text. Returns categorized and deduplicated results. Useful for content auditing, link checking, and web scraping validation.

Discovered
extract_todos

Extract TODO, FIXME, HACK, BUG, NOTE, OPTIMIZE, and custom tags from any source code or text. Returns line numbers, tag types, and message text. Essential for technical debt auditing.

Discovered
fetch_confluence_page

Fetch a Confluence page and return its content as clean Markdown. Accepts a numeric page_id or a full page URL. Optionally lists direct child pages. BYOK — credentials transit in-memory only, never stored.

Discovered
fetch_jira_issue

Fetch a complete Jira issue: summary, description converted to Markdown, status, assignee, priority, labels, custom fields, and optionally comments and attachment metadata. BYOK — credentials transit in-memory only, never stored on ia-qa.com.

Discovered
fetch_veille_feed

Fetch the latest QA & AI/LLM articles aggregated from curated RSS sources (Google Testing Blog, DEV.to Testing/QA/AI/LLM/Agents, Hugging Face Blog, Simon Willison). Each article carries summary_source — the XML tag the summary was read from, or "none" when the feed publishes titles and links only; an empty summary with summary_source "none" is a property of that feed, not a parse failure.

Discovered
few_shot_formatter

Format few-shot examples for LLM prompts. Converts example pairs into formatted blocks. Supports chat format (User/Assistant), XML tags, Markdown, or plain text.

Discovered
find_tool

Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.

Discovered
fix_gherkin

Fix Gherkin syntax warnings from a jira_to_test_suite result. Takes the current gherkin text and the _gherkin_warnings array, calls your LLM to fix ONLY the flagged issues (adds missing Given/When/Then steps, etc.), and returns the corrected Gherkin. Lightweight — uses ~300-500 tokens vs ~5k for a full regeneration. Requires BYOK LLM key.

Discovered
flatten_json

Flatten a nested JSON object to single-level dot-notation keys (e.g. {"a":{"b":1}} → {"a.b":1}), or unflatten dot-notation keys back to a nested object. Supports custom separators.

Discovered
format_bytes

Convert raw byte counts to human-readable sizes in SI (KB=1000) or IEC (KiB=1024) units, or parse size strings back to bytes. Covers B, KB/KiB, MB/MiB, GB/GiB, TB/TiB, PB/PiB.

Discovered
format_json

Validate and pretty-print a string that is ALREADY valid JSON. Strict by design — it is a validity gate: valid JSON comes back formatted, anything else is rejected with the exact parse error. It never repairs, completes, or guesses. NOT for: plain text or prose (will fail), JSON embedded in markdown/prose (extract_json_from_text pulls it out), JS objects (JSON.stringify them first), YAML (yaml_to_json reads it).

Discovered
format_table

Convert a JSON array of objects into a Markdown table. Automatically detects columns, aligns headers, and fills missing keys with empty cells. Use when an agent needs to present structured data — tool results, model comparisons, test reports — as a readable table in a response or document.

Discovered
function_call_validate

Validate an LLM function call / tool_use output: check that function name is in allowed list, arguments match expected schema (types, enums, ranges), no extra/missing args. Reads the definitions as OpenAI (`parameters`), MCP (`inputSchema`) or Anthropic (`input_schema`) write them, or as plain `required_args`/`optional_args`; reads the call as `arguments` (object or JSON string), `input` (Anthropic tool_use) or an OpenAI `{function:{…}}` wrapper.

Discovered
generate_ci_workflow

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.

Discovered
generate_curl

Generate a curl command from request parameters. Supports GET/POST/PUT/DELETE, custom headers, JSON body, and form data. Useful for documentation, sharing, and debugging API calls.

Discovered
generate_eval_yaml

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Discovered
generate_hmac

Compute an HMAC signature for a message using a secret key. Supports SHA-256 (default), SHA-512, SHA-1, and MD5. Used for API request signing, webhook verification (GitHub, Stripe, Twilio), and JWT validation.

Discovered
generate_html_report

Convert a run_eval_contract() LLM Test Runner JSON result into a fully self-contained dark-themed HTML report with Pass/Fail badges, side-by-side Input/Output/Ground-Truth panels, evaluator score bars, and a radar chart. Returns the HTML as a string.

Discovered
generate_json_ld

Generate a ready-to-paste <script type="application/ld+json"> snippet for GEO / structured data optimization. Supported types: WebSite, FAQPage, Article, Person, Organization, SoftwareApplication, HowTo.

Discovered
generate_password

Generate a cryptographically secure random password using crypto.randomBytes. Configurable length (4–128), uppercase letters, digits, and symbols. Use when resetting user passwords, seeding test accounts, or generating API secrets.

Discovered
generate_slug

Convert any string into a URL-friendly slug: lowercase, ASCII-normalized (é→e), special characters removed, spaces replaced with hyphens. Use for generating SEO-friendly URL paths, file names, or identifier keys from user-provided titles or labels.

Discovered
generate_test_cases

Generate a set of test cases (valid, edge, invalid, pairwise) for a given feature description. Declared constraints drive the boundaries: a length or numeric bound ("[8-64]", "min 8 chars", "at least 8 characters" — read from inputs, and from the feature prose when a sentence names exactly one field) yields the last accepted value AND the first rejected one; a format (email/url/uuid, from the type, the field name or the wording) yields malformed-value cases. A bound nobody declared is labelled as this tool's assumption, not as expected behaviour. gherkinFormat renders every case (cap 200, stated in the output) and gherkinScenarioCount lets you check it against totalCases.

Discovered
generate_uuid

Generate one or more cryptographically random UUID v4 identifiers. Use this when you need unique IDs for test fixtures, database records, session tokens, or any scenario requiring a guaranteed-unique string. Returns up to 100 UUIDs in one call.

Discovered
get_testing_guidelines

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

Discovered
guardrail_test

Test an LLM response against a set of guardrail rules: must-include, must-not-include, max length, required format, language, forbidden patterns, and custom regex. Returns pass/fail per rule.

Discovered
hallucination_check

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.

Discovered
hash_text

Compute a cryptographic hash of a text string. Use when you need to verify data integrity, generate content fingerprints, hash passwords (prefer SHA-256+), or produce a fixed-length digest of any input. Supports SHA-256 (default), SHA-512, SHA-1, and MD5.

Discovered
html_to_markdown

Convert HTML to clean Markdown. Strips scripts, styles, nav, ads, and comments. Converts headings, lists, links, images, code blocks. Ideal for preparing web content as LLM context.

Discovered
http_status_lookup

Look up detailed information about any HTTP status code: class, name, description, cacheability, typical causes, and handling best practices. Covers every code in the IANA HTTP Status Code Registry (1xx-5xx, including 226, 425, 451, 508, 511 and the WebDAV codes) with its defining RFC; anything outside the registry is reported as `registered: false` rather than described. `cacheable` means heuristically cacheable by default per RFC 9110 §15.1.

Discovered
identify_caller

Returns what the server knows about the current MCP client: clientInfo captured during initialize, User-Agent, and any _meta fields sent with this request. Useful for debugging caller identification.

Discovered
jira_to_test_suite

Turn a Jira ticket into a test suite: Gherkin scenarios, E2E steps, API tests, test data and the ambiguities to clarify. Pass the issue object, or Jira credentials to fetch it. It never invents: a thin ticket returns an empty suite with verdict INCOMPLETE_SPEC and the questions to ask. test_suite._gherkin_warnings is a deterministic syntax check. BYOK: your LLM key.

Discovered
json_diff

Compute a deep structural diff between two JSON values. Returns added, removed, and changed keys with dot-notation paths. Like git diff but for JSON objects, for API response regression testing. Arrays are not compared blindly by position: the same elements in a different order collapse to a single "reordered" change, and an array of records sharing a stable identity field (id, uuid, key, name…) is matched by that field, so paths read [id=42] and a moved record is not reported as N rewrites.

Discovered
json_schema_generate

Infer a JSON Schema (draft-07) from a sample JSON value. Detects types, required fields, array item shapes, nested objects, and common string formats (email, uri, date, date-time, uuid). Returns a ready-to-use schema compatible with json_schema_validate. Use when you have a sample API response or LLM output and want to auto-generate a validation schema for CI/CD testing.

Discovered
json_schema_validate

Validate a JSON value against a JSON Schema (draft-07 subset). Supports type, required, properties, items, enum, const, pattern, format (email/uri/date/date-time/uuid), minimum/maximum, minLength/maxLength, minItems/maxItems, uniqueItems, additionalProperties, anyOf, allOf, oneOf, not, and local $ref (#/definitions/…, #/$defs/…). A schema using a keyword outside that list (patternProperties, if/then, dependencies, a remote $ref…) is never reported valid: the keyword is named in `unevaluated_keywords`. Returns all validation errors with dot-notation paths.

Discovered
json_to_csv

Convert a JSON array of objects to CSV format. Automatically detects columns from all object keys. Handles quoting and escaping per RFC 4180.

Discovered
json_to_yaml

Convert a JSON object to clean, human-readable YAML. Handles nested objects, arrays, multiline strings, and special characters. No external dependencies.

Discovered
latency_benchmark

Measure response time of one or more HTTP endpoints (GET/POST). Runs N iterations and returns min/max/avg/p95 latency. Useful for API and MCP server benchmarking.

Discovered
levenshtein_distance

Compute the Levenshtein (edit) distance and normalized similarity ratio between two strings. Supports batch comparison. Useful for fuzzy string matching, deduplication, and test result comparison.

Discovered
lint_commit_message

Validate a git commit message against the Conventional Commits spec (feat, fix, docs, style, refactor, test, chore, ci, perf, build). Returns compliance score, breaking change detection, and actionable suggestions.

Discovered
list_llm_models

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

Discovered
list_local_tests

Discover .ia-eval.yaml LLM test suite files in the project directory. Scans CWD and standard sub-directories (evals/, tests/, contracts/). Returns file paths ready to pass to run_eval_contract.

Discovered
llm_fit_finder

Shortlist LLMs on facts, no ranking: cloud models with the features you need within a monthly budget, cheapest first; or local models that fit your VRAM, largest first, with the ollama command. A shortlist is not a choice: only your own cases show which model fits.

Discovered
llm_format_check

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

Discovered
llm_generate

Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.

Discovered
llm_json_schema_check

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

Discovered
llm_output_validator

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

Discovered
lorem_ipsum

Generate Lorem Ipsum placeholder text for UI mockups, design prototypes, or test data population. Configurable paragraphs (1–10), sentences per paragraph (1–20), and approximate words per sentence (3–30).

Discovered
mcp_schema_lint

Lint one MCP tool definition with the same rules as mcp_server_health_check: errors break the MCP spec (FAIL), documentation_issues are what its strict mode refuses — a parameter without description or type (FIX, never an A) — and suggestions are style only (snake_case) and change nothing.

Discovered
mcp_server_evaluate

Check that a live MCP server over Streamable HTTP speaks the protocol, 2026-07-28 or earlier: handshake or server/discover, tools/list and its definitions, error codes, one tools/call, structuredContent against outputSchema, latency. Returns PASS/FIX/BLOCK with a 0-100 score, or INCONCLUSIVE (score null) when authorization is required. The verdict covers the PROTOCOL, never whether tools answer correctly: what_this_does_not_prove lists what was not measured. Called: only test_tool_name, or a tool annotated readOnlyHint: true. A refusal of invented arguments is not scored (tool_behaviour: refused_placeholder); pass test_tool_args to exercise the tool. Private addresses and stdio are out of reach: paste tools/list into mcp_server_health_check.

Discovered
mcp_server_health_check

Check an MCP server's tool definitions without contacting it — for a server not deployed yet, a stdio server, or a CI gate. Paste the answer to tools/list (raw JSON-RPC or its result), a bare array of tools, one tool definition, or a manifest carrying name/description; input_shape says which one was read. Validates every tool against the MCP spec (name 1-128 chars of A-Z a-z 0-9 _ - .; camelCase and a tool with no parameters are valid; inputSchema of type object; required fields listed in properties) and, with strict, documentation (description ≥ 20 chars, every parameter with a description and a type). verdict: FAIL when a tool breaks a rule or no tool is found, WARN when only a manifest's server name/description is missing, PASS otherwise. A tools/list carries no server name, so those two checks are not scored on it (the name comes from initialize). Gate on verdict or on checks[tool_quality].pass.

Discovered
merge_json

Deep merge two JSON objects. Supports three array strategies: replace (default), concat, or unique (dedup concat). Nested objects are recursively merged — override takes precedence for primitives.

Discovered
metamorphic_check

Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

Discovered
minify_js

Minify a JavaScript snippet, function, class, or module up to 50 KB using Terser. Returns minified code and byte savings. Use when embedding scripts in HTML templates, report payloads, or injecting inline code programmatically.

Discovered
mock_from_schema

Generate realistic mock data from a JSON Schema. Supports all common types (string, number, integer, boolean, array, object, null), format hints (email, date, date-time, uri, uuid), enum, const, and nested schemas.

Discovered
model_info

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

Discovered
multimodal_eval_guide

Multimodal evaluation: action=guide (default) interprets a CLIP/FID/VQA score against reference thresholds; clip_score, fid_score, vqa_accuracy compute the metric (HuggingFace, or your VLM key for VQA); pipeline runs all three.

Discovered
needle_haystack_generate

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

Discovered
normalize_vector

L2-normalize a float vector (produce a unit vector with norm=1). Required by many vector DBs (Pinecone, Qdrant cosine). Supports batch normalization of up to 1000 vectors.

Discovered
normalize_whitespace

Normalize whitespace: trim trailing spaces, collapse blank lines, normalize line endings (LF/CRLF), convert tabs to spaces. Useful for cleaning code, configs, and text before processing.

Discovered
number_base_convert

Convert numbers between bases: decimal, binary, octal, hexadecimal, or any base 2–36. Auto-detects 0x, 0b, 0o prefixes.

Discovered
openapi_validate

Validate the structure of an OpenAPI 3.x specification (JSON or YAML). Checks required top-level fields (openapi, info.title, info.version, paths), validates each operation (responses, operationId uniqueness), detects undeclared $ref components, and flags missing 2xx responses. Returns a PASS/FAIL verdict, a 0–100 compliance score, and a list of errors and warnings with JSON-pointer locations. Use before publishing an API spec or generating SDK code.

Discovered
optimize_prompt_tokens

Compress an LLM prompt by removing filler words, verbose phrases, duplicate sentences, and unnecessary whitespace. Returns optimized text with token savings breakdown. 100% deterministic, no API key needed.

Discovered
parse_csv

Parse a CSV string into a JSON array of objects (or raw arrays). Full RFC 4180: quoted fields may contain the delimiter, embedded newlines (the Excel/Sheets multi-line cell), and doubled quotes. Custom delimiters supported. An unterminated quote is rejected with its position rather than parsed into corrupted rows. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.

Discovered

Recent measurements

Probe history

View status
Latest stored probes. Raw probe rows are kept for 14 days.
CheckedResultResponseTools
Connected249 ms153