CLI Reference

Ghostlab installs two equivalent console scripts, ghostlab and rehearsal. New examples use ghostlab.

init

Create a ghostlab.yaml spec — the canonical, human-editable description of the MCP under test — from an existing target JSON. The spec is the entry point for the project flow (init → discover → future plan/test/review stages); every other command keeps accepting raw target/scenario JSON as before.

ghostlab init --target target.json          # writes ghostlab.yaml
ghostlab init --target target.json --out cortex.ghostlab.json

Options: --name (display name), --workspace (artifact directory, default .ghostlab/ next to the spec), --force (overwrite an existing spec). Specs can be YAML or JSON by extension; the built-in YAML reader covers everything ghostlab emits, and an installed PyYAML is picked up automatically for full YAML syntax in hand-edited specs.

discover

Connect to the spec's target, capture the inventory (inspect.json), lint the contract into contract.json/contract.md, probe ui:// widget resources when the server exposes MCP Apps tools, and refresh the spec's capabilities section (tool risk labels, UI resources, artifact provenance). Artifacts land under <workspace>/discover/<timestamp>-<id>/.

ghostlab discover --spec ghostlab.yaml
ghostlab discover --spec ghostlab.yaml --strict          # exit 1 when review gates fail
ghostlab discover --spec ghostlab.yaml --sample safe     # also call read-only tools once

If the spec declares a setup section, discover executes it first and tears it down afterwards: setup.commands run in order (background: true for the server process itself — it is terminated at teardown), setup.health probes (http, tcp, or command) are polled until they pass or time out, and setup.teardown always runs. Logs land in setup.log, and setup.json records per-step status plus a version fingerprint (ghostlab/python/platform/ server versions). --skip-setup bypasses all of it when the target is already running.

--sample calls tools for real, under an explicit safety model:

  • safe — only tools classified read-only (MCP annotations first, heuristics second), with arguments generated from each required parameter's schema (default → examples → enum head → type zero-value). Tools whose arguments can't be generated are skipped with a reason, never guessed.
  • fixture — additionally calls tools listed in setup.fixtures (- {tool: name, arguments: {...}}) with those arguments. A mutating fixture still requires --approve-mutations; a destructive one requires --approve-destructive. After mutating samples, setup.reset hooks (tool or command) restore state.

Sample outcomes are written to samples.json and folded into the contract as findings (failed calls, isError results, declared outputSchema without structuredContent, UI tools that return nothing model-visible).

Contract findings are deterministic (no model calls): schema quality (undocumented/untyped params, required names that don't exist, $ref-heavy schemas hosts translate poorly), risk classification (read-only vs mutating vs destructive, credential-bearing params, UI-producing — MCP tool annotations take precedence over name heuristics), and MCP Apps metadata compatibility (standard _meta.ui.resourceUri vs the openai/outputTemplate alias, dangling ui:// references). With --strict, the spec's review.gates get teeth: no_tool_schema_errors: true fails the run when any error-severity finding exists.

plan

Generate a coverage-driven test-plan.yaml from the latest discover artifacts. Deterministic suites (smoke/edge/apps/security-from-contract) exist for a stated reason (tool coverage, workflow coverage, UI coverage, risk coverage), so the plan doubles as a coverage report and lists untested tools/widgets as gaps. The semantic/security suites also get real, persona-grounded conversational scenarios, generated by the same engine ghostlab generate-dataset uses: ghostlab plan infers a capability profile, proposes personas relevant to the MCP's domain, and generates goal-oriented scenarios per persona — for a language-learning MCP that's things like "a beginner French learner asks for a writing exercise" or "an impatient user pushes for placement testing before onboarding is done".

ghostlab plan --spec ghostlab.yaml                     # generate/regenerate (personas on by default)
ghostlab plan --spec ghostlab.yaml --no-generate       # fast, free, deterministic-only plan
ghostlab plan --spec ghostlab.yaml --personas 3 --scenarios-per-persona 3
ghostlab plan --spec ghostlab.yaml --regenerate        # force fresh personas/scenarios
ghostlab plan --spec ghostlab.yaml --approve           # curate: approve all cases
ghostlab plan --spec ghostlab.yaml --reject security-resource-injection

Each persona and scenario is a real codex call, so generation defaults to a small size (2 personas × 2 scenarios) and is cached: a plan re-run reuses the previously generated dataset (tracked in the spec's test_plan.generated_dataset) instead of calling codex again — pass --regenerate to refresh it, or --no-generate to skip generation entirely. Generated scenario intent routes the case: happy_path/edge_case land in semantic, adversarial lands in security (a persona pushing on a risk). Personas/scenarios are written under <workspace>/generated/<timestamp>-<id>/ alongside a profile.json domain summary.

Suites follow the roadmap taxonomy: smoke (protocol discovery + one minimal call per read-only tool + first-widget render — executable without a model), semantic (real dual-agent scenarios once generated, otherwise inert per-tool-family seeds), edge (missing-required and invalid-enum probes derived from schemas), error-recovery (seeded from sampling failures), apps (render + interact per ui:// resource), security (contract-driven hallucinated-tool/destructive/credential/injection probes, plus generated adversarial-persona scenarios), host-compat (smoke slice per configured host when the spec declares several), and regression (reserved for run-history failures).

Case ids are deterministic, so status curation (proposed / approved / rejected) survives regeneration after a re-discover. A test-plan.md companion and the spec's test_plan summary are refreshed on every run.

test

Execute the test plan across the spec's host adapters and write a results bundle under <workspace>/test/<timestamp>-<id>/ (results.json + results.md, host/version fingerprints included). Progress prints live, per case, as it runs — this matters most for conversational cases, which are a real multi-turn LLM conversation and can take real wall-clock time.

ghostlab test --spec ghostlab.yaml
ghostlab test --spec ghostlab.yaml --suite smoke --suite edge   # CI-able subset
ghostlab test --spec ghostlab.yaml --hosts direct-mcp --approved-only --strict
ghostlab test --spec ghostlab.yaml --no-judge                   # skip the codex judge/critique

Every case runs on each capable host (one result per case × host). The built-in direct-mcp host executes protocol cases deterministically — no model, no variance: discovery must list tools, tool_call with expect.no_error must succeed without isError, and expect.graceful_error passes only when the server rejects bad input in protocol (JSON-RPC error or isError: true) rather than crashing. app_render cases render through the Playwright apps host when ghostlab[apps] is installed (skip with a reason otherwise).

Runner-backed hosts (codex-session/process kinds in hosts) execute conversational cases that carry a concrete execution.scenario — this is the actual dual-agent role-play: a user-emulator session (driven by the case's generated persona and goal) and an agent-under-test session with the target MCP attached go back and forth turn by turn, and every turn — user message, assistant reply, tool calls — prints live. Once the conversation ends, --judge (on by default) scores it with the codex judge: pass/fail is the judge's verdict, not just "did the conversation finish" — a session can complete without the user's goal actually being met. A tool-usability critique also runs and its critique.json feeds ghostlab review's aggregated MCP feedback. --no-judge skips both and falls back to finished-or-not. Seeds still marked needs_generation (no scenario attached yet) skip with instructions to run ghostlab plan --generate.

Cases no host can execute surface as explicit skips, never silence.

The spec's setup section runs before and tears down after, exactly as in discover. With --strict, review.gates.min_pass_rate fails the run when the executed pass rate drops below it.

--repeat N runs the plan N times and writes variance.json: per-case status distribution across attempts, with flaky cases (passed some attempts, failed others) called out separately from broken ones — the difference matters once model-backed hosts join the matrix. --profile bundles CI presets: smoke (smoke+edge suites), nightly (all suites), release (all suites, --repeat 3, strict gates). Explicit flags override the preset.

A minimal GitHub Actions job:

- name: MCP smoke tests
  run: |
    pip install ghostlab
    ghostlab discover --spec ghostlab.yaml --strict
    ghostlab test --spec ghostlab.yaml --profile smoke --strict
    ghostlab review --spec ghostlab.yaml --strict

review

Readiness report over everything the pipeline produced — the release-gate answer to "is this MCP ready, and if not, what do I fix first?".

ghostlab review --spec ghostlab.yaml            # uses the latest test results
ghostlab review --spec ghostlab.yaml --strict   # exit 1 unless verdict is 'ready'

Writes readiness.json / readiness.md next to the test results:

  • Gates — the spec's review.gates evaluated against evidence (min_pass_rate vs executed results, no_tool_schema_errors vs contract findings, no_ui_console_errors vs apps cases, no_high_security_findings vs security cases), each pass / fail / not-evaluated with a reason.
  • Failure clusters — failed cases grouped by category (ui-render, input-validation, tool-runtime, transport-protocol, host-compatibility, security) and detail signature, so repeats of one root cause read as one problem.
  • Repairs — prioritized, concrete recommendations mapped from finding kinds (P1 "fix inputSchema.required" before P4 "add param descriptions"), with the tools they apply to.
  • MCP feedback — every conversational case's tool-usability critique (ghostlab test's judge pass), rolled up: average tool-ergonomics score, deduplicated top recommendations across all runs, and a per-tool table (name clarity, suggestions). This is literally "ask the agent that used the tool how it felt, and aggregate the answers."
  • Verdict — not-ready (a gate failed), needs-work (failures, error findings, coverage gaps, or planned suites nothing executed), or ready.

inspect

Introspect a target MCP server.

ghostlab inspect --target examples/target.json

profile

Create a capability profile from an inspect.json.

ghostlab profile --inspect runs/<id>-inspect/inspect.json

generate-scenarios

Generate grounded scenarios from a capability profile.

ghostlab generate-scenarios \
  --profile runs/<id>-inspect/capabilities.json \
  --n 3 \
  --output-dir scenarios

generate-personas

Generate reusable domain personas from a capability profile.

ghostlab generate-personas \
  --profile runs/<id>-inspect/capabilities.json \
  --n 4 \
  --output-dir personas

generate-dataset

Generate a persona x scenario dataset.

ghostlab generate-dataset \
  --profile runs/<id>-inspect/capabilities.json \
  --personas 3 \
  --scenarios-per-persona 3 \
  --seed 7 \
  --name cortex

review-dataset

Review, flag, approve, or reject dataset cases before spending agent credits.

ghostlab review-dataset \
  --dataset datasets/cortex \
  --profile runs/<id>-inspect/capabilities.json

ghostlab review-dataset --dataset datasets/cortex \
  --approve case-a case-b --reject case-c

run

Run one scenario.

ghostlab run \
  --target examples/target.json \
  --scenario examples/scenario.json \
  --aut-runner runners/mock-aut.json \
  --user-runner runners/mock-user.json

run-dataset

Run every case in a dataset. Use --limit for small development runs and --approved-only to skip unreviewed cases.

ghostlab run-dataset \
  --dataset datasets/cortex \
  --target target.json \
  --aut-runner runners/codex-cortex-aut.json \
  --user-runner runners/codex-user-emulator.json \
  --limit 2

evaluate

Score a completed run into a pass, partial, or fail verdict.

ghostlab evaluate --run runs/<id> --capabilities runs/<id>-inspect/capabilities.json

critique

Critique the MCP server's tool usability from a completed run. Where evaluate asks "did the scenario pass?", critique asks "how do I improve this MCP?": it grades the naming, descriptions, parameter clarity, and error quality of the tools the agent actually exercised, with concrete suggestions. Pass --inspect so the judge can see the real tool definitions.

ghostlab critique --run runs/<id> --inspect runs/<id>-inspect/inspect.json

Writes critique.json and critique.md into the run directory.

compare

Diff two dataset result sets for regressions.

ghostlab compare --base runs/<base>-summary --candidate runs/<candidate>-summary \
  --output comparison.md

scorecard

Aggregate a whole dataset run into one MCP validation report (pass rate, per-tool reliability, hallucination/golden-mismatch counts, efficiency, and recurring tool-design recommendations). No model calls — it reads the per-case artifacts.

ghostlab scorecard --results runs/<id>-summary

Writes scorecard.json and scorecard.md into the summary directory.

doctor

Validate local agent and runner setup.

ghostlab doctor
ghostlab doctor --runners runners/codex-cortex-local-session.json