Run evaluations

The arcade evals command discovers and executes evaluation suites with support for multiple providers, models, and output formats.

Basic usage

Run all evaluations in the current directory:

arcade evals .

The command searches for files starting with eval_ and ending with .py.

Show detailed results with critic feedback:

arcade evals . --details

Filter to show only failures:

arcade evals . --failed-only

Multi-provider support

Single provider with default model

Use OpenAI with default model (gpt-4o):

export OPENAI_API_KEY=sk-...
arcade evals .

Use Anthropic with default model (claude-sonnet-4-5-20250929):

export ANTHROPIC_API_KEY=sk-ant-...
arcade evals . --use-provider anthropic

Specific models

Specify one or more models for a provider:

arcade evals . --use-provider openai:gpt-4o,gpt-4o-mini

Multiple providers

Compare performance across providers:

arcade evals . \
  --use-provider openai:gpt-4o \
  --use-provider anthropic:claude-sonnet-4-5-20250929 \
  --openai-key sk-... \
  --anthropic-key sk-ant-...

When you specify multiple models, results show side-by-side comparisons.

API keys

API keys are resolved in the following order:

Capture mode

Record tool calls without scoring to bootstrap test expectations:

arcade evals . --capture --file captures/baseline --format json

Include conversation context in captured output:

arcade evals . --capture --add-context --file captures/detailed

Capture mode is useful for:

• Creating initial test expectations
• Debugging model behavior
• Understanding tool call patterns

See Capture mode for details.

Output formats

Save results to files

Save results in one or more formats:

arcade evals . --file results/out --format md,html

Save in all formats:

arcade evals . --file results/out --format all

Available formats

Multiple formats generate separate files:

• results/out.txt
• results/out.md
• results/out.html
• results/out.json

Command options

Quick reference

--use-provider

Specify which provider(s) and model(s) to use:

--use-provider <provider>[:<model1>,<model2>,...]

Supported providers:

• openai (default: gpt-4o)
• anthropic (default: claude-sonnet-4-5-20250929)

Examples:

# Default model for provider
arcade evals . --use-provider anthropic
 
# Specific model
arcade evals . --use-provider openai:gpt-4o-mini
 
# Multiple models from same provider
arcade evals . --use-provider openai:gpt-4o,gpt-4o-mini
 
# Multiple providers
arcade evals . \
  --use-provider openai:gpt-4o \
  --use-provider anthropic:claude-sonnet-4-5-20250929

--openai-key, --anthropic-key

Provide API keys explicitly:

arcade evals . --use-provider openai --openai-key sk-...

--capture

Enable capture mode to record tool calls without scoring:

arcade evals . --capture

--add-context

Include system messages and conversation history in output:

arcade evals . --add-context --file out --format md

--file

Specify output file base name:

arcade evals . --file results/evaluation

--format

Choose output format(s):

arcade evals . --format md,html,json

Use all for all formats:

arcade evals . --format all

--details, -d

Show detailed results including critic feedback:

arcade evals . --details

--failed-only

Show only failed test cases:

arcade evals . --failed-only

--max-concurrent, -c

Set maximum concurrent evaluations:

arcade evals . --max-concurrent 10

Default is 5 concurrent evaluations.

--debug

Show debug information for troubleshooting:

arcade evals . --debug

Displays detailed error traces and connection information.

Understanding results

Results are formatted based on evaluation type (regular, multi-model, or comparative) and selected flags.

Summary format

Results show overall performance:

Summary -- Total: 5 -- Passed: 4 -- Failed: 1

How flags affect output:

• --details: Adds per-critic breakdown for each case
• --failed-only: Filters to show only failed cases (summary shows original totals)
• --add-context: Includes system messages and conversation history
• Multiple models: Switches to comparison table format
• Comparative tracks: Shows side-by-side track comparison

Case results

Each case displays status and score:

PASSED Get weather for city -- Score: 1.00
FAILED Weather with invalid city -- Score: 0.65

Detailed feedback

Use --details to see critic-level analysis:

Details:
  location:
    Match: False, Score: 0.00/0.70
    Expected: Seattle
    Actual: Seatle
  units:
    Match: True, Score: 0.30/0.30

Multi-model results

When using multiple models, results show comparison tables:

Case: Get weather for city
  Model: gpt-4o -- Score: 1.00 -- PASSED
  Model: gpt-4o-mini -- Score: 0.95 -- WARNED

Advanced usage

High concurrency for fast execution

Increase concurrent evaluations:

arcade evals . --max-concurrent 20

Save comprehensive results

Generate all formats with full details:

arcade evals . \
  --details \
  --add-context \
  --file results/full-report \
  --format all

Troubleshooting

Missing dependencies

If you see ImportError: MCP SDK is required, install the full package:

pip install 'arcade-mcp[evals]'

For Anthropic support:

pip install anthropic

Tool name mismatches

Tool names are normalized (dots become underscores). If you see unexpected tool names, check your tool definitions and your expected tool calls.

API rate limits

Reduce --max-concurrent value:

arcade evals . --max-concurrent 2

No evaluation files found

Ensure your evaluation files:

• Start with eval_
• End with .py
• Contain functions decorated with @tool_eval()

Next steps

• Explore capture mode for recording tool calls
• Learn about comparative evaluations for comparing tool sources