Benchmarks

The @reactive-agents/benchmarks package is an internal evaluation harness: 20 tasks spanning 5 complexity tiers that you run against a real LLM to measure correctness, latency, token usage, and cost for your provider and model choices — not just framework overhead.

We use it as development tooling to catch regressions; we don’t publish its scores as marketing claims. Run it yourself against your own stack — the numbers that matter are the ones from your provider, your model, and your tasks.

Benchmark Methodology

Industry Standard Alignment

Each task tier maps to a recognized benchmark standard:

Tier	Strategy	Aligned With
Trivial	Single-shot	MMLU-CS · MATH baseline · AgentEval
Simple	Single-shot	HumanEval Easy · BIG-Bench Hard CS · MMLU-Pro SE
Moderate	ReAct (reactive)	HumanEval Medium · BIG-Bench Hard · SWE-bench lite
Complex	Plan-Execute-Reflect	AgentBench · SWE-bench Security · TestEval
Expert	Tree-of-Thought	BIG-Bench Hard algorithms · GAIA Level 3 · MMLU-Pro CS

What Each Benchmark Standard Covers

• HumanEval (OpenAI) — 164 handcrafted code generation tasks evaluated by functional correctness. Our tasks include function implementation, algorithm design, and test generation.
• SWE-bench (Princeton) — Resolving real GitHub issues. We use SWE-bench patterns for bug identification, security vulnerability analysis, and multi-file code review.
• BIG-Bench Hard (Google) — 23 challenging tasks where chain-of-thought is required. We include: algorithmic optimization, logic/fallacy analysis, multi-step word problems, and Big-O complexity reasoning.
• GAIA (Meta) — Multi-step tasks requiring tool use and reasoning. Our Level 3 equivalent task tests production incident response requiring multi-domain knowledge synthesis.
• AgentBench (THUDM) — 8-environment agent evaluation. We use AgentBench patterns for system design, database decomposition, and migration planning tasks.
• MMLU-Pro — Professional knowledge across 14 domains. Tasks cover CS theory (CRDTs, design patterns), software engineering, and architecture decision-making.

Scoring

A task passes if the LLM’s output contains the expected pattern (case-insensitive regex). Patterns are crafted to require substantive, correct answers — they cannot be satisfied by generic responses:

SQL injection fix expected: "parameteriz|prepared|placeholder|$1|?"
CRDT design expected: "CRDT|vector.?clock|logical.?time|merge|commutative|converge"

Running Benchmarks

# Run with Anthropic (recommended for real-world results)
cd packages/benchmarks
bun run src/run.ts --provider anthropic --output report.json

# Run with a specific model
bun run src/run.ts --provider anthropic --model claude-opus-4-8 --output report.json

# Run only trivial + simple tiers (quick sanity check)
bun run src/run.ts --provider anthropic --tier trivial,simple

# OpenAI
bun run src/run.ts --provider openai --model gpt-4o --output report.json

# Gemini
bun run src/run.ts --provider gemini --model gemini-2.5-flash --output report.json

CLI Options

Flag	Description	Default
--provider	LLM provider (anthropic, openai, gemini, groq, xai, ollama, litellm)	test
--model	Model name (uses provider default if omitted)	Provider default
--tier	Comma-separated tier filter	All tiers
--output	Path to save JSON report	(none)

Provider Defaults

Provider	Default Model	Rationale
anthropic	claude-haiku-4-5	Fast, cost-efficient, strong reasoning
openai	gpt-4o-mini	Cost-efficient with strong benchmark performance
gemini	gemini-2.5-flash	Fast inference, competitive pricing
ollama	llama3.2	Local inference, no API cost

Reading the Report

Pass --output report.json to persist a full JSON report (per-task pass/fail, latency, token usage, and cost) for your own analysis. Without --output, results print to the console and nothing is persisted.