ts-bench

Benchmark CLI for comparing AI coding agents on TypeScript workloads. Run an agent, execute tests, get pass/fail — across models and providers. Numbers are directional, not lab-grade.

bun install
bun link          # installs the `ts-bench` command globally
ts-bench --agent claude --model <model>

ts-bench --help for all options.

25 self-contained TypeScript practice exercises. No Docker required.

ts-bench --agent claude --model <model>
ts-bench --agent grok --model grok-build-0.1

Omit --model to use the agent's default model.

Run Grok Build with an xAI API key:

export XAI_API_KEY="xai-..."
ts-bench --agent grok --model grok-build-0.1

In GitHub Actions, ts-bench writes a Grok custom model config so the Grok CLI uses the requested xAI API model id.

Run Antigravity (agy) with a Gemini API key:

export GEMINI_API_KEY="AIza..."
ts-bench --agent agy

agy runs in print mode; run-agent.sh enables the direct Gemini backend (settings.json modelProvider: "gemini") when GEMINI_API_KEY is set.

Frozen baseline for reproducibility: tag v1-final

Real-world tasks from a large monorepo (Expensify). Requires Docker.

Read the introduction: ts-bench v2: Benchmarking Coding Agents on Real TypeScript Monorepo Tasks

./scripts/setup-v2-env.sh                               # one-time setup
ts-bench --dataset v2 --task <id> --agent claude ...    # run

Workflow runs: v1 · v2 · Releases

Task browser: bun run build:swelancer-pages then open docs/swelancer-tasks/. See docs/README.md.

Article: ts-bench v2 introduction

• Handbook — setup, secrets, CI, methodology
• AGENTS.md — runner caveats for Cursor and other agents
• Subscription auth — run agents without API keys (claude, gemini, codex, copilot)
• Token usage — how token counts are collected per agent