markov-ai/autocad-bench

AutoCAD Bench

AutoCAD Bench is a computer-use benchmark for evaluating whether multimodal agents can reproduce dimensioned 2D drawings and reconstruct dimensionally accurate 3D models in desktop AutoCAD.

Each rollout receives a task-specific instruction and reference image, operates an isolated AutoCAD session using mouse and keyboard actions, and is evaluated from the final DWG. Gold drawings and evaluator credentials never enter the model-visible session.

How it works

tasks/manifest.jsonl + reference PNG
                 │
                 ▼
        model harness + agent loop
                 │
                 ▼
     isolated Windows/AutoCAD sandbox
                 │
                 ▼
           final attempt.dwg
                 │
                 ▼
 trusted inspection + deterministic score + vision judgment
                 │
                 ▼
       runs/<benchmark>/ + dashboard

The benchmark contains exactly 50 tasks, numbered task-001 through task-050 with no gaps: 21 2D drawings and 29 3D drawings. A 2D task is completed in Model Space. A 3D task requires true geometry in Model Space and the complete reference sheet on Layout1; evaluator v4 renders and compares that required layout.

Getting started

Requirements:

• Python 3.12
• uv
• Node.js 20+ for the dashboard
• Infrastructure-backend and model-provider credentials only for live runs

Install the Python environment and validate the corpus:

# The development group includes the built-in AWS backend dependencies.
uv sync --dev
uv run autocad-bench validate
uv run autocad-bench validate-audit
uv run autocad-bench list --split all

Published installations that use the built-in backend should install autocad-bench[aws]. Custom backends can depend on the base package and their own infrastructure plugin without installing the AWS SDK.

The wheel contains code, while scorer-side benchmark data can come from either a checkout or a checksum-bound bundle:

autocad-bench bundle-validate --root /path/to/autocad-bench-corpus-v1
export AUTOCAD_BENCH_ROOT=/path/to/autocad-bench-corpus-v1

See Portable benchmark bundles for the data format and Manual releases for reproducible package and corpus publishing.

Run the offline tests:

uv run pytest
cd apps/dashboard
npm ci
npm test
npm run build

Run a benchmark

Copy the AWS example and replace every infrastructure, bucket, and model placeholder with resources you own:

cp configs/aws.example.json configs/my-benchmark.json

uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --skip-direct-model-check

Execute only after provider credentials are exported and preflight passes:

export OPENAI_API_KEY=...

uv run autocad-bench-batch \
  --config configs/my-benchmark.json \
  --output-dir runs/benchmarks/<run-id> \
  --execute

Every rollout is evaluated automatically after its final DWG is retrieved and before its sandbox is cleaned up. Evaluation progress is written atomically for the dashboard.

Inspect results

cd apps/dashboard
npm ci
npm run build
npm run preview

Open http://127.0.0.1:4173. The viewer reads local runs/ data and can lazily retrieve missing trace artifacts from the encrypted benchmark bucket when configured.

Repository layout

autoCAD-bench/
├── tasks/                    # Scorer-side manifest, audit inventory, and source corpus
├── src/autocad_bench/
│   ├── tasks/                # Task loading, validation, and public task models
│   ├── harness/              # Agent loop, model calls, and single-run execution
│   ├── evaluation/           # Trusted DWG evaluation and scoring
│   ├── infrastructure/       # Backend contract, registry, and built-in AWS plugin
│   ├── sandbox/              # Broker client and AWS/AutoCAD lifecycle
│   ├── orchestration/        # Parallel batches, resume, and recovery
│   └── common/               # Shared utilities
├── sandbox/windows/          # Windows broker, evaluator, and AMI bootstrap source
├── infra/cloud-controller/   # Persistent controller deployment assets
├── configs/                  # Bring-your-own infrastructure examples
├── apps/dashboard/           # Local benchmark registry and trace viewer
├── artifacts/gold-cache/     # Versioned trusted evaluator outputs
├── docs/                     # Contracts, architecture, operations, and releases
├── tests/                    # Offline regression suite
└── runs/                     # Generated benchmark results; ignored by git

Active runtime contract

• Broker: windows-autocad-2019-v10
• Evaluator: autocad-2019-r23.0.46-v4
• Per-task action budget: 5,000
• Action batch size: bounded only by the remaining rollout budget

Worker images are operator-owned and must satisfy the pinned broker contract. Broker v11 adds controller-owned DWG save but remains a separate candidate runtime. The complete evaluator-v4 cache contains and verifies all 50 source DWGs.

Documentation

Guide	Purpose
Architecture	Component boundaries and data flow
Benchmark contract	Model-visible and scorer-side task contract
Sandbox contract	AWS lifecycle, admission, and cleanup invariants
Evaluator contract	DWG inspection and Layout1 behavior
Scoring	Deterministic and vision scoring
Parallel rollouts	Batch preflight and execution
Infrastructure plugins	Bring-your-own worker backend contract
Benchmark bundles	Portable task and evaluator-data releases
Manual releases	Reproducible local assembly and verification
Cloud controller	Cloud-owned execution and recovery
Contributing	Development and validation workflow

Security boundary

tasks/manifest.jsonl and the corpus are scorer-side inputs and contain paths to gold DWGs that are private from the model session. Model-visible task serialization excludes gold paths and DWG names. A rollout receives only its reference image and task statement; trusted evaluator exchange uses request-scoped presigned S3 objects after model execution.