Can a language model read a building?
BuiltBench tests whether LLMs reason correctly about a building as an architect does: from site to footprint, plan, section, climate and code. Every answer is checked by geometry, solar and simulation engines, not by another model's opinion.
- TracksSite & Urban, Building, Climate & Performance, Regulation & Compliance
- DepthFive capability levels, from recall to synthesis
- InputsStructured plans and sites, drawing images, and sandboxed tools
- Ground truthDeterministic verifiers, with simulation for performance items
- NameRenamed from "AECbench", which collides with AECBench (2025)
List the spaces passed through on the shortest route from Bedroom 1 to the exit, and the route length.
Benchmark landscape
The benchmarks reviewed for this work, from mainstream LLM suites to AEC, spatial, urban and energy studies. Filter by domain or by how each one decides an answer is correct.
| Benchmark | Year | Ground truth | Inputs | What it found | Tracks |
|---|
Gap map
Each row is a task family BuiltBench needs. Each column is an existing AEC-related benchmark. Most of the site, section and performance rows are empty: no published benchmark tests them with checkable answers.
Recommendation
Four tracks for what is tested, five levels for how deep, three tiers for how the building is shown to the model.
BuiltBench
"Built" as in the built environment: the site, the building and how it performs. The benchmark tests reasoning about that whole object, from parcel to plan, section, climate and code, rather than recall of industry facts.
The working name "AECbench" was dropped because AECBench (Tongji University and ECADI, 2025) is an established Chinese-language AEC knowledge benchmark. A web search on 6 Nov 2026 found no benchmark called BuiltBench. The nearest names, BuildBench, Build-bench and BuilderBench, test software builds and robotics.
Names considered
- BuiltBenchChosen. Names the built environment as a whole.
- VitruvianBenchAlternate. Firmness, commodity, delight; broad and a little grand.
- EnvelopeBenchAlternate. Strong for site and climate, weak for plans and codes.
- SiteToSkinAlternate. Describes the scope, reads as a slogan.
- PartiBenchAlternate. After the architectural parti; distinctive but obscure outside architecture.
Why this split
- Climate is a lens, not a scale. Solar access, shading and comfort apply to the site and to the building alike, so Climate & Performance spans both rather than living inside one.
- Codes get their own track. Applying a rule to geometry, citing the clause and spotting exceptions is a distinct skill, and it maps onto the code-consultant role found in practice.
- Depth cuts across every track. Levels L1 to L5 make the familiar drop from recall to synthesis visible per track, rather than hiding it in one average.
- Urban stays at site scale. City-wide planning is out of scope for v1. The site matters as the envelope that constrains one building.
Design principles
- Every L1 to L4 answer is checked by code that anyone can read and re-run.
- Rules are supplied in the item. The benchmark tests applying rules, not remembering them.
- One canonical input is shown as text, as a drawing, or through tools, so perception and reasoning can be separated.
- Answers must survive rotation, mirroring, renaming and unit changes.
- No score is reported without a clustered confidence interval.
- The neutral core uses synthetic rules. Real codes arrive as separate region packs.
Structured text
PlanJSON, SiteJSON, ClimateSummary and RulePack inputs. The v1 core, at least 60% of items.
Isolates reasoning. In FloorplanQA, JSON inputs scored 59 to 81% while image-only inputs fell to about 19 to 40%.
Drawing images
Plans, sections and site plans rendered from the same canonical input, plus scanned drawings.
At least 30% are paired with a Tier A twin, so the perception gap can be measured directly.
Agentic tools
The model may call geometry, sun-position, shadow, comfort, rule and simulation tools in an offline sandbox.
Budgets on calls, time and tokens. No tool returns the answer directly.
Taxonomy
Every item is tagged with a track and a capability level. Choose a cell to see its task families and a sample item.
How items are scored
Each demo computes its ground truth in your browser, the way BuiltBench verifiers will. Change the inputs and the expected answer changes with them.
Adjacent is not the same as connected
Click any room to route from it. Track B items separate rooms that share a wall from rooms that share a door, a distinction models often blur.
Expected answer, as the verifier stores it
Zoning envelope
The verifier finds the binding constraint among setbacks, coverage, height and FAR. The defaults reproduce the worked example in the spec.
Adaptive comfort band (ASHRAE 55, 80% acceptability)
Overhang depth at solar noon
Depth = window height / tan(noon solar altitude), on the equator-facing facade. Key dates use their defining declination and other days use Spencer's (1971) approximation. The real verifier uses the NREL SPA algorithm via pvlib, accurate to 0.0003 degrees.
Score a model answer
Travel distance against a supplied rule
The rule pack is synthetic and jurisdiction-neutral, so the values cannot be recalled from any real code. The verifier walks a visibility graph inside the rooms, never through walls.
Scoring three example responses
| Response | Why | Score |
|---|
Scoring rule: 0.5 for a correct verdict whose measurement is within 2%, plus 0.5 for the correct clause. Example responses are written to show the rule; they are not outputs from real models.
Same building, different file
Variants are generated from each original with a known effect on the answer. A model that is right on the original but wrong on its variants is reading the file, not the building.
| Question | Answer on this variant | Check |
|---|
Pipeline
From seed data to report card. Choose a stage to see what happens there and what it guarantees.
Statistics
Three reporting rules adopted from the evaluation literature, each shown with simulated numbers so the effect is visible.
Items that share a plan are not independent
Eight questions are asked about each of 50 plans. When a model's success depends on the plan, the naive interval is too narrow. BuiltBench resamples whole plans (a cluster bootstrap), following Miller (2024).
Same pass@1, different reliability
Both models answer 75% of items correctly on one run. The steady model always gets the same items right; the erratic one is right 75% of the time on every item. pass^k, the share of items right in all k runs (tau-bench), separates them.
Families are weighted equally, not items
Generators make some families cheap to grow. If items were weighted equally, the easy, plentiful egress family would set the Building score.
| Track B family | Items | Accuracy |
|---|---|---|
| Circulation and egress | 3,000 | 0.90 |
| Floorplan topology | 1,000 | 0.70 |
| Building profile and section | 300 | 0.40 |
| Item-weighted mean | 0.82 | |
| Family-balanced mean (reported) | 0.67 |
Model selector
Describe what your firm does and the shortlist re-ranks. Hard requirements are checked first, then candidates are ranked by your priorities. Statistically tied candidates share a band. The free version runs on public results in your browser.
What your firm does
Must-pass gates
A gate passes only if the whole 95% interval clears the threshold.
Constraints
Profile as portable JSON
Prototype uses a normal approximation for intervals and bands. The released tool uses published bootstrap replicates, so ties are exact (design D4).
Open, free
- Methodology, specs, generators and verifiers (Apache-2.0)
- Public test items and the public leaderboard
- This selector on public results, in your browser
- Agent evaluation protocol and self-hosted harness
Enterprise, paid
- Private benchmark built from your own IFC models
- Your internal standards and licensed code packs
- Evaluation of your copilots and internal assistants
- Monitoring, silent-update alerts and procurement reports
- On-premises runner: project data never leaves your network
Roadmap
Four releases, each with a gate it must pass before the next begins. Details are in the OpenSpec tasks.md.
- v0Pilot, Tier A
- About 400 items, every track and level
- Two families per track, verifier test suites
- Inspect task package, report card
- Run on four or more models
Gate: expert audit finds under 2% defective items per family - v1Tier A at scale
- At least 1,000 private items per track
- L5 synthesis verifiers, energy ranking
- Invariance, distractor and pass^k subsets
- Judge calibration, IRT and a lite subset
Gate: rubric judge reaches kappa of 0.7 against experts - v1.1Tier B drawings
- Renderer for plans, sections, site plans and sun paths
- Cross-modal pairs for 30% of items
- Scanned drawing subset
Gate: perception gap reported with paired intervals - v2Tools and region packs
- Sandboxed Tier C tool set
- India pack: NBC 2016, ECBC 2017
- US pack: IBC, ASHRAE 90.1 and 55
- First private-split regeneration
Gate: no verbatim code text beyond short quotes
Sources and files
Where the evidence and the methodology live. Every benchmark in the landscape links to its primary source. The full reference list is in docs/research/references.md.
Project files
docs/research/Research synthesis: foundations, methodology, AEC prior art, gap analysis, recommendations, referencesreports/LLM benchmarking and AEC prior art.mdThe full narrative research reportopenspec/changes/define-builtbench-methodology/The methodology as an OpenSpec change: proposal, design, tasks and 15 capability specsopenspec/changes/add-firm-model-selection/Model selection for firms: profiles, ranking, agent protocol and governance. Enterprise specs live in the private builtbench-enterprise repository.site/index.htmlThis explorer. Rebuild withnode site/build.mjsafter editing the research data.
Methodology sources
BuiltBench methodology proposal, issued 6 Nov 2026. Benchmark numbers come from the cited papers; demo numbers are computed live or labelled as simulated.