BuiltBenchby SPAIRLContact us

BuiltBench  /  LLM benchmark for the built environment

Which AI actually understands buildings?

Models answer questions about sites, plans, climate and codes. Geometry, solar and simulation engines check every answer.

See the leaderboardFor AEC firms

§1   Leaderboard

Scores from verified answers, updated with every run.

Awaiting first runBuiltBench v0 pilot  /  402 items  /  Updated 6 November 2026
  1. -Waiting for the first verified run

Ran BuiltBench on your model or agent? Send us the evaluation log and we will verify it and add it to the board.

Add your model

What the pilot contains

USite & Urban100

BBuilding102

CClimate & Performance100

RRegulation & Compliance100

L2L3L4One square per question

Family-balanced track scores; headline is the mean of the four tracks; 95% intervals from a 10,000-resample cluster bootstrap; models whose paired difference interval includes zero share a rank band.

§2   What it tests

Four tracks, from the parcel to the clause.

  • Track U

    Site & Urban

    How much floor area does this parcel allow, and which limit controls it?

  • Track B

    Building

    What is the shortest route from Bedroom 1 to the exit?

  • Track C

    Climate & Performance

    How deep must the overhang be to shade this window at noon?

  • Track R

    Regulation & Compliance

    Does this dwelling meet the travel distance rule?

Track U

Site & Urban

The parcel and what the rules allow on it: street frontage, setbacks, coverage, floor area ratio and height. The model gets the site as data and has to compute the answer, not recall it.

In the pilot: 100 questions

  • Read the siteL225Give the parcel area and the length of edges that face a street.
  • Footprint fitL335Does a 9.5 m by 5.0 m footprint fit inside the setbacks, and at which rotations?
  • Zoning envelopeL440What is the maximum floor area, and which limits control it?

Checked byParcel geometry: setback offsets, buildable area, coverage, FAR and height limits.

Coming nextContext beyond the parcel is specified but not in the pilot yet: overshadowing of neighbours, solar access, and street access and urban form from real city blocks.

Track B

Building

How a building is organised in plan: which rooms touch, which connect, how people move through them and whether the areas add up.

In the pilot: 102 questions

  • Access graphL217List every pair of spaces joined by a door.
  • Adjacent or connectedL317Do Bedroom 2 and the hall share a wall?
  • Reach without a roomL317Can you walk from the living room to the kitchen without passing the hall?
  • Egress routeL417Which rooms are on the shortest route to the exit, and how long is it?
  • Travel distanceL417What is the longest walk from any point to the exit door?
  • Area scheduleL417Which rooms fall short of their minimum area, and by how much?

Checked byThe door graph, and walking routes that stay inside the rooms and pass through door centres.

Coming nextSections and multi-storey stacking, and plans given as drawing images instead of data.

Track C

Climate & Performance

Reading a climate and designing for it: classification, sun angles, shading and thermal comfort, for a real location.

In the pilot: 100 questions

  • Read a climateL220Which month is the coldest?
  • Koppen-Geiger classL325Classify this climate from its monthly temperature and rainfall.
  • Sun positionL415What are the sun's altitude and azimuth at this time and place?
  • Overhang depthL420How deep must an overhang be to shade a 2.3 m south window at noon?
  • Adaptive comfortL420What is the neutral temperature and the 90% comfort band?

Checked byThe NREL solar position algorithm, the ASHRAE 55 adaptive comfort model and Koppen-Geiger thresholds.

Coming nextDaylight, ventilation and energy use ranked against building simulation.

Track R

Regulation & Compliance

Applying a rule pack to a building: which clauses apply, which one governs, and whether the design complies. The rules come with the question, so the model must apply them, not remember a code.

In the pilot: 100 questions

  • Which clauses applyL225List every clause whose conditions this building meets.
  • Which clause governsL325Which clause sets the travel distance limit for this office?
  • Travel distance verdictL425Does the dwelling comply? Give the verdict, the distance and the clause.
  • Find violationsL425List every violation of four clauses, applying any exception.

Checked byA neutral rule pack of 21 clauses, applied to the plan geometry. Verdicts and clause ids must match.

Coming nextRegion packs with real codes: India (NBC 2016, ECBC) and the US (IBC, ASHRAE 90.1).

§3   Method

An answer counts only when the numbers check out.

  • Computed, not judged

    Verifiers recompute every answer. No model grades another.

  • Same building, any file

    Rotate, mirror or rename the plan: the answer must not change.

  • Honest uncertainty

    Every score has an interval. Statistical ties share a rank.

  • Hard to memorise

    Rules ship with the question, and the private set is regenerated.

Method

Computed, not judged

Every question carries its inputs as data, so the right answer can be recomputed exactly. Each question family has its own verifier.

  • Numbers pass within a stated tolerance, for example 2% on lengths.
  • A right number in the wrong unit is caught and labelled.
  • Every miss is coded: format, unit, arithmetic, topology, orientation, rule choice.

Method

Same building, any file

The same plan is rotated, mirrored, moved, renamed and reordered. A model that understands the building gives the same answer every time.

  • Six transforms: rotate, move, mirror, turn north, reorder, rename.
  • The verifiers themselves are tested across 1,440 rotations.
  • Consistency is reported next to accuracy.

Method

Honest uncertainty

A score is an average over a few hundred questions, so it has noise. BuiltBench shows the noise instead of hiding it.

  • 95% intervals from 10,000 bootstrap resamples.
  • Questions about the same building are resampled together.
  • Models whose difference is within the noise share a rank band.
  • Every question family weighs the same within its track.

Method

Hard to memorise

The buildings and rules are generated for the benchmark, so the answers cannot be learned from the internet.

  • A synthetic rule pack ships with each question.
  • The leaderboard set is encrypted and refreshed every six months.
  • A canary string in every file shows if a model was trained on it.

§4   Open benchmark

The method is public. The test set stays private.

Code under Apache-2.0, data under CC BY 4.0.

Open the research explorerBack to the leaderboard

§5   BuiltBench for Firms

Know which LLM your firm should subscribe to, and switch every quarter.

Illustration for one firm's priorities

  • Your projects

    Private questions built from your own models.

  • Your codes and tools

    Your standards, and the assistants your teams already use.

  • A call every quarter

    A ranking for your priorities: keep or switch.

Contact us

Tell us how your firm uses AI today. We will show you what to subscribe to.

Email management@spairl.siBook a call