AI · Evaluation
How to measure AI accuracy in spec matching: the metrics that matter
Every AI vendor has a percentage. The useful question is: a percentage of what, measured how, and what happens to the rest?
Short answer
Measure AI spec matching with a defined test set of real spec lines and expert-approved answers, then report: precision (proposed matches that are correct), recall/coverage (lines the system could match), top-k accuracy (a correct equal among the first k alternates), human acceptance rate (matches approved without edits), severity-weighted errors (a wrong voltage counts more than a wrong finish), and business outcomes: time-to-quote and resubmittal rate. A single “match rate” is only meaningful with its definition and sample.
Key takeaways
- Define the test set and what counts as correct before quoting a number.
- Report precision and coverage together; one without the other misleads.
- Use top-k for alternates, where several answers can be right.
- Weight errors by consequence.
- Tie AI metrics to business outcomes (time-to-quote, resubmittals).
Start with the test set
Before any number, define:
- The sample: real spec lines from real projects, across types (linear, downlights, controls, emergency), with their difficulty mix.
- The answer key: what an expert would approve as correct or equal for each line.
- The rules: is an approved equal correct, or only the basis of design? Does a wrong option code count as wrong?
Without that, a “98% match rate” could mean almost anything.
Model metrics
| Metric | Question it answers |
|---|---|
| Precision | Of the matches proposed, how many were correct? |
| Coverage (recall) | Of all lines, how many did the system propose a match for? |
| Top-k accuracy | Is a correct equal among the first k alternates? |
| Calibration | When the system says it is confident, is it right? |
Report precision and coverage together. A system can look precise by skipping hard lines.
Workflow metrics
| Metric | Why |
|---|---|
| Human acceptance rate | Matches approved without edits |
| Edit distance | How much a reviewer changed |
| Lines needing manual work | Where people still spend time |
Weight errors by severity
| Error | Consequence | Weight |
|---|---|---|
| Wrong voltage, dimming, certification | Unusable or non-compliant product | High |
| Wrong output, optic, CCT | Rejected submittal | Medium |
| Wrong finish or accessory | Resubmittal or change | Low |
A system with fewer total errors can still be worse if its errors are the expensive kind.
Business outcomes
AI metrics only matter if they move the business: time-to-quote, resubmittal rate, share of lines handled automatically and win rate. These are the numbers buyers and investors care about.
Publishing a number responsibly
When you publish a match rate, publish its definition with it: sample size, period, what counted as correct, and how coverage was handled. The research literature does this as standard practice; procurement claims should too. For how the matching itself works, see AI product matching for lighting; for why grounding matters, see LLMs in construction procurement.
Frequently asked questions
What is a good accuracy for AI spec matching?
It depends on definitions. A system can show high precision by declining hard lines, or high coverage with more errors. Ask for both, on a stated sample, with severity weighting.
What is top-k accuracy?
The share of lines where a correct product appears among the first k candidates. It suits alternates, where more than one product can be an acceptable equal.
Why does human acceptance rate matter?
It measures what experts actually approved without edits, which is closer to real value than an automated score.
Which business metrics show AI is working?
Time from receiving a schedule to sending a quote, resubmittal rate, share of lines needing manual work, and quote win rate.
References
Keep reading
- AI · Product matching AI product matching for lighting: how an equal is found and ranked How AI product matching works for commercial lighting: why it is an entity-resolution problem, how candidates are generated and filtered by hard constraints (voltage, dimming, mounting, certification), how equals are scored on photometrics and colour, and why every match needs evidence and a human decision. →
- AI · Language models LLMs in construction procurement: where they help, where they fail, and how to ground them What large language models are good at in construction and lighting procurement (reading specs, extracting attributes, drafting documents), where they fail (confabulated products and numbers, hidden uncertainty), and the controls that make them usable: grounding in verified data, retrieval, structured outputs, validation and human approval. →
- AI · Roadmap AI in lighting procurement: a technology roadmap from spec to site A technology roadmap for AI in commercial lighting procurement: reading specifications, matching every line to verified products, assembling quotes and submittals, and the next stages, learning from review, supply-chain intelligence and agentic workflows. Where AI helps today, where people stay in the loop, and what to measure. →
- Platform · Spec matching AI spec matching for commercial lighting SuppLux matches every line of a lighting specification to verified products and ranks equal or better alternates by photometrics, electrical data, mounting and lead time, with a specialist review before quoting. →