How Should We Validate AI Output?
Working with AI · Part 4
A quote-review tool gets 96 of 100 orders right. Four misspellings and a quote using another customer’s contract price both fit that headline score. The consequences are very different.
Connect errors to operational consequences
Separate wrong product IDs, missing terms, and prices quoted without evidence. Record where each error can be found and what has to be undone after a quote reaches a customer. Include high-value orders and unusual contracts even when they are rare. Test ordinary orders, expired contracts, and orders received at renewal. Check that the tool routes cases with insufficient evidence to a reviewer and measure the time needed to resolve them.
Turn release criteria into operating conditions
An automatic step might require a verified contract rate, an order quantity within the agreed range, and a traceable source. Overlapping contracts and mismatched customer IDs should trigger a review. Compare early drafts with the existing process and record how each failure was identified. Include those cases in the next round of validation.
Check both the format and the terms
Structured outputs can require customer IDs and prices in a draft. Verify those values against the correct contract and effective date. NIST’s generative AI risk profile discusses checking provenance and citations during evaluation and monitoring. For a held quote, measure whether the reviewer resolves the case by the deadline.
OpenAI on Structured Outputs · NIST’s generative AI risk management profile
Two checks for quotation errors
A 90% pass rate can conceal a recurring problem. If all three material errors involve the contract expiry date, add that case to the acceptance tests before expanding the scope.
- Check required fields
- Verify terms against the contract
- Material error or missing evidence → hold
- Verified → complete review
Illustrative sample. Each quote is classified once by its primary outcome.
Test changes, renewals, and multiple price lists separately. Record whether the held cases were resolved on time so the next release can be compared on the same basis.