Nov 14
·

The United Times

United in Changing Times

Artificial intelligence · Technology

Office AI finds its hardest test in the exception queue

Demo journalism: An invented procurement exercise shifts attention from fluent answers to the cost of detecting and correcting mistakes.

By Selin Marrowby · 5 min read

Office AI finds its hardest test in the exception queue
Illustration · The United Times

In a fictional procurement exercise, 18 municipal offices test an AI system against 24,000 archived administrative requests. The software correctly routes 91% of routine submissions, but performance falls to 63% when documents contain conflicting addresses or unusual ownership arrangements. Those invented results illustrate a familiar evaluation problem: a strong average can conceal weaknesses precisely where a service needs careful judgment most.

The proposed contract therefore pays for completed, verified cases rather than generated responses. It also reserves €1.6 million for human review, appeals and record correction. That changes the economic comparison with conventional software: the relevant figure is the cost of a dependable workflow, not the price of a model query. Faster drafting creates limited value if staff must reconstruct the evidence behind every difficult answer.