Artificial intelligence · Technology
Office AI finds its hardest test in the exception queue
Demo journalism: An invented procurement exercise shifts attention from fluent answers to the cost of detecting and correcting mistakes.
By Selin Marrowby · 5 min read

In a fictional procurement exercise, 18 municipal offices test an AI system against 24,000 archived administrative requests. The software correctly routes 91% of routine submissions, but performance falls to 63% when documents contain conflicting addresses or unusual ownership arrangements. Those invented results illustrate a familiar evaluation problem: a strong average can conceal weaknesses precisely where a service needs careful judgment most.
The proposed contract therefore pays for completed, verified cases rather than generated responses. It also reserves €1.6 million for human review, appeals and record correction. That changes the economic comparison with conventional software: the relevant figure is the cost of a dependable workflow, not the price of a model query. Faster drafting creates limited value if staff must reconstruct the evidence behind every difficult answer.