Designing trust into AI: a UX for verifying answers

AI has come a long way since chatGPT was released in 2022. Back then hallucinations were not just possible but a guarantee. Using it for serious work was out of the question. Asking legal or medical advice professionally? Forget it. But quite quickly improvements happened - models got smarter, RAG was popularised. But while chatting to AI has become more reliable it's not 100% reliable. Even 1% wrong answer is too much for some industries and erodes trust.

Imagine a model as a driver in a vast 3D landscape where every possible sentence is a location and the terrain is shaped like the most probable order of words. The question drops the driver at a location and its job is to find a smooth, natural feeling route to the destination. It always follows the path of least resistance.

But when you ask about something rare, obscure, or nonexistent, there's no worn path. The model doesn't say "sorry, I don't know where I am", it does what it's supposed to do, follow the smoothest route to the destination. And while it may appear confident, it's found itself completely lost, producing the wrong citation, health advice, calculation.

Confidence scoring

It is possible to ask the LLM how confident it is. The question could even be hidden from the user and surfaced as a 'confidence o-meter'. While tempting to use, the confidence score has a fatal flaw - it is just another question to the model, so it's subject to the same unreliability. The model doesn't know whether it's right, so it generates a confidence number the same way it generates the answer itself: by plausibility, not by actually checking. What's more, scores are relative. So rejecting responses with a confidence score below 0.7 is near meaningless when the LLM doesn't know the scale.

Verification over confidence

There is currently no foolproof way to verify LLM citations. If there were then there would be no need for UX, the verification could be hidden away from the user's eyes. Instead verification is a spectrum of increasing confidence. We must layer measures to increase the effectiveness of it.

Some solutions in our toolkit are:

• Inline citation links.
• Programmatically extracted passages from the document for human-in-the-loop verification.
• Limit the search to an authoritative corpus as opposed to the LLM's general knowledge
• Verification from smaller models.

In the design below I combined citation marks that link to excerpts below the message. Hallucinations are dramatically highlighted in red.

What the UX gets right:

1. The LLM's question is grounded in an authoritative corpus - not its general knowledge, not web search results.
2. Pattern matching is used to look up the LLM's claims in the documents. LLM doesn't check its own answers.
3. Claims are traceable. The human can see a programmatically extracted excerpt from the document as proof that the citation does exist.
4. Hallucinations are highlighted in red to build trust.

The tradeoffs:

This system relies heavily on the LLM returning the exact text that's in the document and false positives are possible. In this example we're able to pattern match titles that start with numbers. But for something more general, like medical documents, those false positives would increase. In that scenario the LLM's query could be re-run before it gets to the user to decrease the possibility of that false positive, and the human verification UX would not be optional.

Success State

Error State

Hallucinations happen, and no automated step fully catches them. But that's exactly why this design exists: the goal was never to remove human verification, only to make it as effortless as possible.