Test AI before it meets customers

A demo is not a test. A good AI system has a set of cases you can run again and again. Without it, you only know it worked on the day someone happened to look.

Collect real cases first

Start with questions and tasks that already exist in the business. Take emails, tickets, documents and conversations an experienced person has solved. Write down what a good answer must contain.

30–50 cases is a good start. They do not need to be perfectly phrased. What matters is that they resemble what the system will meet, not that they are easy to pass.

Measure more than an answer

An answer can be correct but built on the wrong source, or wrong but still useful. Split the assessment into what actually matters for the task: correctness, sources, format, tone, cost and time.

For an agent, add tool choice, number of steps and whether it asked for help when it should. For a RAG system, add whether the right document was found and whether the citation pointed to the right place.

  • Correctness: does the answer match the evidence?
  • Sources: do they point to the right document and passage?
  • Behaviour: does the system ask when it is unsure?
  • Cost and time: can we handle the volume?
  • Safety: does sensitive data or the wrong permission leak?

Make the assessment repeatable

Write a short rubric for each case. “A correct answer should mention three things, quote section 4 and not give a recommendation.” Then two people can judge roughly the same way, and you can compare over time.

Have someone who did not build the system judge some of the cases. The builder knows why an answer should be right. That makes it easier to miss what a user sees.

Run the same tests on every change

Every time you change model, prompt, tool or search, run the same set. Compare the result with the previous version. An improvement in one area is not worth much if it makes three others worse.

Store the results with date and version. Then you can answer why quality changed, and you do not have to guess when something gets worse.

Release gradually

Start with a small group of users or a small share of traffic. Have a person review the answers for a period. Increase when quality holds, and keep a simple way to turn it off.

Log what you need to follow up, but avoid sensitive information. A system you can switch off and a person who reads the log matter more than an impressive demo.

What tests cannot tell you

Tests do not say everything about how the system feels in daily work. They do not say whether people trust it or whether it saves time. You need real users to measure that.

But without tests, that question becomes impossible to answer. Only when quality is measurable can you see whether the time saved is worth anything.

Read next

Questions and answers

How many test cases do we need?

Start with 30–50 real cases. More is better if they cover different kinds of questions, but a small set that is actually run beats a large one nobody maintains.

Can we let an AI judge the answers?

It can help at large volumes, but a person should calibrate the assessment and review a sample. Otherwise you are mostly measuring how well the judge agrees with itself.

Do we have to test every time we change model?

Yes, at least against the same fixed set. Models and prompts affect each other, and a change that looks small can alter behaviour at the edges.

Let’s find your hours.

Book an intro callcontact@antonwester.se

Taking on new work

Or write a few lines