What are evals?

By now everyone is aware of the non-deterministic nature of LLMs. Given this non-determinism how do we validate that our LLM-driven program is working as expected? What happens when we change the model? What happens when real users interact with the program - does it behave as expected? The solution to this is a combination of offline and online evaluations - evals for short. If you have written any LLM-driven program you have always been left with the unsettling feeling of how the program will respond to queries which I have not tried. Evals will help you there to an extent. There is no foolproof solution for this given the non-deterministic nature of LLMs.

You may have seen the GPT-4o rollback happening due to the extra agreeability OpenAI introduced due to a prompt change Sycophancy in GPT-4o. Evals are supposed to catch issues like that.

In my personal experience an AI agent went from previously reliable tool calling to broken tool calling when a model was upgraded. Luckily we had evals which caught this before the model change was deployed to production.

So, basically an eval is a repeatable set of experiments which help us measure how our AI application behaves.

In addition to these, evals will allow you to answer questions like:

• Can we switch to a cheaper model and maintain the quality?
• What improvements happened in experience due to the prompt change and what got worse?
• Are we ready to ship?

Components of an eval

Every eval will have three components - a dataset, a task and scorers.

Dataset

A dataset is a set of test cases. Standard test case terminology applies input which results in expected output with metadata for analysis. For example if we are building a system for answering factual questions, the input/output pairs may be like:

There can be datasets which have no well-defined outputs but will have guidelines on expected output. For example in a customer service agent the output will be expected to be polite and relevant to the question asked.

Task

The task is the functionality that we are evaluating. It is the function which converts the dataset input to the output. For example, in the case of the factual questions it might be "Answer the question truthfully" or for a customer service agent "Answer the customer questions politely and factually".

Scorers

There are multiple ways of scoring an eval:

• deterministic: A piece of code determines whether the task succeeded or not. Problems like the factual questions agent which have a single output will use these kinds of scorers.
• LLM as a judge: Confusing as it is we can use another LLM to judge the output of our LLM under test. Cases where the answer cannot be measured deterministically like the customer support scenario will use this.
• Human review: A human goes through each response manually and scores. As you can imagine this gets tedious very fast. In my opinion, the aim of evals is to minimize the human review process.

Manual evals

Let's try a manual eval to get a flavor of things. The agent that we built in the post Building a Coding Agent : Part 9 - Adding Command Handling will be used for this. First let's remove all the registered MCP servers to keep it simple. Let's try with the first question.

You: What is the capital of India?
LLM: The capital of India is New Delhi.

As you can see the response is a sentence but we would prefer short responses with just the answer. So, let's tweak the prompt for achieving the necessary response style. The prompt was changed from the developer-centric prompt to Respond factually, generate short answers. However, that doesn't help and we still get the same answer. Let's try with Respond factually, generate the shortest answer possible. This works, and we get the desired answer. Now we will use the other two questions also to check if this works as expected.

You: What is the capital of India?
LLM: New Delhi
You: What is the longest river in the world?
LLM: The Nile River.
You: Who wrote Harry Potter?
LLM: J.K. Rowling.

The answers are close - only the longest river question gets a slightly different answer which is also correct. We can now either further tweak our prompt or an alternate option is to tweak the dataset as The Nile River is also a valid answer and live with the prompt.

What we have done is a simple manual eval process which took 3 input queries and iterated the prompt to generate the correct response style. There is more to evals than this but it helps us get a flavor of things.

In this series, we will explore writing an eval framework for a customer service agent. The most common example of all eval tutorials. It is the todo list app of the LLM world.