Evals dataset

I could have generated random questions for this exercise but there is a nice publicly available dataset called SQuAD. This has a set of 100K+ questions which can be answered by a model. Let me pick 20 random questions from the set. The questions are such that no context is needed for answering those. These are 20 questions with actual answers and 10 made up questions. I used Opus 5 to make up some unanswerable questions as the actual dataset unanswerable questions depend on the context in the dataset which we are not using here.

The complete dataset and raw results are on a separate page.

Get an answer to a question from the LLM

We will use an LLM to get an answer for a question. Let's keep it simple, pass in a config, system-prompt and a question and get a response back.

(defn- get-answer
  [config system-prompt question]
  (let [messages [{:role "system" :content system-prompt}
                  {:role "user" :content question}]
        response (openai/create-chat-completion {:model (:model config)
                                        :messages messages}
                                       (select-keys config [:api-key :api-endpoint :impl]))]
    (get-in response [:choices 0 :message :content])))

A simple scorer

Let's write a simple scorer. It will only check if one of the expected answers is fully present within the LLM response. Also, if the LLM response contains "Not Known" it will match an unanswerable question.

(defn scorer
  [expected-list actual]
  (let [correct-answer? (if (seq expected-list)
                          (some #(str/includes? actual %) expected-list)
                          (str/includes? actual "Not Known"))]
    (if correct-answer? 1 0)))

Scoring the list of questions

Now armed with the function to get an answer from a LLM endpoint and a scorer, we can write a simple score-question function.

(defn- score-question
  [config prompt question]
  (let [system-prompt (:content prompt)
        q (:question question)
        a (:answers question)
        actual (get-answer config system-prompt q)]
    {:question q
     :expected-answers a
     :actual actual
     :score (scorer a actual)
    }))

Results from Qwen 3.0 0.6B

The following table shows results from running the evals against a Qwen 3.0 0.6B model. It is a simple model so, the results are not very impressive. Even then they are still good for such a small model - 13 answers correct out of 30, a 43.3% correctness rate. The prompt given to the model was:

Answer questions concisely. There is no need for full sentences. Say, Not Known if you do not know or are unable to infer.

And, that prompt shows up partially in some of the answers - like the question about the European population killed by the Black Death. There are some interesting hallucinations also like the Portugese city where the Rhine reaches the sea. Our scorer also shows its limitations where case mismatches cause an answer to be marked as fail. Like the 20.8% gas question. Also, the model sometimes gives correct answers albeit partial (like Newton instead of Isaac Newton)

The full Qwen result table is on the results page.

Results from Gemma-4-E2B

Gemma being a larger model scores better 15/30 - around 50%. Again the limitations of our scorer show up which flags correct answers as wrong due to the case mismatch or punctuation issues. Let's look at alternate scorers to fix this issue.

The full Gemma substring result table is on the results page.

F1 scorer

The F1 scorer combines values of precision and recall to generate a score of the answer. Where Precision (P) = matching tokens/predicted tokens and Recall (R) = matching tokens/expected tokens. Precision punishes padding of answers, whereas Recall punishes omission of answers.

The F1 score is defined as 2PR/(P + R)

Ideally we should normalize the generated answers to improve the precision and recall metrics but to keep things simple I will just do a lower case of the words.

(defn- normalize
  [answer]
  (map str/lower-case (filter #(> (count %) 0) (str/split answer #"\s+|\.|,|-|!"))))

We the normalized tokens we can compute precision and recall for the answer as below:

(defn- get-precision-recall
  [predicted expected]
  (let [predicted-set (set (normalize predicted))
        expected-set (set (normalize expected))
        common (clojure.set/intersection predicted-set expected-set )
        predicted-count (float (count predicted-set))
        expected-count (float (count expected-set))
        common-count (float (count common))]
    {:precision (precision common-count predicted-count)
     :recall (recall common-count expected-count)}))

And using the precision and recall scores we can compute the f1 score:

(defn- f1-score
  [predicted expected]
  (let [{:keys [precision recall]} (get-precision-recall predicted expected)
        denom (+ precision recall)
        score (if (> denom 0) (/ (* 2 precision recall) denom) 0)]
    {:precision precision
     :recall recall
     :f1 score}))

Gemma-4-E2b f1 scores

With the f1 scorer we can see better matches. We have 14 perfect scores and if we include partial matches 21/30 answers are good. Again, the f1 scorer is better than the exact match scorer but still leaves a lot to be desired. Semantically similar words will still be flagged as incorrect answers. Negations of answers are ignored. This leads us into more complex scorers like semantic scoring or LLM as a judge which we will look at in the next post.

The full Gemma F1 result table is on the results page.

What are evals?

By now everyone is aware of the non-deterministic nature of LLMs. Given this non-determinism how do we validate that our LLM-driven program is working as expected? What happens when we change the model? What happens when real users interact with the program - does it behave as expected? The solution to this is a combination of offline and online evaluations - evals for short. If you have written any LLM-driven program you have always been left with the unsettling feeling of how the program will respond to queries which I have not tried. Evals will help you there to an extent. There is no foolproof solution for this given the non-deterministic nature of LLMs.

You may have seen the GPT-4o rollback happening due to the extra agreeability OpenAI introduced due to a prompt change Sycophancy in GPT-4o. Evals are supposed to catch issues like that.

In my personal experience an AI agent went from previously reliable tool calling to broken tool calling when a model was upgraded. Luckily we had evals which caught this before the model change was deployed to production.

So, basically an eval is a repeatable set of experiments which help us measure how our AI application behaves.

In addition to these, evals will allow you to answer questions like:

• Can we switch to a cheaper model and maintain the quality?
• What improvements happened in experience due to the prompt change and what got worse?
• Are we ready to ship?

Components of an eval

Every eval will have three components - a dataset, a task and scorers.

Dataset

A dataset is a set of test cases. Standard test case terminology applies input which results in expected output with metadata for analysis. For example if we are building a system for answering factual questions, the input/output pairs may be like:

There can be datasets which have no well-defined outputs but will have guidelines on expected output. For example in a customer service agent the output will be expected to be polite and relevant to the question asked.

Task

The task is the functionality that we are evaluating. It is the function which converts the dataset input to the output. For example, in the case of the factual questions it might be "Answer the question truthfully" or for a customer service agent "Answer the customer questions politely and factually".

Scorers

There are multiple ways of scoring an eval:

• deterministic: A piece of code determines whether the task succeeded or not. Problems like the factual questions agent which have a single output will use these kinds of scorers.
• LLM as a judge: Confusing as it is we can use another LLM to judge the output of our LLM under test. Cases where the answer cannot be measured deterministically like the customer support scenario will use this.
• Human review: A human goes through each response manually and scores. As you can imagine this gets tedious very fast. In my opinion, the aim of evals is to minimize the human review process.

Manual evals

Let's try a manual eval to get a flavor of things. The agent that we built in the post Building a Coding Agent : Part 9 - Adding Command Handling will be used for this. First let's remove all the registered MCP servers to keep it simple. Let's try with the first question.

You: What is the capital of India?
LLM: The capital of India is New Delhi.

As you can see the response is a sentence but we would prefer short responses with just the answer. So, let's tweak the prompt for achieving the necessary response style. The prompt was changed from the developer-centric prompt to Respond factually, generate short answers. However, that doesn't help and we still get the same answer. Let's try with Respond factually, generate the shortest answer possible. This works, and we get the desired answer. Now we will use the other two questions also to check if this works as expected.

You: What is the capital of India?
LLM: New Delhi
You: What is the longest river in the world?
LLM: The Nile River.
You: Who wrote Harry Potter?
LLM: J.K. Rowling.

The answers are close - only the longest river question gets a slightly different answer which is also correct. We can now either further tweak our prompt or an alternate option is to tweak the dataset as The Nile River is also a valid answer and live with the prompt.

What we have done is a simple manual eval process which took 3 input queries and iterated the prompt to generate the correct response style. There is more to evals than this but it helps us get a flavor of things.

In this series, we will explore writing an eval framework for a customer service agent. The most common example of all eval tutorials. It is the todo list app of the LLM world.

If we look at the code for the agent right now, the chat loop is messy. It looks like

Get the user input -> Invoke the LLM

The code for the main loop looks like right now.

(loop [user-message (read-user-input!)
           messages [(get-system-prompt)]]
      (when (some? user-message)
        (let [new-messages (add-message-to-history messages user-message)
              {:keys [history usage]} (get-assistant-response new-messages config mcp-tools combined-registry)
              assistant-message (:content (last history))]
          (display-assistant-response! assistant-message)
          (dbg-print usage)
          (recur (read-user-input!) history))))

There is no way currently to do other things besides quit. That is also possible because we had added a hard-coded check inside the read-user-input! function to see if the user entered a quit message. When the user enters quit we return an empty input the same as if the user enter a EOF via Ctrl+D. This terminates the chat loop which is checking for the presence of some input from the user.

If you look at other agents available they provide /commands which allow the user to modify things like the conversation history, change models and so on. However, with our current read-user-input! method none of that is possible.

Let us refactor the method to make it extensible easily. We will achieve this by converting the simple loop into a state transition loop. We will add a new function handle-user-input! which will process special commands and indicate a state transition via a next state return value. Also, we want to be able to achieve things like clearing conversation history and changing models, so the handle-user-input! function needs to be able to change the state. To achieve this we will create a state map which is consists of the following:

{
  :history [] ; A vector which holds the conversation history
  :prompts {} ; A map which holds default prompts like the system prompt
  :config {} ; A map containing the model information
  :tools [] ; A vector of tools available to the model. Either MCP or coded tools
  :tool-registry {} ; A map of tool name to the invocation function. This will allow us to handle tool calls from the model
  :next-state :key ; The next state which the LLM chat loop should transition to
}

In this version we will support three states:

• :quit Quit the app
• :llm Invoke the LLM API with the current history
• :user Get input from the user

With these three states available our chat loop becomes simpler. The handle-user-input! function takes in the current state and returns a new state. This makes it easy for us to implement commands like clearing history, changing the model etc. As we can change the configuration which is stored inside the state map. After the implementation of our handle-user-input! function the main loop looks like:

(loop [{:keys [next-state] :as current-state} (state/handle-user-input! initial-state (read-user-input!))]
      (cond
        (= next-state :quit)
        (do
          (println "Exiting")
          (doseq [server servers]
            (println "Closing " (:name server))
            (mcpclient/close-client (:client server))))

        (= next-state :llm)
        (let [{:keys [history] :as response} (get-assistant-response current-state)]
          (display-assistant-response! response)
          (recur (state/handle-user-input! (assoc current-state :history history) (read-user-input!))))

        (= next-state :user)
        (recur (state/handle-user-input! current-state (read-user-input!))))

Our handle-user-input! function can be written as:

(defn handle-user-input!
  [{:keys [history prompts] :as state} input]
  (if
   (str/starts-with? input "/")
    (let [args (str/split input #" ")
          command (-> (first args) (subs 1) str/lower-case keyword)]
      (cond (= command :quit)
            (assoc state :next-state :quit)

            (= command :clear)
            (do
              (println "Clearing history")
              (assoc state :next-state :user
                     :history [(:system-prompt prompts)]))

            (= command :debug)
            (do
              (println "====== Current State ======")
              (pprint/pprint state)
              (println "===========================")
              (assoc state :next-state :user))

            (= command :model)
            (let [model-name (second args)
                  config (utils/read-config! (str "llm-" model-name ".edn"))]
              (if (some? config)
                (do
                  (println "Switching model to: " model-name)
                  (assoc state :config config :next-state :user))
                state))

            :else
            (assoc state :next-state :user)))
    (assoc state :next-state :llm
           :history (add-message-to-history history {:role "user" :content input}))))

With this function it is trivial to add new commands. I have added commands for quitting, clearing history, generating debug output and switching models. This is much better than the old loop which could only handle quit commands. This will set us up for adding more commands like saving and loading conversations as well. I think this kind of clean state pattern is easily achievable in Clojure which forces immutability on the programmer. If I had used a different programming language which allowed mutation easily, I would have state changes all over the code. This also makes it very easy to test this code as well as the inputs and outputs are predictable.

The full listing of the code is here