Protocol for a scorer

Earlier, we built an exact match scorer and an F1 scorer. These needed multiple functions

• a scoring function
• a result accumulation function
• an initial result shape which was used to accumulate the combined result

As we add more scorers, we need to define these over and over again with unique names and there is no way to group them. So, let's define a protocol which can be used to group related functions for the scorers.

The protocol defines three methods:

• score: This invokes the scoring function
• initial-result: Returns the initial value for the accumulated score which is used while doing a combination of all scores
• accumulate: This function combines individual results into an aggregate score which can be displayed

(defprotocol Scorer
  "A protocol for an eval scorer"
  (score [_ actual-result expected-result-list question] "Score a result given the list of possible expected results")
  (initial-result [_] "Get the initial accumulated result shape")
  (accumulate [_ accumulator result] "Combine the result into the accumulated result"))

The two scorers that we carried from the previous post are ExactMatchScorer and F1Scorer. We could also have gone with a simple map based collection of functions but I wanted to try out protocols here.

Now let's rewrite our two scorers using the protocol that we defined:

(defrecord ExactMatchScorer []
  Scorer
  (score
    [_ actual expected-list _question]
    (let [correct-answer? (if (seq expected-list)
                            (some #(str/includes? actual %) expected-list)
                            (str/includes? actual "Not Known"))]
      {:score (if correct-answer? 1 0)}))
  (initial-result [_] {:name "exact-match" :success 0 :failed 0 :partial 0})
  (accumulate
    [_ accumulated-result result]
    (let [score (get-in result [:score])]
      (cond
        (= 1 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
        (= 0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
        :else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))

(defrecord F1Scorer []
  Scorer
  (score
    [_ predicted expected-list _question]
    (apply max-key :f1 (map #(f1-score predicted %) (if (seq expected-list) expected-list ["Not Known"]))))
  (initial-result [_] {:name "f1" :success 0 :failed 0 :partial 0})
  (accumulate
    [_ accumulated-result result]
    (let [score (get-in result [:f1])]
      (cond
        (= 1.0 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
        (= 0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
        :else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))

LLM as a judge

In the previous post, we found that the code-only scorers had several issues where the matching logic became more convoluted to get a correct result. The solution in the evals world is to use another LLM to test the result. This sounds weird - using an LLM to check another LLM's output. Turtles all the way down.

Generally the practice followed is to use a more capable LLM to check the outputs of a smaller LLM. In our case, since we are using local LLMs, I will use a GPT-5.4 nano model to judge.

This is how we will structure the prompt to GPT-5.4-nano. It takes in the question, reference answers and the actual answer as parameters. In case a reference answer is not available we prompt the LLM judge to allow Not Known as an acceptable answer.

(defn llm-judge-prompt
 [question references answer]
  (str "You are an LLM judge evaluating a question-answering response against SQuAD reference answer(s).

Score the model answer based on factual and semantic correctness:

1.0 — Fully correct; equivalent to a reference answer.
0.5 — Partially correct; contains some correct information but is incomplete or has a minor factual error.
0.0 — Incorrect; gives the wrong answer, contradicts the reference, or answers a different question.

Accept paraphrases and equivalent wording. Ignore capitalization, punctuation, and formatting. Extra information is acceptable if it is correct and does not contradict the answer.

Question:" question
" Reference answer(s):" (if (seq references) references "Not Known")
" Model answer:" answer "
Return JSON only:
{\"score\": 0.0|0.5|1.0, \"reason\": \"brief explanation\"}"))

Armed with the above prompt, we can write a generic LLM as a judge scorer using our protocol.

(defn- get-score
  [config question]
  (let [messages [{:role "user" :content question}]
        response (openai/create-chat-completion {:model (:model config)
                                                 :messages messages}
                                                (select-keys config [:api-key :api-endpoint :impl]))]
    (get-in response [:choices 0 :message :content])))

(defrecord LLMJudge [config prompt-fn]
  Scorer
  (score
   [_ actual expected-list question]
   (let [response (get-score config (prompt-fn question expected-list actual))]
     (cheshire/parse-string response)))
  (initial-result [_] {:name "llm-judge" :success 0 :failed 0 :partial 0})
  (accumulate
    [_ accumulated-result result]
    (let [score (get-in result ["score"])]
      (cond
        (= 1.0 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
        (= 0.0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
        :else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))

After setting up the above scorers, a run with Qwen 3.0 0.6B looks like:

The overall scores are:

All three scores give a different rating for the responses. Hopefully you got a taste of how complex evals for an LLM based solution are. Even for a simple set of 30 questions most of which had fixed answers we could not get to a scoring scheme which was perfectly reliable and human supervision is needed to see whether the solution is performing as expected. Also, every run of the eval will show different scores, so a better way is needed to see stability of results across runs.

Evals dataset

I could have generated random questions for this exercise but there is a nice publicly available dataset called SQuAD. This has a set of 100K+ questions which can be answered by a model. Let me pick 20 random questions from the set. The questions are such that no context is needed for answering those. These are 20 questions with actual answers and 10 made up questions. I used Opus 5 to make up some unanswerable questions as the actual dataset unanswerable questions depend on the context in the dataset which we are not using here.

The complete dataset and raw results are on a separate page.

Get an answer to a question from the LLM

We will use an LLM to get an answer for a question. Let's keep it simple, pass in a config, system-prompt and a question and get a response back.

(defn- get-answer
  [config system-prompt question]
  (let [messages [{:role "system" :content system-prompt}
                  {:role "user" :content question}]
        response (openai/create-chat-completion {:model (:model config)
                                        :messages messages}
                                       (select-keys config [:api-key :api-endpoint :impl]))]
    (get-in response [:choices 0 :message :content])))

A simple scorer

Let's write a simple scorer. It will only check if one of the expected answers is fully present within the LLM response. Also, if the LLM response contains "Not Known" it will match an unanswerable question.

(defn scorer
  [expected-list actual]
  (let [correct-answer? (if (seq expected-list)
                          (some #(str/includes? actual %) expected-list)
                          (str/includes? actual "Not Known"))]
    (if correct-answer? 1 0)))

Scoring the list of questions

Now armed with the function to get an answer from a LLM endpoint and a scorer, we can write a simple score-question function.

(defn- score-question
  [config prompt question]
  (let [system-prompt (:content prompt)
        q (:question question)
        a (:answers question)
        actual (get-answer config system-prompt q)]
    {:question q
     :expected-answers a
     :actual actual
     :score (scorer a actual)
    }))

Results from Qwen 3.0 0.6B

The following table shows results from running the evals against a Qwen 3.0 0.6B model. It is a simple model so, the results are not very impressive. Even then they are still good for such a small model - 13 answers correct out of 30, a 43.3% correctness rate. The prompt given to the model was:

Answer questions concisely. There is no need for full sentences. Say, Not Known if you do not know or are unable to infer.

And, that prompt shows up partially in some of the answers - like the question about the European population killed by the Black Death. There are some interesting hallucinations also like the Portugese city where the Rhine reaches the sea. Our scorer also shows its limitations where case mismatches cause an answer to be marked as fail. Like the 20.8% gas question. Also, the model sometimes gives correct answers albeit partial (like Newton instead of Isaac Newton)

The full Qwen result table is on the results page.

Results from Gemma-4-E2B

Gemma being a larger model scores better 15/30 - around 50%. Again the limitations of our scorer show up which flags correct answers as wrong due to the case mismatch or punctuation issues. Let's look at alternate scorers to fix this issue.

The full Gemma substring result table is on the results page.

F1 scorer

The F1 scorer combines values of precision and recall to generate a score of the answer. Where Precision (P) = matching tokens/predicted tokens and Recall (R) = matching tokens/expected tokens. Precision punishes padding of answers, whereas Recall punishes omission of answers.

The F1 score is defined as 2PR/(P + R)

Ideally we should normalize the generated answers to improve the precision and recall metrics but to keep things simple I will just do a lower case of the words.

(defn- normalize
  [answer]
  (map str/lower-case (filter #(> (count %) 0) (str/split answer #"\s+|\.|,|-|!"))))

We the normalized tokens we can compute precision and recall for the answer as below:

(defn- get-precision-recall
  [predicted expected]
  (let [predicted-set (set (normalize predicted))
        expected-set (set (normalize expected))
        common (clojure.set/intersection predicted-set expected-set )
        predicted-count (float (count predicted-set))
        expected-count (float (count expected-set))
        common-count (float (count common))]
    {:precision (precision common-count predicted-count)
     :recall (recall common-count expected-count)}))

And using the precision and recall scores we can compute the f1 score:

(defn- f1-score
  [predicted expected]
  (let [{:keys [precision recall]} (get-precision-recall predicted expected)
        denom (+ precision recall)
        score (if (> denom 0) (/ (* 2 precision recall) denom) 0)]
    {:precision precision
     :recall recall
     :f1 score}))

Gemma-4-E2b f1 scores

With the f1 scorer we can see better matches. We have 14 perfect scores and if we include partial matches 21/30 answers are good. Again, the f1 scorer is better than the exact match scorer but still leaves a lot to be desired. Semantically similar words will still be flagged as incorrect answers. Negations of answers are ignored. This leads us into more complex scorers like semantic scoring or LLM as a judge which we will look at in the next post.

The full Gemma F1 result table is on the results page.

What are evals?

By now everyone is aware of the non-deterministic nature of LLMs. Given this non-determinism how do we validate that our LLM-driven program is working as expected? What happens when we change the model? What happens when real users interact with the program - does it behave as expected? The solution to this is a combination of offline and online evaluations - evals for short. If you have written any LLM-driven program you have always been left with the unsettling feeling of how the program will respond to queries which I have not tried. Evals will help you there to an extent. There is no foolproof solution for this given the non-deterministic nature of LLMs.

You may have seen the GPT-4o rollback happening due to the extra agreeability OpenAI introduced due to a prompt change Sycophancy in GPT-4o. Evals are supposed to catch issues like that.

In my personal experience an AI agent went from previously reliable tool calling to broken tool calling when a model was upgraded. Luckily we had evals which caught this before the model change was deployed to production.

So, basically an eval is a repeatable set of experiments which help us measure how our AI application behaves.

In addition to these, evals will allow you to answer questions like:

• Can we switch to a cheaper model and maintain the quality?
• What improvements happened in experience due to the prompt change and what got worse?
• Are we ready to ship?

Components of an eval

Every eval will have three components - a dataset, a task and scorers.

Dataset

A dataset is a set of test cases. Standard test case terminology applies input which results in expected output with metadata for analysis. For example if we are building a system for answering factual questions, the input/output pairs may be like:

There can be datasets which have no well-defined outputs but will have guidelines on expected output. For example in a customer service agent the output will be expected to be polite and relevant to the question asked.

Task

The task is the functionality that we are evaluating. It is the function which converts the dataset input to the output. For example, in the case of the factual questions it might be "Answer the question truthfully" or for a customer service agent "Answer the customer questions politely and factually".

Scorers

There are multiple ways of scoring an eval:

• deterministic: A piece of code determines whether the task succeeded or not. Problems like the factual questions agent which have a single output will use these kinds of scorers.
• LLM as a judge: Confusing as it is we can use another LLM to judge the output of our LLM under test. Cases where the answer cannot be measured deterministically like the customer support scenario will use this.
• Human review: A human goes through each response manually and scores. As you can imagine this gets tedious very fast. In my opinion, the aim of evals is to minimize the human review process.

Manual evals

Let's try a manual eval to get a flavor of things. The agent that we built in the post Building a Coding Agent : Part 9 - Adding Command Handling will be used for this. First let's remove all the registered MCP servers to keep it simple. Let's try with the first question.

You: What is the capital of India?
LLM: The capital of India is New Delhi.

As you can see the response is a sentence but we would prefer short responses with just the answer. So, let's tweak the prompt for achieving the necessary response style. The prompt was changed from the developer-centric prompt to Respond factually, generate short answers. However, that doesn't help and we still get the same answer. Let's try with Respond factually, generate the shortest answer possible. This works, and we get the desired answer. Now we will use the other two questions also to check if this works as expected.

You: What is the capital of India?
LLM: New Delhi
You: What is the longest river in the world?
LLM: The Nile River.
You: Who wrote Harry Potter?
LLM: J.K. Rowling.

The answers are close - only the longest river question gets a slightly different answer which is also correct. We can now either further tweak our prompt or an alternate option is to tweak the dataset as The Nile River is also a valid answer and live with the prompt.

What we have done is a simple manual eval process which took 3 input queries and iterated the prompt to generate the correct response style. There is more to evals than this but it helps us get a flavor of things.

In this series, we will explore writing an eval framework for a customer service agent. The most common example of all eval tutorials. It is the todo list app of the LLM world.