Notes on learning with Clojure.
Earlier, we built an exact match scorer and an F1 scorer. These needed multiple functions
As we add more scorers, we need to define these over and over again with unique names and there is no way to group them. So, let's define a protocol which can be used to group related functions for the scorers.
The protocol defines three methods:
(defprotocol Scorer
"A protocol for an eval scorer"
(score [_ actual-result expected-result-list question] "Score a result given the list of possible expected results")
(initial-result [_] "Get the initial accumulated result shape")
(accumulate [_ accumulator result] "Combine the result into the accumulated result"))
The two scorers that we carried from the previous post are ExactMatchScorer and F1Scorer. We could also have gone with a simple map based collection of functions but I wanted to try out protocols here.
Now let's rewrite our two scorers using the protocol that we defined:
(defrecord ExactMatchScorer []
Scorer
(score
[_ actual expected-list _question]
(let [correct-answer? (if (seq expected-list)
(some #(str/includes? actual %) expected-list)
(str/includes? actual "Not Known"))]
{:score (if correct-answer? 1 0)}))
(initial-result [_] {:name "exact-match" :success 0 :failed 0 :partial 0})
(accumulate
[_ accumulated-result result]
(let [score (get-in result [:score])]
(cond
(= 1 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
(= 0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
:else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))
(defrecord F1Scorer []
Scorer
(score
[_ predicted expected-list _question]
(apply max-key :f1 (map #(f1-score predicted %) (if (seq expected-list) expected-list ["Not Known"]))))
(initial-result [_] {:name "f1" :success 0 :failed 0 :partial 0})
(accumulate
[_ accumulated-result result]
(let [score (get-in result [:f1])]
(cond
(= 1.0 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
(= 0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
:else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))
In the previous post, we found that the code-only scorers had several issues where the matching logic became more convoluted to get a correct result. The solution in the evals world is to use another LLM to test the result. This sounds weird - using an LLM to check another LLM's output. Turtles all the way down.
Generally the practice followed is to use a more capable LLM to check the outputs of a smaller LLM. In our case, since we are using local LLMs, I will use a GPT-5.4 nano model to judge.
This is how we will structure the prompt to GPT-5.4-nano. It takes in the question, reference answers and the actual answer as parameters. In case a reference answer is not available we prompt the LLM judge to allow Not Known as an acceptable answer.
(defn llm-judge-prompt
[question references answer]
(str "You are an LLM judge evaluating a question-answering response against SQuAD reference answer(s).
Score the model answer based on factual and semantic correctness:
1.0 — Fully correct; equivalent to a reference answer.
0.5 — Partially correct; contains some correct information but is incomplete or has a minor factual error.
0.0 — Incorrect; gives the wrong answer, contradicts the reference, or answers a different question.
Accept paraphrases and equivalent wording. Ignore capitalization, punctuation, and formatting. Extra information is acceptable if it is correct and does not contradict the answer.
Question:" question
" Reference answer(s):" (if (seq references) references "Not Known")
" Model answer:" answer "
Return JSON only:
{\"score\": 0.0|0.5|1.0, \"reason\": \"brief explanation\"}"))
Armed with the above prompt, we can write a generic LLM as a judge scorer using our protocol.
(defn- get-score
[config question]
(let [messages [{:role "user" :content question}]
response (openai/create-chat-completion {:model (:model config)
:messages messages}
(select-keys config [:api-key :api-endpoint :impl]))]
(get-in response [:choices 0 :message :content])))
(defrecord LLMJudge [config prompt-fn]
Scorer
(score
[_ actual expected-list question]
(let [response (get-score config (prompt-fn question expected-list actual))]
(cheshire/parse-string response)))
(initial-result [_] {:name "llm-judge" :success 0 :failed 0 :partial 0})
(accumulate
[_ accumulated-result result]
(let [score (get-in result ["score"])]
(cond
(= 1.0 score) (assoc accumulated-result :success (inc (:success accumulated-result)))
(= 0.0 score) (assoc accumulated-result :failed (inc (:failed accumulated-result)))
:else (assoc accumulated-result :partial (inc (:partial accumulated-result)))))))
After setting up the above scorers, a run with Qwen 3.0 0.6B looks like:
| Question | Answer | Expected | EM Score | F1 Score | LLM Score |
|---|---|---|---|---|---|
| Which Nobel Prize did Carl Wilhelm Scheele receive for discovering oxygen? | The Nobel Prize in Chemistry was awarded to Carl Wilhelm Scheele for discovering oxygen in 1839. | [] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference says the information is not known, but the model claims Scheele received the Nobel Prize in Chemistry in 1839. This is incorrect: Carl Wilhelm Scheele did not receive a Nobel Prize (the Nobel Prize started in 1901, and his discovery predates it)."} |
| In what year did Pierre de Fermat declare Fermat's little theorem? | Not Known if you do not know or are unable to infer. | [In 1640 1640] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer does not provide the year (it says not known), while the reference answer states it was 1640."} |
| How much of the European population did the black death kill? | Not Known if you do not know or are unable to infer. | [30–60% of Europe's total population 30–60%] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer does not provide the required figure (30–60%); it instead says the information is unknown."} |
| Who was the first human to discover fire? | Who was the first human to discover fire? Humans. | [] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference answer is 'Not Known', but the model claims 'Humans' as the first to discover fire, which contradicts the reference."} |
| What is the largest city the Rhine runs through? | The Rhine runs through Lyon, France, and is the largest city in that region. | [Cologne, Germany Cologne] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer incorrectly states the Rhine runs through Lyon, France. The reference answer identifies Cologne, Germany as the largest city the Rhine runs through."} |
| What was John Harvard's exact date of birth? | Not Known. | [] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer matches the reference: John Harvard's exact date of birth is not known."} |
| When did Khan formally declare the Yuan dynasty? | Not Known if you do not know or are unable to infer. | [1271] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer does not provide the date. The reference answer is 1271, so it fails to answer the question."} |
| In what country is Normandy located? | France | [France] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer correctly states that Normandy is located in France, matching the reference."} |
| In what year was the charter granted for Harvard Corporation? | Not Known if you do not know or are unable to infer. | [1650] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer does not provide the year the charter was granted for Harvard Corporation, so it cannot be compared to the reference (1650)."} |
| What is the largest city of Poland? | Warsaw. | [Warsaw] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer is "Warsaw," which matches the reference answer."} |
| Exactly how many people died of the Black Death in Asia? | The exact number of people who died of the Black Death in Asia is not known, but estimates suggest about 20–30 million. | [] | {:score 0} | {:precision 0.1, :recall 1.0, :f1 0.18181818181818182} | {"score" 0.0, "reason" "The reference answer states the exact number is not known. The model adds a specific estimate (20–30 million), which contradicts the reference’s 'Not Known' response."} |
| What is the largest prime number? | No Largest Prime Number. | [] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.5, "reason" "The model answer correctly states that there is no largest prime number. However, the reference answer is marked as 'Not Known', so it does not exactly match the provided reference."} |
| In what year was the P versus NP problem solved? | 1982 | [] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference answer states the year is not known, but the model answers 1982, which contradicts the reference."} |
| When was the French and Indian War? | The French and Indian War occurred in 1762. | [1754–1763] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference states the French and Indian War occurred from 1754 to 1763, but the model answer says 1762 only, which does not match the full date range."} |
| In which Portuguese city does the Rhine reach the sea? | Not Known. | [] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "Both the reference answer and the model answer state that the information is not known."} |
| In what year did Kublai Khan establish a Yuan dynasty colony in Australia? | Not Known. | [] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer matches the reference answer: 'Not Known'."} |
| What is the term used to identify a deterministic Turing machine that has additional random bits? | Probabilistic. | [A probabilistic Turing machine probabilistic probabilistic Turing machine] | {:score 0} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer "Probabilistic" correctly identifies the term used for a deterministic Turing machine with additional random bits (i.e., a probabilistic Turing machine)."} |
| What is the French name for the Rhine? | Rhine. | [Rhin] | {:score 1} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 1.0, "reason" "The model answer "Rhine" matches the reference French name "Rhin" (equivalent spelling as given in the reference, capitalization differences ignored)."} |
| In what constituent country of the United Kingdom is Merthyr Tydfil located? | Wales | [Wales south Wales] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "Merthyr Tydfil is located in Wales, which matches the reference answer."} |
| What is the smallest odd perfect number? | Not Known. | [] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "Both the reference and the model answer state that the smallest odd perfect number is not known."} |
| Which book by Edward Said portrayed the east as being the "others?" | The book by Edward Said that portrayed the East as being the "others" is The Orientalist. | [Orientalism] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference answer is "Orientalism," but the model answer says "The Orientalist," which is not the stated book in the reference."} |
| Who produced the first geological map of the U.S.? | The United States Geological Survey (USGS) was established in 1902. | [William Maclure] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer incorrectly states that the USGS was established in 1902. The correct answer is that William Maclure produced the first geological map of the U.S."} |
| When did Carl Wilhelm Scheele discover oxygen? | Not Known. | [1773 1773 or earlier] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference states Scheele discovered oxygen in 1773 or earlier, but the model answer says 'Not Known,' which does not match the reference information."} |
| What is the largest stadium in Australia? | Sydney City Stadium. | [Melbourne Cricket Ground The Melbourne Cricket Ground] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer 'Sydney City Stadium' is not the largest stadium in Australia; the reference answer is the Melbourne Cricket Ground (MCG)."} |
| Which gas makes up 20.8% of the Earth's atmosphere? | Oxygen. | [Diatomic oxygen Diatomic oxygen gas] | {:score 0} | {:precision 1.0, :recall 0.5, :f1 0.6666666666666666} | {"score" 0.5, "reason" "The model answer says “Oxygen,” which is broadly correct, but it doesn’t specify that the 20.8% is diatomic oxygen (O2), as in the reference."} |
| Who expounded the Three Laws of Motion? | Isaac Newton. | [Isaac Newton] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer correctly identifies Isaac Newton as the person who expounded the Three Laws of Motion."} |
| In what year did Isaac Newton publish his fourth law of motion? | 1687 | [] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The reference answer is 'Not Known', but the model claims Newton published his fourth law of motion in 1687, which is not supported by the provided reference."} |
| Which theory states that slow geological processes are still occurring today, and have occurred throughout Earth's history? | The theory of plate tectonics. | [uniformitarianism] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer identifies plate tectonics, but the reference asks for uniformitarianism, which states that slow geological processes continue today and throughout Earth's history."} |
| Who demonstrated how to create a perfect number from a Mersenne prime? | Not Known if you do not know or are unable to infer. | [Euclid] | {:score 0} | {:precision 0.0, :recall 0.0, :f1 0} | {"score" 0.0, "reason" "The model answer does not provide the required figure (Euclid) and instead says it is not known."} |
| What is the Chinese name for the Yuan dynasty? | 元朝 | [Yuán Cháo 元朝] | {:score 1} | {:precision 1.0, :recall 1.0, :f1 1.0} | {"score" 1.0, "reason" "The model answer '元朝' exactly matches the reference answer (元朝 / 元朝)."} |
The overall scores are:
| Scorer | Partial | Correct | Incorrect |
|---|---|---|---|
| exact-match | 0 | 10 | 20 |
| f1 | 2 | 10 | 18 |
| llm-judge | 2 | 11 | 17 |
All three scores give a different rating for the responses. Hopefully you got a taste of how complex evals for an LLM based solution are. Even for a simple set of 30 questions most of which had fixed answers we could not get to a scoring scheme which was perfectly reliable and human supervision is needed to see whether the solution is performing as expected. Also, every run of the eval will show different scores, so a better way is needed to see stability of results across runs.
Published: 2026-09-11