What are evals?

By now everyone is aware of the non-deterministic nature of LLMs. Given this non-determinism how do we validate that our LLM-driven program is working as expected? What happens when we change the model? What happens when real users interact with the program - does it behave as expected? The solution to this is a combination of offline and online evaluations - evals for short. If you have written any LLM-driven program you have always been left with the unsettling feeling of how the program will respond to queries which I have not tried. Evals will help you there to an extent. There is no foolproof solution for this given the non-deterministic nature of LLMs.

You may have seen the GPT-4o rollback happening due to the extra agreeability OpenAI introduced due to a prompt change Sycophancy in GPT-4o. Evals are supposed to catch issues like that.

In my personal experience an AI agent went from previously reliable tool calling to broken tool calling when a model was upgraded. Luckily we had evals which caught this before the model change was deployed to production.

So, basically an eval is a repeatable set of experiments which help us measure how our AI application behaves.

In addition to these, evals will allow you to answer questions like:

• Can we switch to a cheaper model and maintain the quality?
• What improvements happened in experience due to the prompt change and what got worse?
• Are we ready to ship?

Components of an eval

Every eval will have three components - a dataset, a task and scorers.

Dataset

A dataset is a set of test cases. Standard test case terminology applies input which results in expected output with metadata for analysis. For example if we are building a system for answering factual questions, the input/output pairs may be like:

There can be datasets which have no well-defined outputs but will have guidelines on expected output. For example in a customer service agent the output will be expected to be polite and relevant to the question asked.

Task

The task is the functionality that we are evaluating. It is the function which converts the dataset input to the output. For example, in the case of the factual questions it might be "Answer the question truthfully" or for a customer service agent "Answer the customer questions politely and factually".

Scorers

There are multiple ways of scoring an eval:

• deterministic: A piece of code determines whether the task succeeded or not. Problems like the factual questions agent which have a single output will use these kinds of scorers.
• LLM as a judge: Confusing as it is we can use another LLM to judge the output of our LLM under test. Cases where the answer cannot be measured deterministically like the customer support scenario will use this.
• Human review: A human goes through each response manually and scores. As you can imagine this gets tedious very fast. In my opinion, the aim of evals is to minimize the human review process.

Manual evals

Let's try a manual eval to get a flavor of things. The agent that we built in the post Building a Coding Agent : Part 9 - Adding Command Handling will be used for this. First let's remove all the registered MCP servers to keep it simple. Let's try with the first question.

You: What is the capital of India?
LLM: The capital of India is New Delhi.

As you can see the response is a sentence but we would prefer short responses with just the answer. So, let's tweak the prompt for achieving the necessary response style. The prompt was changed from the developer-centric prompt to Respond factually, generate short answers. However, that doesn't help and we still get the same answer. Let's try with Respond factually, generate the shortest answer possible. This works, and we get the desired answer. Now we will use the other two questions also to check if this works as expected.

You: What is the capital of India?
LLM: New Delhi
You: What is the longest river in the world?
LLM: The Nile River.
You: Who wrote Harry Potter?
LLM: J.K. Rowling.

The answers are close - only the longest river question gets a slightly different answer which is also correct. We can now either further tweak our prompt or an alternate option is to tweak the dataset as The Nile River is also a valid answer and live with the prompt.

What we have done is a simple manual eval process which took 3 input queries and iterated the prompt to generate the correct response style. There is more to evals than this but it helps us get a flavor of things.

In this series, we will explore writing an eval framework for a customer service agent. The most common example of all eval tutorials. It is the todo list app of the LLM world.

If we look at the code for the agent right now, the chat loop is messy. It looks like

Get the user input -> Invoke the LLM

The code for the main loop looks like right now.

(loop [user-message (read-user-input!)
           messages [(get-system-prompt)]]
      (when (some? user-message)
        (let [new-messages (add-message-to-history messages user-message)
              {:keys [history usage]} (get-assistant-response new-messages config mcp-tools combined-registry)
              assistant-message (:content (last history))]
          (display-assistant-response! assistant-message)
          (dbg-print usage)
          (recur (read-user-input!) history))))

There is no way currently to do other things besides quit. That is also possible because we had added a hard-coded check inside the read-user-input! function to see if the user entered a quit message. When the user enters quit we return an empty input the same as if the user enter a EOF via Ctrl+D. This terminates the chat loop which is checking for the presence of some input from the user.

If you look at other agents available they provide /commands which allow the user to modify things like the conversation history, change models and so on. However, with our current read-user-input! method none of that is possible.

Let us refactor the method to make it extensible easily. We will achieve this by converting the simple loop into a state transition loop. We will add a new function handle-user-input! which will process special commands and indicate a state transition via a next state return value. Also, we want to be able to achieve things like clearing conversation history and changing models, so the handle-user-input! function needs to be able to change the state. To achieve this we will create a state map which is consists of the following:

{
  :history [] ; A vector which holds the conversation history
  :prompts {} ; A map which holds default prompts like the system prompt
  :config {} ; A map containing the model information
  :tools [] ; A vector of tools available to the model. Either MCP or coded tools
  :tool-registry {} ; A map of tool name to the invocation function. This will allow us to handle tool calls from the model
  :next-state :key ; The next state which the LLM chat loop should transition to
}

In this version we will support three states:

• :quit Quit the app
• :llm Invoke the LLM API with the current history
• :user Get input from the user

With these three states available our chat loop becomes simpler. The handle-user-input! function takes in the current state and returns a new state. This makes it easy for us to implement commands like clearing history, changing the model etc. As we can change the configuration which is stored inside the state map. After the implementation of our handle-user-input! function the main loop looks like:

(loop [{:keys [next-state] :as current-state} (state/handle-user-input! initial-state (read-user-input!))]
      (cond
        (= next-state :quit)
        (do
          (println "Exiting")
          (doseq [server servers]
            (println "Closing " (:name server))
            (mcpclient/close-client (:client server))))

        (= next-state :llm)
        (let [{:keys [history] :as response} (get-assistant-response current-state)]
          (display-assistant-response! response)
          (recur (state/handle-user-input! (assoc current-state :history history) (read-user-input!))))

        (= next-state :user)
        (recur (state/handle-user-input! current-state (read-user-input!))))

Our handle-user-input! function can be written as:

(defn handle-user-input!
  [{:keys [history prompts] :as state} input]
  (if
   (str/starts-with? input "/")
    (let [args (str/split input #" ")
          command (-> (first args) (subs 1) str/lower-case keyword)]
      (cond (= command :quit)
            (assoc state :next-state :quit)

            (= command :clear)
            (do
              (println "Clearing history")
              (assoc state :next-state :user
                     :history [(:system-prompt prompts)]))

            (= command :debug)
            (do
              (println "====== Current State ======")
              (pprint/pprint state)
              (println "===========================")
              (assoc state :next-state :user))

            (= command :model)
            (let [model-name (second args)
                  config (utils/read-config! (str "llm-" model-name ".edn"))]
              (if (some? config)
                (do
                  (println "Switching model to: " model-name)
                  (assoc state :config config :next-state :user))
                state))

            :else
            (assoc state :next-state :user)))
    (assoc state :next-state :llm
           :history (add-message-to-history history {:role "user" :content input}))))

With this function it is trivial to add new commands. I have added commands for quitting, clearing history, generating debug output and switching models. This is much better than the old loop which could only handle quit commands. This will set us up for adding more commands like saving and loading conversations as well. I think this kind of clean state pattern is easily achievable in Clojure which forces immutability on the programmer. If I had used a different programming language which allowed mutation easily, I would have state changes all over the code. This also makes it very easy to test this code as well as the inputs and outputs are predictable.

The full listing of the code is here

Using Ollama models

Now that we built a decently working coding agent, let us see if we can make it run against a local model. There are a lot of claims on the internet about local models being the future of this space. Let us validate if that claim holds true.

We will use Ollama to run a local model. Luckily for us it supports the OpenAI API out of the box. So, if you connect to http://localhost:11434/v1, you can reuse the agent that we already wrote to try out the different models.

My setup is a AMD Ryzen 1600 desktop with 32GB RAM and a RTX 3060 12 GB. So, I am both CPU and GPU poor :D.

Llama 3.2 (3B)

The first one I tried is Llama 3.2 (3b). We have to look for models which support tool calling for the agent to work. For example, the Gemma models from Google do not support tool calling and so cannot be used with our agent. The model download was relatively small for Llama 3.2 - around 2 GB. It was decently fast also due to it fitting in the VRAM of my device completely. I could get responses within 2 seconds. However, that was where the positives ended. The model hallucinates tool calls and calls them incorrectly - mostly it is unable to generate the correct tool call format. I gave it a task of writing a Javascript FizzBuzz program and this is what I got.

You : Write me a FizzBuzz Javascript program with the tools available to you
Tool : run_shell_command
{:command
 "node -e <garbage generated>}
LLM : It seems like the shell command is not executed correctly.

Here's a simplified version of FizzBuzz written in JavaScript using a conventional syntax:

function fizzBuzz(n) {
    let output = '';
    for (let i = 1; i <= n; i++) {
        if (i % 3 === 0 && i % 5 === 0) {
            output += 'FizzBuzz ';
        } else if (i % 3 === 0) {
            output += 'Fizz ';
        } else if (i % 5 === 0) {
            output += 'Buzz ';
        } else {
            output += i + ' ';
        }
    }
    console.log(output);
}

fizzBuzz(20);

This function prints out the numbers from 1 to n with a twist: multiples of three are replaced by "Fizz", multiples of five by "Buzz", and multiples of both by "FizzBuzz".

To call this function, you can run it in your environment:

javascript
function fizzBuzz(n) {
    let output = '';
    for (let i = 1; i <= n; i++) {
        if (i % 3 === 0 && i % 5 === 0) {
            output += 'FizzBuzz ';
        } else if (i % 3 === 0) {
            output += 'Fizz ';
        } else if (i % 5 === 0) {
            output += 'Buzz ';
        } else {
            output += i + ' ';
        }
    }
    console.log(output);
}

fizzBuzz(20);
Debug : {:completion_tokens 334, :prompt_tokens 227, :total_tokens 561} Response took -  3.772  s
You : quit

So, this model is not performing well. It could not even call the tool to read/write files correctly.

Qwen 2.5 coder (1.5B)

I had heard a lot of great things about the Qwen coding models. So, I tried the smallest of those next - the one with 1.5B parameters. Ollama downloads around 1GB of model data for this to run. I asked it what the program in the current folder does?

You : What does the program in the current folder do?
LLM : The program in the current folder is a Python script named <script-name>, and it uses the run_shell_command function from the tools section to execute the command find . -type f -exec cat {} + > results.txt.

json
{
  "name": "run_shell_command",
  "arguments": {
    "command": "find ./ -type f -exec cat {} + > results.txt"
  }
}
Debug : {:completion_tokens 91, :prompt_tokens 2405, :total_tokens 2496} Response took -  3.842  s
You : quit

No dice. It is also not able to run any tools. Maybe I should try with a larger model.

Qwen 3 VL (8b)

Maybe the size of the model is an issue. Let us try with a larger model. The latest Qwen 3 VL (8b) parameters. This one manages tool calls perfectly and is able to answer what the code in the current project does.

You : What does the program in the current folder do?
Tool : list_directory
{:path ". "}
Tool : list_allowed_directories
{}
Tool : list_directory
{:path "/home/sgopale/Source/code-editing-agent"}
Tool : list_directory
{:path "/home/sgopale/Source/code-editing-agent/src"}
Tool : list_directory
{:path "/home/sgopale/Source/code-editing-agent/src/agent"}
Tool : read_text_file
{:path "/home/sgopale/Source/code-editing-agent/src/agent/core.clj"}
Reasoning : Okay, so the user asked what the program in the current folder does. After listing the directories and checking the contents, I found that the main program is likely in the 'agent' directory's core.clj file. The tool response shows that core.clj is a Clojure script.

Looking at the content, it defines the namespace `agent.core`, which includes functions for interacting with LLM APIs, handling tool calls, and a main entry point `-main`. The code uses libraries like `agent.mcpclient`, `agent.state`, and `agent.tools`, which suggest it's an assistant or agent system that communicates with an LLM (like OpenAI) and handles tool invocations.

Specifically, the `-main` function initializes the system by reading configuration from `llm-ollama.edn`, getting tools from `agent.tools`, and processing user input. It uses a loop to handle user interactions, calling the LLM for responses, and managing tool calls. The code includes features like displaying responses, handling tool registration, and exiting the application.

The file also has a commented `-main` call, indicating it's meant to be run as the main entry point. The program seems to be an assistant that processes user inputs through an LLM, executes tools (like API calls or functions), and manages the conversation flow.

So putting it all together, the program is a CLI-based assistant that uses LLMs and tools to interact with users, likely part of a larger application for AI-assisted tasks.
LLM : The program in the current folder is a Clojure-based interactive agent/assistant system designed to interact with LLMs (Large Language Models) and execute tools based on user input. Below is a breakdown of its purpose and functionality:

Key Features & Purpose


1. LLM Integration


- Connects to external LLM APIs (e.g., Ollama, OpenAI) via wkok.openai-clojure.api and agent.mcpclient.
- Handles tool invocations (e.g., functions like mcpclient/get-servers) and processes responses.
2. Tool-Based Workflow


- Registers and executes custom tools (e.g., from agent.tools namespace).
- Dynamically manages tool responses (e.g., parsing tool calls, handling errors).
- Supports nested interactions where LLM output triggers tool execution, which then feeds back into the conversation.
3. User Interaction Loop


- Starts with a system prompt and reads user input via read-user-input!.
- Processes inputs through a loop:


- Calls the LLM to generate responses.
- Handles tool calls (e.g., invoke-tool).
- Displays results to the user (display-assistant-response!).
- Continues until the user exits (:quit).
4. Configuration & State Management


- Loads configuration from llm-ollama.edn (e.g., API keys, endpoints).
- Tracks conversation history (:history) and tool registry (:tools-registry).
- Maintains state across interactions (e.g., state/handle-user-input!).
5. Debugging & Logging


- Uses dbg-print for debug logs (e.g., tool registration, errors).
- Prints warnings for edge cases (e.g., multiple LLM responses).

Intended Use Case

This program is part of a CLI-friendly AI assistant system, likely used for:


- Building chatbots or interactive workflows.
- Testing LLM tool integration (e.g., connecting to APIs like Ollama).
- Processing user queries with dynamic tool chains (e.g., "Explain X, then fetch data from API Y").
It’s designed to be configurable (via llm-ollama.edn), extendable (by adding tools to agent.tools), and interactive (via read-user-input!).

Summary

The program is an LLM-driven assistant that:


- Interacts with users through a loop.
- Executes tools dynamically (e.g., API calls, custom functions).
- Handles errors and logs for debugging.
- Is configured via llm-ollama.edn and tools in agent.tools.
It’s ideal for AI-powered workflows requiring dynamic tool execution and natural language interactions. To run it, call (-main) in the Clojure REPL.
Debug : {:completion_tokens 1140, :prompt_tokens 7151, :total_tokens 8291} Response took -  44.885  s

Finally success. This model works reliably with tool calling and does a decent job of summarizing what the code does. However, it is too slow for me. It took approximately 45 seconds to process this prompt. Comparing it with an Azure hosted GPT-5 mini model, the online model takes 11 seconds. The online model is running at the cheapest config possible so can be faster if we choose to pay more.

So, even if the future of AI agents is local models. I believe we are not in that future yet, at least for coding agents which can call tools. Maybe these models are good for answering questions about code. I have not tried them for that use case. I will stick to online hosted models for my workflows.