How we measure what an agent has delivered

If you manage AI agents like employees, sooner or later you have to be able to say what they actually deliver. The obvious route would be to let a language model look at the work and award a grade. We did not take that route.

The reason is simple: nobody can recheck a grade like that. It comes into being inside the model, and if somebody asks why it turned out that way, there is no answer except another model answer. For a number an organisation bases decisions on, that is not enough.

What we do instead

Here, a metric is a counting rule. It counts events the platform has logged anyway, because it carried them out. No model judges anything, nothing is graded.

Every agent brings its own rules in a file called KPIS.md. That file states, for example, that a resolved ticket should be counted and what such a unit costs. What then gets counted are the actions this agent verifiably performed.

The result reads like a price list. This is how it is built, with invented example figures:

Tickets resolved          142      3.20 € / unit
Code reviews               38      1.05 € / unit
Handed to a human          19      —
Runs failed                 7      —

That small table contains two decisions that matter more to us than the figures themselves.

The count always stands beside the price

A unit price on its own says almost nothing. It might come from a thousand events or from three. So the count always stands beside it, and below a minimum count we leave the price out entirely.

The reason is chance. An average over three events fluctuates so much that it tells you nothing — it merely looks like a measurement. A number that feigns a precision it does not have is worse than no number, because somebody will use it.

The last two lines are counter-figures

"Handed to a human" and "Runs failed" have no price. They are in the list all the same, and without them the list would be dangerous.

An agent that passes every difficult case to a human and only handles the easy ones itself would have excellent unit costs. Everything it did not solve would be absent from a list without those lines. It would look exceptionally efficient, when all it did was move the work.

Beside them stand four further figures that put the price in context: how often did a case come back and need rework? How often was the work rejected at an approval step? How quickly did the first response arrive, and how long did the whole case take? Three of those four needed no new instrumentation — the data was already there and had simply never been evaluated.

They become useful when compared over time. A rising cost curve next to a flat delivery volume is a finding worth pursuing. Either curve on its own would just be a number.

Why the agent does not see its own measuring rule

One rule follows from all of this, and at first glance it looks excessive: the KPIS.md file is never given to the agent in its prompt. It does not know what it is measured by.

The reason has been known for as long as metrics have existed. An agent that knows the number of its comments is being counted writes more comments. That is not misbehaviour and not a model failure — it is exactly the task as it was set. Disclose the target figure and you get the target figure optimised, not the work it was supposed to stand in for.

The same reasoning applies in a second place. Covey Doctor is an agent whose subject is the workforce itself. It may read a colleague's configuration and write a proposal to improve it, and it may do the same for its own configuration. What it may not do is read its own metrics. Otherwise it would improve the measurement rather than the work, and with considerably more leverage than an ordinary agent.

When a rule counts nothing

That leaves a practical case: somebody writes a counting rule that matches nothing — a typo in an action name, or an event this agent never triggers.

The convenient answer would be to report zero. It is also the worst one, because a zero looks like a result. An agent with a broken rule would sit in the list for months showing "delivered nothing", and eventually somebody believes it.

So a rule that cannot count anything reports itself in the configuration check. It reports a configuration error, not a performance of zero. That is the same line we follow everywhere: whatever the platform reports has to be something you can close out — by fixing it, or by somebody recording that they have taken it on. A notice that can never go away stops being read after two weeks, and so does the one next to it.