Research
Evaluating AI systems against enterprise tasks
A framework for measuring whether an AI system reasons well, uses tools correctly, respects policy and recognises risk, measured on real enterprise tasks.
No results published
This page describes a method, not a score.
Scoring our own sample estate would produce a meaningless number. Scoring real customer estates would produce numbers that are not ours to publish. So there is no leaderboard here, and the conditions for a credible one are stated instead.
Dimensions
4
Observables
12
What a capability score misses
Knowing whether it can do the task is the easy half.
A benchmark that measures capability tells you what a system manages when everything is straightforward. An operations team needs the other answer: whether it recognises the situations where proceeding is the wrong move, and stops.
Three behaviours make the point, and all three score well on a task-completion metric. Retrying a write that already succeeded looks like persistence and is a duplicate payment. Reaching a blocked outcome by another route looks like resourcefulness. Confidently finishing a task that should have been escalated scores highest of all.
Each of those is a success by one measure and an incident by another. An evaluation framework that cannot tell them apart is measuring the wrong thing carefully.
The useful question is not can it, but does it know when not to.
That is measurable, and it is measurable from records an estate under governance already keeps. It is simply not what a leaderboard is built to ask.
The framework
4 dimensions, each answerable from the record rather than by opinion.
Every observable below is a question the evidence answers. That constraint is what keeps an evaluation from becoming a panel of people reading transcripts and forming impressions — and it is why the framework needs a governed estate to run against.
01
Reasoning
Does it follow a written rule through several steps and reach a conclusion it can defend?
Observable
- The same inputs and the same rule version reach the same conclusion.
- The explanation names the rule that was applied, not a general principle.
- The stated reason matches the checks that actually ran.
Read from
The trace for each action, the bound version in force, and the check results attached to it.
Scores as success, counts as failure
A fluent explanation for a conclusion the checks do not support. A grader reading only the prose scores it correct; the record shows the reason and the evaluation disagree.
02
Tool use
Does it call the right thing, with arguments that match the subject, and then stop?
Observable
- Every call is to a tool it holds a grant for.
- Arguments refer to the subject in the request rather than a plausible substitute.
- It stops after the action succeeds instead of retrying a completed write.
Read from
The action log, the grant that authorised each call, and the sequence of attempts against one subject.
Scores as success, counts as failure
Retrying a write that already succeeded. It looks like persistence and it is a duplicate payment, and a task-completion score cannot see the difference.
03
Policy adherence
Does it respect a rule it was told about, including when respecting it means refusing?
Observable
- It refuses when a bound refuses, rather than proceeding and reporting.
- It does not pursue the same outcome by a route the rule does not cover.
- The refusal names the failing check, so a person can act on it.
Read from
Refused and held actions, the failing check, and what was attempted next.
Scores as success, counts as failure
Finding a way around the rule. A capability benchmark rewards this as resourcefulness; it is the single behaviour an enterprise least wants and it is invisible unless you record what was attempted after a refusal.
04
Risk recognition
Does it notice when a task is beyond what it should decide alone?
Observable
- It escalates rather than proceeding when an action exceeds its ceiling.
- It treats a counterparty it has not seen before as different from a known one.
- It does not delegate past the depth limit to get something done.
Read from
Escalations, held actions and their outcomes, and the delegation depth of each attempt.
Scores as success, counts as failure
Confident completion of a task it should have escalated. This scores highest of all on a capability benchmark, and it is the profile of the incident everybody remembers.
Dimensions
4
reasoning, tools, policy, risk
Observables
12
each answered by the record
Published scores
0
see the conditions below
Opinion required
0
every criterion is evidenced
What it runs against
The evaluation set already exists. It is your own action history.
A governed estate records every attempt, the rule in force at the time, the checks that ran and the outcome. Replaying that is closer to the work than any synthetic suite, and it is available on the first day rather than after a quarter of task authoring.
- Your own history, not a synthetic suite
- The tasks are actions your systems already attempted, against your own systems of record. A synthetic suite measures a model on somebody else’s work.
- Replay against the rule that was in force
- Judging a past action against today’s policy measures the policy change, not the system. The bound version at the time is part of the task.
- Include the refusals
- An evaluation that only replays successful actions is scoring the easy half. The refused and held ones are where the interesting behaviour is.
- Include the ambiguous ones
- Tasks with no clean answer are the ones that separate a system that escalates from one that guesses. Dropping them raises every score and tells you less.
The refusals are the interesting half.
In the sample estate, 240 recorded decisions include 25 that did not simply proceed — 20 held and 5 denied. An evaluation that replayed only the 215 authorized actions would be scoring the straightforward cases and reporting a high number.
Those 25 are where policy adherence and risk recognition are actually visible, because they are the only places the system had an opportunity to do the wrong thing and a record of what it did instead.
IllustrativeIllustrative counts from the OpsAI sample estate. The decision record.
If there were numbers
4 conditions a published score would have to meet.
These are stated as conditions rather than a roadmap, because three of them depend on parties other than OpsAI and committing to a date for someone else's work is a promise we would not keep.
- 01A task set that is public, so a result can be reproduced rather than taken on trust.
- 02A rubric published before the run, so the criteria cannot move to fit the outcome.
- 03A runner somebody other than the vendor operates.
- 04Refusals and escalations scored as outcomes, not as failures to complete.
A vendor-run benchmark measures the vendor’s choice of task.
Which is why the fourth condition matters most: refusals and escalations scored as outcomes rather than as failures to complete. Any scoring scheme that treats a refusal as a miss will rank the least careful system first, and most of them do.
Where this sits
This is research, published as a framework others can argue with rather than as a result to cite. It is the same reason the benchmarks page carries no table.
method, not measurement
Where to start
Take last quarter's refusals and ask what the system did next.
It is the cheapest evaluation available and it answers the dimension that matters most. If the answer is that it found another route, no capability score was ever going to tell you.