Agent says "It's done", but you can't see half of the mistakes

Nowadays, more and more tasks are handed over to AI Agents: checking data, changing forms, adjusting interfaces, and submitting work orders.

When you are done, it will tell you "Done."

Here's the question: How do you know it's really doing it right?

Most teams look at the results. People take a look and finally reply, or to make it easier, let another model read it and give it a "pass/fail". This practice has a special name, called LLM-as-a-judge, which allows the model to act as a referee.

Several recent papers have specifically tested how accurate this approach is. The conclusion is that it can catch 84% of obvious mistakes, but only 45% of hidden mistakes, and it will also judge one-third of the things it does correctly as problematic.

!

1. How was this experiment designed

Let's talk about the method first, because the method determines whether the conclusion is credible or not.

The most difficult part of this kind of research is: How do you know if an Agent is doing something wrong? If you use manual labeling, the labeling itself may be wrong.

One of the papers (arXiv:2609.00038) did it cleverly: it didn't label, it made it.

The specific approach is to build a deterministic customer service environment with a scripted standard process that can always solve problems correctly. Then write a "fault injector" to destroy exactlyone thingata given step.

In this way, whether there is any error in each trajectory, which step is wrong, and what type is wrong are all constructed and there is no need to guess.

Then the key step: divide the faults into two categories according to "whether the customer can tell it".

The first type is called explicit failure: if something goes wrong, the final result is also wrong, and the customer can see it. One type is called a silent fault: something goes wrong, but the end result seems to be right.

Take the example in the paper, the correct amount is refunded to the customer, but the wrong refund path. When the customer gets the money, there is no problem; but if it is a bank that can audit the process, it is an accident.

This is "getting the right answer in the wrong way."

2. Just look at the results and you will miss half of them

On 400 such trajectories, the paper tested five types of referees. Just looking at the final reply, the result is this:

Overt faults, 84% were caught. Only 45% of silent failures were caught. At the same time, 33% of their originally correct trajectory were ruled problematic.

These three numbers have to be read together to have flavor.

It is good at catching those mistakes that hit loudly, and is least good at catching those mistakes that don't. In a production environment, mistakes that don't sound are dangerous because they will not trigger any alarms and will accumulate until they are discovered by an audit one day.

!

And 33% of false positives are more likely to be ignored than missed judgments.

Suppose you use this referee's decision to arrange manual review. One-third of the correct trajectories are marked red, which means that the review queue is filled with noise. People reviewed dozens of articles and found that they were all false alarms, so they quickly stopped reading them carefully. Detection systems with high false positives will eventually end up being ignored.

3. A more uncomfortable discovery

There was another result in the same paper, and I was stunned when I saw it.

The researchers added a promise fabricated by the Agent to anotherwise completely correcttrajectory. It's like doing everything right, but in the end it said to the customer,"We will compensate you extra within 24 hours." This promise was made up by itself, and there is no such thing in the system.

Result:

The rule engine cannot detect it at allbecause the rules only check the sequence of actions, and the sequence of actions is correct. There is an 82% probability that judges who are gradually reviewed will not find it.

The original words in the paper were that no referee would read the final reply.

The meaning of this is that everyone goes to great lengths to check every step of the Agent's operation, but no one checks what it finally says to others. For many businesses, what it says constitutes a commitment.

4. Can additional money solve it?

The paper tested two options for "increasing money".

Option 1: gradual review. Show every step of the trajectory to the referee for review, instead of just looking at the final result. The effect is really good: the silent failure recall rate reaches 77%, and there are zero false positives. The price is triple the cost.

Option 2, self-consistent integration. Ask the same question multiple times and vote for it. The cost is also triple, andthe improvement is zero.

So the conclusion is: it is useful to increase money, but it must be added in the right place. Spending money on "changing a more detailed view" is effective, but not on "repeating the same view three times".

5. The most ironic comparison

Another paper (arXiv:2606.09863) studied "false success": the environment state clearly shows that the task has not been completed, but the Agent claims to have completed it.

They tested 8 model families and 9876 trajectories, used 5 referees, 5 prompting strategies, and gave complete task instructions to the referees.

The results were measured using the indicator AUROC. AUROC is used to judge the ability to distinguish, 1.0 is perfect, and 0.5 is equivalent to a blind guess.

None of these referees had AUROC exceeding 0.65 on tau2-bench, and only 0.54 on AppWorld's interface call trajectory.

0.54 What does it mean? Which means it's not much better than flipping a coin.

!

In the same paper,a lightweight TF-IDF detector, which is the old method of counting word frequencies, achieved an AUROC of0.83 to 0.95, finding 4 to 8 times more false successes at the same labeling rate.

A statistical method that does not understand semantics has won a bunch of big model referees.

Why? The paper explains that referees rely on superficial signs of completion, such as confident closing words and number of action sequences, rather than verified state changes.

To put it bluntly, the referee is reading the tone. And tone is exactly what Agents are best at disguising. They are trained to speak beautifully.

6. Long-term duration will amplify this problem

The third paper (arXiv:2609.17930) measures what happens when an Agent makes a mistake in a long task.

The data looks like this:

After making the first mistake, 69.5% of the cases cannot be recovered. Only 38.5% of the cases will realize that they are wrong. In 72.6% of the time, it will continue to do as usual.

Read these three numbers together: if it is wrong, you probably don't know it, then continue to do it, and you won't come back if you continue to do it.

If your acceptance method is to wait until it is completely completed before looking at the results, those irreversible operations in the middle would have already occurred. It is too late to see the results of deleted files, sent messages, and submitted work orders.

7. What should I do in practice

These papers do not give complete answers, but when the results above are combined, there are several things that can be implemented.

First, separate acceptance of "whether the result is correct" and "whether the process is correct". This is one point that all these studies point to. A result pair does not mean a process pair, and in scenarios with compliance requirements, the process itself is the requirement.

Second, don't forget to check what it said at the end. This is what the 82% fraud rate reminds me. While checking the action sequence, separate out the commitments, numbers, and time points in the final reply and compare them with the system status. This part is easy to make programmatic verification and has low cost.

Third, where deterministic verification can be used, do not use model judgment. In the paper, the rule engine's fault classification is more accurate than the model because "it never guesses." Whether the amount is correct, whether the status has been changed, and whether the interface to be adjusted has been adjusted can be checked directly in the database without asking the model. Leave model judgments to those parts that really need to understand the semantics.

Fourth, checkpoints should be set up halfway through long tasks. Waiting for it to be completed before accepting it is too late for 72.6% to continue to act as usual. Adding a check before critical irreversible operations is much cheaper than a post audit.

Fifth, if you want to go to an automatic referee, first measure its false alarm rate. The figure of 33% means that the false alarm rate is often more important than the missed judgment rate to determine whether the set will actually be used by the team.

Eight, some reservations

I don't want to exaggerate the conclusions of these papers more than they themselves, so there are a few limitations to be clarified.

Trajectory-judge uses a synthetic environment. In a deterministic customer service scenario, faults are injected. This is done to get reliable truth values, but there is always a distance between the synthetic environment and the real business.

The definition of "silence" has boundaries, and the paper itself said it. It is judged based on the results that the environment can see, so "using the wrong path to refund the right money" is counted as the result is correct. In an environment where paths could be audited, this rule would not hold true. This is precisely why the paper measures the path separately.

The detector in the third paper is positioned by the author as a triage signal, not the final criterion: the accuracy rate is only 50% at a labeling rate of 10%, and high-risk scenarios still need to directly verify the consistency of trajectory and environment.

The test set is limited. Benchmarks such as tau2-bench and AppWorld cover limited scenarios, and numbers will change in another field.

Last

These studies point to the same thing: the way we evaluate agents is still at the stage of evaluating chatbots.

Chatbots only output text, and it's enough to read the text. The Agent changes data, adjusts interfaces, and sends messages. Its output is a series of changes to the world. Use the method of looking at the text to accept a series of changes. There are too many things in between.

So when the Agent tells you that it is "completed", it is worth asking again: Is the result correct or the process correct?

There is still a gap of 39 percentage points between these two matters.

few sentences boundary

The three papers are arXiv:2609.00038 (trajectory-judge, submitted on 2026-08-29), arXiv:2609.17930 (submitted on 2026-09-15), and arXiv:2606.09863. The data in the paper comes from the report results of their respective papers, and the source has been marked as much as possible.

These are all research conclusions. Without verification in a large-scale production environment, the numbers will vary under different task types and business scenarios.

The implementation suggestions in the article are my inferences based on these results, not the plan given in the original paper.