AI Simulation Intelligence Report: Potential Risks to Military Decision Making

AI simulated intelligence report: What military commanders fear most is not "no intelligence", but "seemingly perfect false intelligence"

During a red-blue confrontation exercise last year, the Blue AI intelligence system generated a "Red Mobile Brigade Assembly Coordinate Report" with a standardized format, professional wording, and even a comparison layer of satellite images. The commander ordered a precise strike, and the impact point was 47 kilometers away from the real target-the "mobile brigade" did not exist at all. It was a model that pieced together exercise photos from training data three years ago, satellite cloud images last year, and speculative posts on a military blog into an illusion of "semantic self-consistency." After the event, the system log showed a confidence of 94%. This is not complicated, but it exposes a fatal flaw: The large model optimizes the "probability distribution of the next token" rather than the "real distribution of the battlefield."*

Confidence Trap: Why 94% certainty is the most dangerous

I talked to engineers on the intelligence automation project of a Joint Staff Headquarters. The biggest headache for them was not the model's "nonsense", but the model's "serious nonsense." In the traditional intelligence process, analysts must mark the source level when writing research and judgment reports: Level A trustworthy, Level B to be checked, and Level C reference. When AI generates a report, all paragraphs are output equally, and the confidence score is just a linear mapping of the softmax probability. does not mean that "the intelligence has been cross-verified".

What's even more ridiculous is the failure mode of RAG (Retrieval Enhanced Generation) in military scenarios. What is stored in the vector database is "historical text". When the retrieved Top-k fragments are put together, they are extremely smooth semantically, but logically they may be distorted. For example, when "a certain missile has a range of 500 kilometers" and "an island is 480 kilometers away from the base" are retrieved, the model will automatically infer that "the island can be covered"-completely ignoring engineering constraints such as ballistic correction, wind direction, and warhead weight. Human analysts will ask,"What is the ballistic coefficient?" The model will only say,"It is speculated that it can be covered based on the data."

Steelman: Opponents will say,"Human analysts can make mistakes"

To be fair, the strongest argument of the opposition must be laid out: ** Human intelligence analysts can also commit confirmation bias, anchoring effects, and even fabricate the strategic illusion of "Iraq possesses weapons of mass destruction."* Before the Iraq War in 2003, CIA analysts judged the "aluminum tube" as a centrifuge component and drew the "mobile biological laboratory" as a satellite image interpretation chart. In the end, they were all wrong. Since people are unreliable, why not let AI that is 1000 times faster, is not tired, and can process terabytes of multi-source data in parallel run the process?**

This objection sounds reasonable, but misses two asymmetries:

First, Asymmetry of error costs . If an analyst writes a wrong report, he will at most rewrite it, take the blame, and transfer posts; once the AI system is deployed to the "automated target generation" process, an illusion of coordinates can trigger live ammunition strikes, civilian casualties, and strategic misjudgments. An internal test by a defense contractor in 2023 showed that under the guidance of the "high confidence illusion", the missed detection rate of manual reviewers was as high as 37%-- Because the report is too real, even the "confidential ★ ★" watermarks in the format and standardized "source reliability rating table" are complete. This is automated bias: the human brain subconsciously lowers the critical threshold when faced with "seemingly professional" output.

Second, scale asymmetry . An analyst writes 3 reports a day, and 100 analysts only make 300 reports. The AI system deployed to cutting-edge nodes can generate 3000 "customized intelligence briefings" per second and distribute them to each platoon commander's tablet. The marginal copying cost of errors approaches zero , while the verification cost remains linear-you have to send someone to check every coordinate and every layer in every report. This account is simply unfair.

Why smart people do stupid things together: There is no "illusion rate" in purchasing indicators

This matter has the theme "Why do smart people do stupid things together?" It's not that the engineers are stupid, nor that the commanders are greedy, but that the incentive structure is misaligned.

At the time of project establishment, the KPIs were "intelligence output timeliness reduced from 4 hours to 30 minutes","multi-source data fusion coverage rate of 95%" and "model F1 value on the verification set is 0.92." No one wrote the "probability of false strikes caused by illusions" into the contract because this indicator cannot be run through in the acceptance demonstration-the demonstration data set is clean, and the real battlefield is full of dirty data, network disconnection, interference, and deceptive bait. In order to win orders, manufacturers must list competing products in the "generation fluency", even at the expense of the "rejection rate." For the sake of political achievements, the command department must write "significantly improving decision-making efficiency" in the acceptance report. Everyone pushes the system to a cliff in their own rationality.

I have seen the acceptance site of a certain model of intelligent intelligence terminal: the demonstrator entered "ship dynamics in a certain sea area", and the system spat out a 15-page report in 12 seconds, and the scene applauded. I took a closer look. The "latest satellite image time" quoted in the report was three days ago, and the test server used for the demonstration was not connected to the real-time satellite downlink at all. This is playing "a machine that can write reports", not testing "intelligence that can fight wars."

Lao Chen in ### Data Center

Lao Chen is a senior image analyst in the intelligence center of a theater and has worked for 22 years. Last year, the unit deployed the "Intelligent Research and Judgment Assistance System", and the leader said "Free your hands and focus on high-value analysis."

Late one night, the system pushed a "prediction on the deployment of the enemy's new stealth drone", which was accompanied by thermal imaging comparison maps, radar cross-section estimation tables, and even engine models. Lao Chen stared at the screen for 20 minutes, his finger hanging over the "Confirm Distribution" button and did not press it.

"The infrared signature of the engine nozzle of this thing is wrong." He murmured and compared the feature database that was manually built three years ago-the parameters quoted by the system came from brain-filled posts by netizens on an open source military forum, and were retrieved by the vector library as "highly relevant text."

He returned the report with a comment: "The source has not been traced back, and the characteristic parameters are doubtful. It is recommended to review manually." At the morning shift meeting the next day, the director frowned: "Doesn't this system just ask you to read less information? You've been reviewing for half an hour. Where's your efficiency?"

Lao Chen did not speak. He found a notebook from 2015 in the drawer of his work station. The title page read: "The value of intelligence is not how beautifully it is written, but whether it dares to take responsibility."

Don't expect "people in the loop" to save lives, and make "uncertainty" the first citizen

Since "people in the loop" will fail in the face of high-confidence illusions, and RAG cannot solve the problem of "semantic self-consistency ≠ factual truth", what should we do?

I don't believe that the "larger model" can cure the problem. When GPT-5 comes out, it will still seriously write "exercise false targets" as "actual combat deployment", because there is never a "refusal to generate when uncertain" item in the training target-if the refusal rate is high, the ranking on the list will fall.

There are only three hard bones that can be implemented in engineering:

1. Mandatory traceability chain + encryption certificate . Each sentence generation conclusion must be bound to an untamperable raw data hash (satellite raw frames, radar raw IQ data, signal raw recording). If the chain is missing, it will be marked red and cannot be distributed. This is not to add a footnote to incorporate traceability into the data bus protocol and preserve traceability just like TCP maintains sequence.
2. Normal injection of confrontational red teams . The three stages of training, fine-tuning, and deployment require a dedicated red team to create a "high-confidence illusion" feeding system: falsifying satellite images, falsifying signal parameters, and falsifying open source intelligence. The indicator is not Accuracy, it is False Positive Rate at 99% Confidence-when the model is at its most confident, does it dare to admit defeat?
3. Uncertainty quantification as the first deliverable . The front end does not display "Confidence Level 94%", but displays "This conclusion relies on 3 unverified open source fragments, 1 expired satellite layer, and 0 real-time sensor direct connections. Recommended manual verification priority: P0." Making "don't know" clear has more military value than making "know" round.

Don't be kidnapped by "intelligence", war cannot tolerate smooth lies

To put it counter-intuitively: The safest military AI is not the one that can write a report like a human, but the one that can't hold back at critical moments and throws you a bunch of raw data that "insufficient evidence, please manually check".

Nowadays, the procurement logic is completely reversed: the manufacturer checks the "generated quality", the headquarters checks the "demonstration effect", and the acceptance team checks the "function list". No one pays for "strategic misjudgments caused by illusions" because the consequences are too far away, the responsibilities are too scattered, and the attribution is too difficult. By the time the war really breaks out and the missiles fly into the open space, it will be too late to pursue the "model confidence level of 94%".

Let me make a bet: In the next five years, the first military to cause a major accident in actual combat due to AI hallucination intelligence will not be the most technologically backward, but the one that most aggressively deploys the closed-loop "fully automated intelligence generation → decision-push" and lacks the culture of red team confrontation. Smart people often do stupid things because they are too good at treating "good presentation" as "easy to use in war."

War has no room for smooth lies. No matter how similar the report looks, no matter how confident it is, and no matter how standard the format is- Intelligence without a traceability chain is waste paper for intelligence disguise.