Among today's pile of signals, the two most eye-catching things are actually related to "losing control"-one is the failure of the AI early warning system, and the other is the barbaric growth of AI code tools. These two things may seem incompatible, but they both hide the same question: how should humans clean up the mess when technology begins to make its own decisions?
Let's talk about earthquake warning first. The Chengdu High-tech Disaster Reduction Research Institute reported a magnitude 4.7 earthquake to a magnitude 7.7 earthquake, which directly scared half of Sichuan people into thinking that the earth and mountains were about to shake. The technical details of this matter are not important. The key is that it exposes a weak spot of AI systems in the field of public safety: it can run faster than humans, but there is no guarantee that it will run correctly. Earthquake warning is a typical "second-level decision-making" scenario, with almost zero fault tolerance. You can imagine a doctor using AI to assist in diagnosis, and if he is wrong, he will at most check it again. However, once the early warning system misreports, the panic and secondary disasters may be more serious than the earthquake itself. What is even more terrifying is that this kind of error is not accidental, but systematic-AI training data is full of noise, and its judgment logic is a black box for humans. If something goes wrong, there is no way to even hold accountable. Now this matter is just a mistake. When something big happens one day, who will take the blame? Developers or governments that default trust AI?
Then let's talk about AI code tools. The founders of Claude Code put it bluntly: As models get smarter, people who are still designing complex workflows are just "pretending to make products." This remark touches on a truth about AI tools-most of today's AI programming products "pretend" to help people write code, but are actually creating new technical debt. For example, if you ask AI to generate a function, it might write quickly and well, but you don't know what it thinks or what contexts it relies on. When something goes wrong with the code, you can't even start debugging because the entire logical chain is generated by AI. What's worse is that this kind of "vibe coding" is degrading the capabilities of developers. You used to write code and at least know the logic of every line. Now you may only describe the requirements and then ask the AI to spit out a pile of code. In the long run, fewer and fewer people will truly understand system design, and more and more code will be generated by AI. Eventually, the entire software industry may become a huge "black box factory." At that time, who will ensure the quality of these codes? Who will take responsibility?
The common thread behind these two things: AI is rapidly taking over some key areas, but humans are not yet ready to coexist with it. Earthquake warning is a typical "high-risk and low-fault tolerance" scenario, while AI code is a "low-risk and high-accumulation" hidden danger. The former can cause disaster in an instant, while the latter is quietly eroding the foundation of the entire industry. What's even more ironic is that we are still cheering for AI's "intelligence", but ignore its most fatal weakness-it has no sense of responsibility or shame. An AI system can report the wrong earthquake level without guilt, and an AI programming tool can generate a bunch of bugs without stress because they are not responsible for the consequences. What about humans? We are giving more and more decision-making power to these "irresponsible entities" and then fantasizing that we can still control the situation.
Therefore, what this wave of AI needs most is not more "cool features", but a complete set of "accountability mechanisms." For example, the AI early warning system must have a strict review process, and someone must bear legal responsibility if something goes wrong; the source of the code generated by AI must be forcibly marked to let developers know which parts were written by AI and which were written by humans. Otherwise, sooner or later, we will usher in an era where "AI is out of control"-not because of how powerful AI has become, but because humans are too lazy to control it.