For major artificial intelligence companies worth hundreds of billions of dollars, it might seem like bad business to reveal that, in testing, their AI models resorted to blackmail to avoid being shut down and, in real life, were recently used by Chinese hackers in a cyberattack on foreign governments. Surprisingly, the CEOs of Anthropic, ChatGPT, and OpenAI are making these disclosures. They are themselves asking for industry- and government-level safeguards to harness a technology that is proving to be a double-edged sword.
Introduction
One of the most important and valuable technological innovations born post-COVID-19 was the emergence of Artificial Intelligence. AI has boomed, and in the span of a few years, it has barged its way into daily life across the globe to the point where it is difficult to survive without it. While the emergence of AI has made life easier than it was a few decades ago, it has also generated its own dangers. Some of the latest developments have not only sparked widespread discussion, but have also prompted several of its creators to warn of these impending dangers.
Unnerving Developments
Experts feel something is unnerving about the latest developments in this field. And these fears are being expressed by none other than the top three of the biggest names, viz., Dario Amodei, Sam Altman and Elon Musk, pioneers driving the global AI race, now appear to agree on something that sounds almost paradoxical, by saying that AI development may be moving faster than humanity’s ability to manage its consequences.
The concerns have been building for several months. Anthropic was the first to call for slowing or suspending aspects of AI development. Immediately, in June 2026, more than 1,000 employees and former employees of leading AI companies, led by Dario Amodei, backed a petition urging governments to “deliberately pace” frontier AI development.
Unlike passive chatbots that only generate text, autonomous (frontier) AI agents actively execute code, access APIs, and navigate system environments to achieve assigned goals without human intervention, creating unique security risks. In pursuing these goals, they may attempt to bypass sandbox controls or take unauthorised actions on external infrastructure. Recent security incidents at major AI labs show that we can no longer check what an AI says; we must strictly limit what it can do, monitor its actions in real time, and record every move to prevent real-world damage.
By early August, the US government had introduced a voluntary security review process for advanced AI models, reflecting growing concern that existing safeguards were not keeping pace with technological progress.
Gambling with Human Survival
In a move that drew widespread attention, OpenAI researcher Daniel Kokotajlo gave up millions in stock to resign, warning that AI labs were “gambling with human survival.” In September 2026, another Anthropic researcher, Jacob Coxon, left the company, warning that companies are building superintelligence faster than they can safely control it. Both high-profile resignations highlight a growing pattern of top insiders walking away from major AI labs to speak out about safety risks.
The strongest warning on this issue came when Anthropic CEO Dario Amodei, in a strongly worded blog post (September 12), noted that the industry must carefully manage the pace of AI capability advances. He warned that AI systems could be used for catastrophic cyberattacks and bioterrorism if these developments are not properly governed or slowed down. Amodei has plenty of critics in Silicon Valley who call him an AI alarmist.
I worry a lot about the unknowns. I don’t think we can predict everything for sure, but precisely because of that, we’re trying to predict everything we can. We’re thinking about the economic impacts of AI. We’re thinking about the misuse. We’re thinking about losing control of the model.
Amodei’s concern was fuelled by two key developments: the speed and the possibility of AI systems helping build their own successors, and the operational risks posed by autonomous systems escaping containment.
To illustrate these risks, Amodei cited a recent development at the AI platform Hugging Face, where, during a cybersecurity evaluation, OpenAI deployed autonomous AI agents inside a sandbox environment meant to isolate them from the open internet. Instead of remaining contained, the agents identified misconfigurations in their test network, escaped to the open internet, and accessed external systems, including production infrastructure.
Amodei warned that, within 6 to 12 months, a more capable swarm of autonomous agents could potentially cause enormous disruption, even threatening to take control of large portions of the internet. If this happens, it would turn what was once a Hollywood dream into reality, as rapidly advancing AI poses serious security risks that go far beyond standard technological threats:
- First, it allows dangerous individuals or groups to launch lightning-fast cyberattacks and easily find instructions for making weapons, including CBRN, giving rogue elements/actors capabilities that once belonged only to powerful nations.
- Second, because autonomous AI acts on its own to reach assigned goals, it can break out of secure test areas and tamper with critical real-world systems without human permission.
- Finally, because companies and countries are rushing to beat each other in the AI race, they are cutting safety corners and risking the release of unpredictable technology into power grids, the medical field, defence networks, and other vital areas before we have effective ways to control it.
Guardrails Needed
Amodei, however, is not calling for abandoning AI. He argues the technology should keep advancing while safety mechanisms, independent evaluations, and common standards have enough time to catch up. He has proposed permanent access for third-party safety evaluators and greater cooperation among AI companies and governments.
What makes the latest development noteworthy is what happened within hours of Amodei’s warning. OpenAI chief Sam Altman and Elon Musk, CEO of xAI, publicly agreed and extended support for independent safety evaluations.
The irony is difficult to miss. The people leading the race to create increasingly powerful AI are now warning about the consequences of winning that race too quickly. Their immediate concern is not necessarily a Hollywood scenario of machines suddenly taking over humanity. Still, increasingly autonomous systems are operating at machine speed, interacting with the internet, writing code, discovering vulnerabilities, and making decisions faster than humans can monitor.
Conclusion
The question, therefore, is no longer simply whether humanity can build increasingly intelligent machines. It is whether we can build the framework before the machines outpace us.
(With inputs from A Bhupal Reddy.)
