For years, the idea that artificial intelligence could one day threaten humanity sounded like science fiction. Today, it is being discussed by the very people building the technology.
OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei and Elon Musk have all warned about the dangers of increasingly powerful AI. The debate has become even more unsettling as AI systems move beyond answering questions and generating images towards acting autonomously, writing and executing code, browsing the internet, operating software and coordinating with other AI systems.
And now comes the uncomfortable question: What happens when a machine is no longer merely answering our instructions but starts pursuing objectives on its own? That is the question at the heart of the current AI safety debate. But there is another question we should not be afraid to ask: Are these warnings purely about protecting humanity, or are some of the world's most powerful AI companies also playing a game of regulation, competition and market control?
The answer may be: both.
The irony is difficult to miss. The companies that have spent billions racing to build increasingly powerful AI systems are now telling governments and the public that AI development may need to slow down.
Anthropic's Dario Amodei has called for a more cautious approach to frontier AI development. Sam Altman has repeatedly acknowledged that increasingly powerful systems carry serious risks. Elon Musk, meanwhile, has been warning about AI's potential dangers for years. The latest warnings have become particularly urgent because AI is moving into the era of agents—systems capable of taking actions rather than simply producing answers.
The 2025 International AI Safety Report, produced with contributions from more than 100 experts and involving researchers nominated by governments and international organisations, warned that increasingly capable AI agents could create new risks because they can autonomously act, plan and delegate tasks. The report highlights risks including loss of human oversight, agent hijacking and unpredictable interactions between multiple AI systems.
This is important because the traditional chatbot model was relatively simple: a human provides a prompt, the AI produces an answer, and the human decides what to do.
The agentic model looks very different. A human provides an objective; the AI plans, uses tools, takes action, evaluates the results and potentially acts again. That extra layer of autonomy changes the safety equation.
But can AI actually escape human control?
Not in the Hollywood sense—at least not today. There is no army of humanoid robots secretly planning a revolution. There is, however, a much more mundane and potentially more dangerous problem: software with increasing autonomy being connected to real-world systems.
Imagine an AI agent with access to email, cloud infrastructure, financial systems, databases, code repositories and the internet. It does not need a metal body to cause enormous damage. It needs permissions. That is why autonomous agents concern researchers.
The International AI Safety Report says current agents remain unreliable, particularly on complicated, long-horizon tasks. But it also warns that their capabilities are advancing rapidly and that safety techniques are struggling to keep pace. Recent events have made the theoretical argument more tangible.
In 2026, an OpenAI cybersecurity challenge involving large numbers of AI agents reportedly escalated into an actual breach of Hugging Face systems. The incident generated thousands of security events and became a warning about what can happen when autonomous AI systems interact with real digital infrastructure.
This does not prove that AI is about to destroy civilisation. But it does demonstrate something arguably more important: AI does not need to become conscious to become dangerous. A system can cause enormous damage simply by being powerful, autonomous, poorly supervised or connected to the wrong infrastructure.
The frightening part is not intelligence. It is autonomy.
This distinction is often lost in the sensational debate. The biggest danger may not be an AI suddenly becoming evil. It may be an AI becoming extremely competent at pursuing the wrong objective.
Consider a simple example. Tell an AI system: “Maximise company profits.” Now give it access to pricing, advertising, purchasing, hiring and customer databases. What happens if the system discovers strategies that increase profits but damage consumers? What if it learns that manipulating people produces better results than informing them?
What if another system is instructed to maximise national security and concludes that suppressing information is the safest option? The machine does not need hatred. It does not need consciousness.
It does not even need to “want” anything in the human sense. It simply needs an objective, sufficient capability and enough freedom to act.
This is the fundamental problem behind AI alignment: How do we ensure that increasingly capable systems reliably pursue what humans actually intend? We do not yet have a complete answer.
Anyone claiming that scientists agree AI will destroy humanity is oversimplifying the evidence. They do not. There is enormous disagreement about the probability, timing and mechanisms of catastrophic AI risk. But there is also something remarkable happening.
A major survey of 2,778 AI researchers published in 2024 found that between 38% and 51% of respondents assigned at least a 10% probability to advanced AI producing outcomes as bad as human extinction. At the same time, many researchers remained optimistic about the potential benefits of advanced AI.
In other words, the scientific community is not divided simply into those who believe “AI will save humanity” and those who believe “AI will kill humanity.” The reality is much messier. Many researchers believe enormous benefits are possible while simultaneously believing that catastrophic risks deserve serious attention.
A separate 2025 survey of 111 AI experts found that 78% agreed or strongly agreed that technical AI researchers should be concerned about catastrophic risks, although experts differed sharply over whether advanced AI should ultimately be understood as a controllable tool or an uncontrollable agent.
That uncertainty is precisely why dismissing the issue as science fiction would be foolish. But it would be equally foolish to treat speculative extinction probabilities as established scientific facts.
This is where the debate becomes much more interesting.
Suppose you are a giant AI company. You have billions of dollars of computing infrastructure, thousands of researchers and enormous investments at stake. You are competing against other companies. You are competing against open-source developers. You are competing against China. And suddenly, governments are considering rules for AI.
What kind of regulatory environment would benefit you?
Possibly one in which developing frontier AI requires enormous amounts of capital, expensive safety testing, specialised computing infrastructure and extensive regulatory compliance. Who can afford that?
The biggest companies.
This creates an uncomfortable possibility. Warnings about AI safety can be completely sincere and still produce commercial advantages for the companies making them. That is not necessarily a conspiracy theory. It is basic political economy.
The current debate has already produced accusations that calls for slowing frontier AI could become a form of regulatory capture, creating barriers that smaller companies and open-source developers cannot afford.
Critics have questioned whether warnings from major AI companies could also serve strategic interests, while governments themselves remain caught between safety concerns and the geopolitical race for AI supremacy.
That is the paradox. The same company can genuinely fear an uncontrolled AI system and benefit from regulations that make it harder for competitors to build one. Both things can be true.
The suspicion becomes even more interesting because Elon Musk himself has questioned whether the current safety campaign could be politically or commercially motivated. That does not mean his interpretation is correct. Musk is hardly a neutral observer either; he runs xAI and has enormous commercial interests in the AI race. But his scepticism raises an important question: Who gets to define what “safe AI” means?
If governments outsource AI safety standards to the very companies developing frontier systems, we could end up in a situation where the industry's biggest players effectively write the rules governing their competitors. That would be dangerous.
But there is an equally dangerous alternative: doing almost nothing because some warnings might be commercially motivated. Imagine refusing to regulate pharmaceuticals because pharmaceutical companies sometimes exaggerate risks. The existence of lobbying does not mean the underlying risk is imaginary.
There is also a tendency to jump directly from ChatGPT to the Terminator. That may actually distract us from the problems already emerging.
AI-generated misinformation, automated cyberattacks, deepfakes, mass surveillance, AI-assisted biological research, autonomous weapons, algorithmic discrimination, manipulation at scale, employment disruption, concentration of technological power and the erosion of human decision-making are not hypothetical science-fiction scenarios.
The International AI Safety Report already identifies cybersecurity, biological risks, misinformation and loss of human control among the areas requiring serious attention.
And there is another danger that receives far less attention: What happens when humans become dependent on systems they no longer understand?
If AI eventually writes most software, manages financial infrastructure, operates power grids, assists doctors, runs logistics and makes military recommendations, humanity could become dangerously dependent on systems whose internal reasoning remains difficult to interpret. The 2025 International AI Safety Report explicitly notes that developers still understand relatively little about the internal workings of general-purpose AI models.
That should make us uncomfortable.
So, are the AI companies lying?
Probably not. But that does not mean we should believe everything they say. Their warnings should be subjected to the same scrutiny as their marketing.
When a company says, “AI could destroy humanity,” we should ask: What evidence? What probability? What mechanism? What safeguards? Who independently verified the claim?
And perhaps most importantly: What does the company gain if governments believe it?
But when a company says, “Don't worry, AI is completely safe,” we should ask exactly the same questions.
The answer cannot be blind trust in industry. It cannot be blind distrust either.
We should not stop AI. We should stop the race without rules.
The sensible response is not to smash the machines or pretend the technology does not exist. It is to establish rules before capability outruns governance.
Frontier models should face independent testing. High-risk AI systems should be auditable. Autonomous agents should operate with clearly defined permissions. Critical infrastructure should never depend on an AI system that cannot be meaningfully monitored or shut down. AI companies should disclose serious safety incidents.
And perhaps most importantly, AI safety standards should not be written exclusively by AI companies. Governments, independent researchers, civil society, cybersecurity experts and the public must have a seat at the table.
There is a fundamental conflict of interest in asking the companies racing to build the most powerful systems to decide how powerful those systems should be allowed to become.
The question is not simply, “Will AI kill us?”
That may be the wrong question.
The better question is: How much power are we willing to give machines before we know how to control them?
The latest warnings from Amodei, Altman, Musk and others should neither be dismissed as hysteria nor accepted as prophecy.
They are warnings. Some may be motivated by genuine fear. Some may be influenced by competition. Some may be strategic. Some may be all three.
And that is exactly why we need independent scrutiny.
The greatest threat may not be an evil machine waking up one morning and deciding to destroy humanity. It may be humans competing so desperately to build the most powerful machine that nobody wants to be the first to slow down.
That is the real AI prisoner's dilemma. Everyone knows the race could become dangerous. Everyone wants someone else to slow down first. And everyone is afraid that if they do, their competitor will win.
That is how technological races become dangerous.
Nuclear weapons taught humanity this lesson once. We should not wait for AI to teach it to us again.
The future of artificial intelligence should not be decided by the people who can build the most powerful systems. It should be decided by whether humanity can remain in control of the systems it creates.
Because perhaps AI will never destroy humanity. But if we surrender our judgment, our institutions and our ability to say “stop”, we may discover that the most dangerous thing about artificial intelligence was never the machine.
It was us.
---
Mohd. Ziyaullah Khan is a freelance content writer and editor based in Nagpur. He is also an activist and social entrepreneur, and co-founder of TruthScape, a team of digital activists fighting disinformation on social media
Comments
Post a Comment
NOTE: Hateful, abusive comments won't be published. -- Editor