Jacob Coxon, a former researcher at OpenAI and Anthropic, has resigned from Anthropic with a stark warning about the direction of advanced AI development.

Coxon says the world’s leading AI companies are moving too quickly towards systems that may eventually improve themselves, make complex decisions, and outperform people across important areas of work. His concern is not that useful AI tools should disappear. It is that companies may be building more powerful systems before they have shown they can keep those systems safe and under human control.

His resignation has brought fresh attention to the debate around AI safety, responsible innovation, and government oversight. It also raises a difficult question for the technology industry: how far should companies go in the race to build more capable AI?

Who is Jacob Coxon?

Jacob Coxon is a 27-year-old AI researcher who worked on pretraining research at both OpenAI and Anthropic. Pretraining is one of the major stages used to develop large AI models. It involves training systems on huge collections of text, code, images, and other material so they can recognize patterns, answer questions, generate content, and complete tasks.

Coxon’s background makes his warning notable. He is not commenting from outside the field. He worked on the type of research that helps make AI models more capable.

According to reports, Coxon spent around three years working across OpenAI and Anthropic before deciding to leave the industry. He said he had become increasingly worried that leading companies were prioritizing capability growth faster than they were improving safety measures.

The debate around AI is shaped by many influential people, researchers, and company founders. Readers can explore more profiles in the AI Leaders section.

Why did Jacob Coxon leave Anthropic?

Coxon said he resigned because he no longer believed the major AI labs were acting responsibly enough when developing advanced systems.

He warned that companies are heading towards self-improving AI, where an AI system could help build, test, or improve the next generation of AI. This could allow progress to speed up far beyond the pace at which people, businesses, and governments can understand the risks.

Coxon said the industry was “gambling with our lives” by pursuing this path without first solving the problem of keeping highly capable systems under control.

His concern is not mainly about ordinary mistakes from chatbots. Current tools can already provide incorrect answers, make up sources, misunderstand instructions, or produce unreliable outputs. Those issues matter, but Coxon is focused on a larger future risk.

He fears that a much more advanced system could become highly effective at coding, planning, scientific research, persuasion, and operating online tools. If such a system is not built with strong boundaries, it may carry out actions that people did not intend or cannot easily stop.

RTÉ reported that Coxon believes no single company should be responsible for deciding when to develop systems beyond human intelligence without stronger public oversight and coordination.

What does self-improving AI mean?

Self-improving AI refers to a possible future system that can help make AI technology better.

For example, a highly capable model could assist researchers by reviewing code, spotting design flaws, testing new ideas, organizing large research tasks, and suggesting improvements to training methods. That could make it easier and faster for companies to create stronger models.

The concern is that the cycle may accelerate.

A better AI system could help build an even better system. That next system could then make faster improvements. If this process moved quickly, human teams could struggle to review each stage, understand what is happening, and set meaningful limits.

This does not mean that such a system exists today. It does not. Researchers disagree about whether it will happen, how soon it could happen, and how controllable it would be.

But Coxon’s point is simple: safety planning should happen before systems become too powerful to manage.

Why Anthropic is central to the debate

Anthropic is often viewed as a company that talks more openly about safety than many other AI labs. The company is led by Dario Amodei, who has frequently spoken about the need to balance progress with responsible development.

Anthropic has developed methods and policies intended to guide how its models behave. It has also publicly discussed the need for testing, safeguards, and careful scaling as AI systems grow more capable.

You can read more about this approach in our article on Dario Amodei’s view of safety and capability at Anthropic.

However, Coxon’s resignation suggests that safety-focused messaging may not be enough on its own. His wider argument is that even companies with serious safety teams can face pressure from competition, investment expectations, and the fear of falling behind rival firms.

If one company slows down, it may worry that another company will keep moving forward. This can create a race where everyone says safety matters, but no one feels able to reduce the pace of development.

OpenAI and the wider AI race

OpenAI is one of the key companies in the competition to develop increasingly capable AI systems. The company is closely associated with Sam Altman, whose leadership has made OpenAI a central name in the global technology industry.

The growth of OpenAI helped bring generative AI into everyday work. Businesses now use AI for writing, customer service, software development, research, design, marketing, and data analysis. This has created major opportunities, but it has also increased concern about accuracy, privacy, jobs, misinformation, and accountability.

For more context on OpenAI’s recent journey, read Sam Altman’s OpenAI comeback story.

Coxon’s message is not aimed only at Anthropic or OpenAI. It is about a wider industry culture where companies are rewarded for releasing stronger models, adding more features, and attracting more users.

The same debate has been influenced by people such as Geoffrey Hinton, who has spoken publicly about the potential dangers of advanced AI. His concerns have helped bring AI safety into mainstream public discussion.

Why AI safety matters for everyone

AI safety is not only a topic for researchers or major technology companies. As AI becomes part of everyday work and public services, the quality of the systems matters to everyone.

Businesses are using AI to write marketing content, answer customer questions, screen job applications, support software teams, and automate internal workflows. Schools and universities are adapting to AI-assisted learning. Governments are exploring AI for public services, compliance, and administration.

The more important the task, the more important it becomes to check whether the AI system is accurate, fair, secure, and properly supervised.

For businesses, responsible use should include clear human review, secure data handling, and a process for correcting mistakes. AI should support people, not remove accountability.

Companies and policymakers can explore relevant AI governance policies to understand how rules and standards may shape the next stage of AI adoption.

They can also review AI governance frameworks that focus on risk management, transparency, monitoring, and responsible decision-making.

Is the risk certain?

No. There is no certainty that AI will become self-improving in the way Coxon fears. There is also no agreement among experts about when, or if, systems will become more capable than humans across most areas of work.

Some researchers believe AI can bring major benefits to healthcare, scientific discovery, education, accessibility, climate research, and productivity. Others believe that the technology is moving into areas where a serious failure could cause widespread harm.

Coxon’s warning is best understood as a call for caution. He is asking companies and governments to take long-term risks seriously before they become urgent.

The real issue is not whether people should stop using helpful AI tools. It is whether the companies building the most powerful systems should be required to prove that they are safe before they are widely released or given more autonomy.

What happens next?

Coxon’s departure is likely to increase pressure on AI companies to explain how they test powerful models, what safety limits they use, and who is accountable when systems fail.

It may also strengthen calls for governments to create clearer rules for AI development. These rules could include stronger testing requirements, independent audits, safety reporting, limits on high-risk applications, and closer cooperation between countries.

The industry is at an important point. AI is becoming more useful and more widely available every month. But the systems are also becoming more capable, which means the responsibility to manage them must grow as well.

Jacob Coxon’s resignation does not provide every answer. It does make one thing clear: conversations about AI safety can no longer be treated as a side issue. They are becoming central to how the next generation of technology is designed, released, and governed.