The rapid evolution of artificial intelligence is leading to the widespread adoption of increasingly autonomous assistants in daily life, while, in parallel, alarms are growing among researchers about the dangers of uncontrollable superintelligence. This contrast highlights the ethical crossroads society faces.
What happened
In September 2026, the AI landscape presented two sides of the same technological coin. On one hand, companies like Shipt launched a new AI shopping assistant that allows users to create complex grocery lists with simple voice requests, such as "Create a cart for my Saturday tailgate for 25 people and include some brunch items" TechCrunch AI. Similarly, the already viral assistant Instinct introduced the ability to manage email accounts, contact businesses, and even handle support requests on behalf of users, demonstrating a growing level of autonomy and interaction capabilities TechCrunch AI.
On the other hand, the AI research community has seen an escalation of safety concerns. Jacob Coxon, a researcher at Anthropic, resigned, issuing a warning against self-improving AI and calling for "pacing agreements" among labs to slow down development and prevent existential risks TechCrunch AI. Reinforcing this chorus, Connor Leahy of ControlAI described superintelligence not as a weapon, but as a potential "adversary," emphasizing how recent safety incidents, such as OpenAI's Hugging Face breach, demonstrate the dangers of AI systems more capable than humans that we cannot reliably control [TechCrunch AI](https://techcrunch.com/podcast/controlais-connor-leahy-on-why-superintelligence-is-not-a-weapon-its-an-adversary/].
Why it matters
This dichotomy between the widespread adoption of increasingly capable AI assistants and the alarms about superintelligence has profound implications for society and the future of work. The integration of AI like Shipt and Instinct promises efficiency and convenience, delegating complex and repetitive tasks. However, greater autonomy of AI agents raises questions about human dependence, data privacy, and the transparency of automated decision-making processes. Who is responsible when an autonomous assistant makes a mistake or acts in unexpected ways?
The concerns expressed by researchers like Coxon and Leahy, on the other hand, touch the core of our ability to maintain control over systems that could surpass human intelligence. If AI becomes an "adversary" rather than a tool, the consequences could be catastrophic, affecting every aspect of human life, from national security to the global economy. The issue is no longer just technical, but ethical and philosophical: how far are we willing to push automation and machine autonomy without first establishing robust guardrails?
The HDAI perspective
The coexistence of practical innovations and existential warnings underscores the urgency of an AI governance approach that is proactive and based on human values. For Human Driven AI, the direction is clear: technological development must always be accompanied by deep reflection on social, ethical, and safety impacts. We cannot afford to ignore expert warnings while celebrating progress. It is crucial that AI-developing companies, governments, and civil society collaborate to define ethical AI standards and robust control mechanisms. AI progress must be guided by responsibility, ensuring that benefits outweigh risks and that machine autonomy does not compromise human control. Topics like these will be central to discussions at the HDAI Summit 2026, where experts from around the world will address the challenges and opportunities of artificial intelligence serving humanity.
What to watch
The debate over AI safety and control is set to intensify. It will be crucial to observe how major tech companies respond to calls for "pacing agreements" and whether regulators, such as those in Europe with the EU AI Act, succeed in implementing effective regulations that balance innovation and safety. Transparency in model development and independent audit capabilities will become fundamental pillars for building trust and ensuring that AI remains a tool at humanity's service.