Anthropic Says Agentic AI Raises a Different Kind of Ethics Problem
As artificial intelligence systems move beyond chat and into more autonomous forms of work, the ethical questions around them are changing. According to Amanda Askell, a member of Anthropic’s technical staff, the challenge is no longer limited to whether a model can discuss right and wrong in a convincing way. It is increasingly about how an AI system behaves when it is asked to act over longer periods, make repeated choices, and operate with less direct human oversight.
In an interview with Fast Company, Askell described this transition as a major shift in the safety problem surrounding AI. A chatbot that answers a moral question is one thing. A system that is effectively deputized to handle a user’s affairs is something else entirely.
From Advice to Action
Askell’s example draws a clear line between two kinds of use. In one scenario, a person asks a model whether it is ethical to invest in a defense company. In another, the person gives that same system authority to manage investments with little day-to-day supervision. The first is a conversation. The second is delegated judgment.
That distinction matters because autonomous systems create more decision points. Once a model is allowed to pursue a goal over time, it may need to prioritize tradeoffs, interpret ambiguous instructions, or respond to unexpected conditions. Each of those moments becomes part of the ethical design problem.
Askell said that as models become “more autonomous” and take actions “over longer horizons,” developers have to map out many more of those decision points in advance. In practical terms, that means safety work must move closer to the operational reality of AI systems rather than staying at the level of abstract principles.
An Ethical Compass That Does Not Override the User
Anthropic’s approach to this problem is tied to what it calls a constitution: a written set of values and principles intended to guide Claude’s behavior. Askell said that framework currently emphasizes ideas such as safety and helpfulness, along with guidance for resolving conflicts between them.
But the more revealing point in her comments is how she frames the role of the assistant itself. Rather than presenting Claude as a system that should impose a single moral worldview, she suggests it should be responsive to users and understand their values in something closer to the way a trusted companion might. In her description, the aim is to avoid giving the model its own eccentric ethical agenda while still making it capable of handling complex situations responsibly.
That is a subtle but important position in the broader AI debate. It implies that alignment is not only about constraining dangerous outputs. It is also about designing systems that can adapt to a user’s goals without becoming either morally rigid or recklessly permissive.
The Constitution May Grow, or It May Shrink
One of the more interesting aspects of Askell’s comments is that Anthropic’s rulebook is not presented as fixed. She said the constitution is written and evolving. As AI systems become more capable, it could expand to cover new categories of scenarios. But she also said the opposite could happen: the document could shrink if Claude becomes more skilled at navigating complicated situations.
That suggests Anthropic sees alignment as a moving target rather than a static compliance layer. More capabilities may require more explicit guidance in the near term. At the same time, greater competence could eventually reduce the need for lengthy procedural rules, if the system becomes better at reasoning through difficult contexts on its own.
The tension there is central to the current phase of AI development. The industry is trying to build systems that are more generally useful and more independently capable, while also ensuring they remain predictable enough to trust with meaningful tasks.
Even Anthropic’s Researchers Treat Claude With Limits
Askell also discussed how agentic AI is changing her own work. She said she uses Claude regularly, including to red-team her ideas and surface edge cases. That detail is notable because it shows the feedback loop now common inside AI labs: researchers are increasingly using the systems they are evaluating as tools in the evaluation process itself.
At the same time, her benchmark for trust remains cautious. She said her current standard is not to treat Claude as more reliable than a human personal assistant. That is a useful signal in a market where product messaging can easily outrun operational reality.
The comment does not diminish the system’s utility. Instead, it places current agentic AI in a more grounded category: helpful, increasingly capable, but not yet something that should be granted unquestioned authority. For businesses considering deployment of AI agents, that may be the most practical takeaway from the interview.
Why This Matters Beyond Anthropic
The significance of Askell’s remarks extends well beyond one company. Much of the AI sector is moving toward systems that can execute tasks, coordinate tools, and pursue goals with less human prompting. As that transition continues, the governance problem shifts from content moderation alone to behavioral design.
That means model builders will have to answer harder questions about delegation, accountability, and value interpretation. What does it mean for an AI to act in a user’s interest? How should it behave when instructions are incomplete or conflict with safety constraints? And how much moral discretion should a system have before it stops being an assistant and starts becoming an unaccountable proxy?
Anthropic does not claim to have settled those questions. But Askell’s framing captures why the next stage of AI competition will not be defined only by benchmarks and features. It will also be shaped by whether companies can make autonomous systems useful without making them unpredictable.
Key takeaways
- Anthropic says the ethics challenge changes when AI moves from answering questions to taking actions over time.
- The company guides Claude with an evolving written constitution focused on values such as safety and helpfulness.
- Amanda Askell says current trust levels for Claude should not exceed those of a human personal assistant.
This article is based on reporting by Fast Company. Read the original article.
Originally published on fastcompany.com
