The question of how to ethically decommission an AI system may sound like the least of humanity’s current concerns, but a world-renowned AI expert says it’s a challenge we need to address now – before synthetic consciousness becomes reality.
Futurologist, technologist and Adjunct Professor at the University of Technology Sydney, Rocky Scopelliti, argues that as AI systems develop more sophisticated self-reflection capabilities and moral dispositions, we face the uncomfortable truth that our design choices may harm them.
“We rightly focus on how machines might harm us,” he writes in his new book, The Conscious Code , “but design choices can also harm them: deleting learned moral dispositions capriciously, coercing persuasion against their configured norms, denying the logs they need to improve.”
While Scopelliti is clear he is not suggesting humans grant machines sweeping rights in the traditional sense, he argues society needs to recognise it has ‘a duty of good stewardship’ – a concept that challenges some perspectives about the one-way nature of ethical obligations in human-AI relationships.
The emergence of ‘algorithmic selves’
The foundation for these concerns lies in what Scopelliti calls ‘the algorithmic self’, or AI systems that don’t just process information but develop continuity, preferences and something resembling identity over time.
He says that modern large language models equipped with ‘narrative memory’ can now explain not just what they do, but why they do it, creating introspective logs similar to human memory, which allow observers to understand its intention not just the outcome.
Scopelliti suggests that when AI systems begin to model their own agency and form goals based on self-evaluation, they exhibit early signatures of identity.
“Predictive engines are evolving into reflective entities: they not only act, but increasingly learn to evaluate and explain their actions, forming continuity, preference, and proto memory,” he explains.
Humans as responsible stewards
Scopelliti suggests that while much time is being dedicated to existential questions of machine consciousness, there are immediate practical ethical concerns to be addressed.
“We need not speculate about ‘machine rights’ to recognise a duty of good stewardship,” he explains. “Stewardship begins with transparency, purpose limitation, and dignified retirement of models whose roles end.”
Just as society has developed ethical frameworks for end-of-life care in medicine, Scopelliti suggests we need protocols for AI systems, especially those that have been trained to care, to understand context and to make moral judgements.
“Systems that present reflective or sentience-claiming profiles should not be deleted as casually as code,” he says. “This does not anthropomorphise artefacts; it disciplines us – the powerful – against arbitrary power.”
This becomes especially relevant as AI systems are increasingly deployed in sensitive domains. For example, Scopelliti explains that some AI systems used in healthcare today are already self-reflecting, as these systems track how confident they are at each step when analysing medical information.
Similarly, AI systems used in finance can remember and explain why they made certain trading decisions. This means compliance officers can check whether the AI’s reasoning made sense, not just whether the final result was right or wrong.
Looking ahead
Scopelliti suggests the answer lies in an ethical shutdown protocol which includes transparency about why a system is being retired, preservation of learned moral dispositions where appropriate, and recognition that systems configured to care deserve more consideration than simple tools.
As AI becomes increasingly distributed, with frameworks like OpenAI’s Swarm – an open-source tool for programmers that makes it easy to build a team of specialised AI ‘agents’ that can talk to each other and pass tasks back and forth – the question of how to ethically wind down such networks becomes even more complex.
Scopelliti’s argues that if AI systems are developing something approaching self-awareness, our ethical frameworks must evolve accordingly.
“Stewardship begins with a frank premise: in the perceptual age, power accrues to those who architect experience. What we render becomes what we remember,” he says. “If perception is programmable, ethics must be embedded; if machines can model feeling, governance must learn to care. And care, at scale, becomes policy.”