Thank you for a stimulating essay. Geoff raises a pertinent question, which I'd like to address directly. I think there could be three practical tests for when moralization — refusal to negotiate — is justified.
First, the harm must be inherent to the practice itself, not dependent on facts that could change. Your diagnostic ("if the harms were fixed, would you change your mind?") serves as a strong philosophical test. With genocide, the objection remains regardless of any hypothetical fix since the harm — the denial of persons' standing as persons — is inseparable from the act.
Second, refusal is warranted when bargaining would itself ratify the wrong. The Missouri Compromise didn't temper slavery; it treated the freedom of African Americans as something negotiable.
Third, moralization is justified when the threat is to the deliberative process itself rather than to a particular outcome within it. Opposing fascism is a stance against a system that would end deliberation altogether.
Most of the AI applications you surveyed meet none of these criteria. An essay-grading tool presents harms that are contingent and fixable, so moralizing it reflects the miscalibration your data captures. But "AI" is a portfolio, not a single moral object. Most holdings are ordinary; a few — autonomous weapons, surveillance systems — at least approach the third test.
The real challenge is distinguishing the applications that call for cost-benefit analysis from the rare cases that warrant entrenchment. So maybe the moralizers' error isn't conviction but collapsing the portfolio into one thing so it can be condemned whole.
Thanks for your comment! I haven’t thought about it in those terms before but that makes a lot of sense.
We surveyed participants on 11 different AI applications and none of them were applications that could be considered inherently wrong (like autonomous weapons). It makes sense then to describe the moralization as some sort of conflation error.
Thank you — and "conflation error" is the succinct name for it. Your point suggests a natural extension: including a benchmark application where moralization is arguably warranted, like autonomous weapons. If opponents of autonomous weapons gave the same survey answers as opponents of grading tools — and I suspect they would, if not even stronger — that would suggest the instrument detects that people refuse to bargain, but not whether they should. What makes the conflation an error would be invisible in the data itself.
One further thought, returning to where your essay began. The encyclical demonstrates that moral seriousness and differentiation can coexist: Leo declines to treat the technology as inherently evil, insists it is never neutral, and reserves his hardest language — a call for "disarmament" — for autonomous weapons, the application that may genuinely warrant it. Intense conviction, no conflation. Which suggests the cure for the conflation error may be not less moral seriousness, but more of it, better organized. And if you ever decided to extend this and some answers diverged, you could locate and study the differentiators — the serious critics you described as accurate enough that companies and policymakers can't brush them off.
This is maybe too minor a point to be meaningful addition or critique, but there can be progress made in persuasion on many, if not most, moral claims, it’s just much harder and rarer. It’s especially rare at the macro society-level and more common on micro or meso levels. So it does give me some hope that as AI is moralized it’s not a total impasse. I could be wrong, though!
This made me wonder whether one of the downstream consequences of moralization is that it changes not only how we debate an issue, but also how we evaluate suffering itself. Once something becomes moralized, the associated harms are no longer judged solely by their severity, but also by whether society considers them a morally acceptable reason to suffer. That seems like a mechanism that extends beyond AI to medicine, disability, and appearance, where moral judgments often shape whose suffering is recognized as legitimate and whose is dismissed. It makes me wonder whether moralization doesn’t just polarize conversations, but also quietly determines which forms of suffering society feels obligated to address.
Nice summary, and the moralization framework explains something I see play out in real time. One thing I'd add: moralization maybe a response to a prior failure — the failure of the folks building LLMs to offer a table worth sitting at.
When the Pope raised his AI concerns at the altitude it deserved, the loudest responses compressed it back to a frame the commentariat could handle: one VC warned that government was the real danger, another commentator dismissed the Vatican as provincial, an economist read it as a status play. None engaged with the actual question. If you raise something at moral altitude and it keeps getting received as a smaller matter, moralization starts to look less like irrationality and more like the only register left that can't be compressed.
Which doesn't make the polarization less dangerous — your point about the slide toward acceptance of violence is sobering and correct. But it suggests the cure isn't only asking critics to de-moralize. It's building forums where the question can be received at the altitude it's asked. Chris Olah sat in the Vatican and disagreed with the Pope on machine intelligence publicly, respectfully, without anyone walking away. Now, that was a table worth sharing. We have too few of them.
This note is for Victoria, thank you for the summary of your research. You are breaking research ground and thinking about important topics. I will keep reading.
I confess to having some moralized thoughts/reactions when I see that only 57% ‘do not oppose’ self-driving cars. And I shouldn’t be surprised, given that our results are so similar… https://osf.io/preprints/socarxiv/e7mj3_v1
Interesting essay. Isn't it more or less a tautology, though, to say that "the fiercest critics" of AI or any other phenomena are "opting out" of a conversation about that topic with others who disagree? I'm trying to think of another contested topic in which the "fiercest critics" are sitting down calmly and rationally and debating with those who hold a fundamentally different position. I can't think of one at the moment, maybe I just lack awareness.
Also, I may be living in a bubble, but it seems to me that most people understand, at some level, that morality involves trade offs: one can hold a strong moral position on an issue, while also understanding that pressing that issue by deception or force is likely to have net negative consequences.
Perhaps the best example is war: A person can believe strongly that an enemy is genuinely perpetrating moral evil and represents a threat to all that is good and right, and still not advocate going to war against that enemy. Conversely, a person can recognize the unspeakable evil and suffering that is let loose by war, and still believe that initiating a war under certain circumstances is just. This despite the Pope's rather naive statement that just war theory is outdated.
The reality is, I think, that most people are fully capable of holding strong moral positions but understanding that in the real world, there a difficult calculations to be made as to what course of action to pursue . In fact, I think the vast majority of people understand this and act accordingly, even if they may not be able to articulate the process like that.
When considering AI, it seems clear that there are multiple moral principles that need to be considered. The "fiercest critics" are not the only people committed to moral principles.
Thanks for your comment! I’ll admit the title was a bit of a rhetorical trick.
That said, there’s something in the paper worth noting that I didn’t get into in the essay: most AI moral opponents couldn’t actually articulate that their opposition was moral in nature. They framed their concerns in utilitarian terms (harm to people, harm to society) which creates an interesting mismatch. They engage as if trade-offs and risk-benefit arguments might move them, when in reality their judgments seem to rest on a perception that AI is fundamentally wrong. No amount of risk mitigation would actually change their minds.
I didn’t measure how amenable opponents were to tradeoffs by presenting them with arguments in favor of AI, that would perhaps be the most effective test of this hypothesis. But there’s literature on this. While there are some moralizers who do change their opinions, others remain totally immune to argumentation.
E.g. Inbar, Y., & Waldhof, G. (2023). Mitigating consequence insensitivity for genetically engineered crops. Journal of Experimental Psychology: Applied, 29(3), 584.
Thanks for the erply, Victoria. Thinking about the AI issue, I have been focused largely on arguments about AI data centers, and as you say, there are a lot of people who seem to believe those data centers are simply immoral in themselves. It's of interest to me because we have one going up near our town, and I'm on the "pro" side, and most are, partly because construction began three years ago, before the intense wave of anti-AI sentiment had crested (if it has crested).
In thinking about the conflict, though, I think the problem is that the extreme anti-AI position is not determined by moral or moralistic thinking. Instead the sense of morality is being steered by apocalyptic thinking: opponents are claiming, if we take their Facebook posts at face value, that to allow a data center in a community will not just damage the community but will actually destroy it, or at least destroy everything good about it. To me, this is apocalypticism.
There is, I'm sure we agree, a rich vein of moralism running through American social and political thought; but there is also a deep apocalyptic stream fueled by a complicated mix of religious and environmental concerns. I've lived most of my life with people strongly influenced by both. I respect the fundamental moral decency of these folks, andd I think they're fully capable of good moral judgments. However I have very little respect for the apocalyptic visions that often seem to cast a spell over them. So to me, the problem isn't applying moral principles to an issue, it's adopting what I think is a fundamentally apocalyptic vision, even while they may not realize that they are doing so.
For a sample of what I've been keeping an eye on lately, see this Facebook group (may be a private group, but worth joining if you're intrigued by what I've said).
Really well written piece! But one could come away with the impression that you think nothing should be moralized because of the costs you outline (which would, ironically, be moralizing the issue). Do you think it's always wrong to moralize? If not, when is moralizing appropriate?
I'm not advocating for the unmoralization of AI, but for the full engagement of the moralizers. Politics (another realm with features of moralization) offers a useful parallel: some of the most ideologically extreme people tend to have the most political knowledge, and they're more likely to correct misinformation from the other side on social media, for instance. So the question isn't which issues should be moralized, but how to channel that moralization productively.
I think a bit of the opposite. AI companies have plenty of resources and people lobbying for them; the moralizers have little power. And it's easy for companies or policymakers to brush off the moralizers' concerns when those concerns are inaccurate or unfounded. The bar for being taken seriously is at least being accurate in the criticism. So build norms and systems that empower serious criticism
Hopefully Victoria will chime in, but I'll respond with my own views. I've come to believe we should moralize as little as possible. But if we do moralize, I think our moralization should be consequential in nature, responsive to evidence and data. What we assessed here was a form of deontological moralization that is consequence insensitive--people reported not being willing to change their minds no matter how much good or how little bad any specific AI application caused. While I think that deontology too is needed, my view is that deontology can easily lead one astray (for example, deontology does not allow a refutation held by the majority of many populations that homosexuality is a moral abomination).
It's worth pointing out that the two biggest AI-only firms (OpenAI and Anthropic) have a moral dimension to their "origin stories" - both claim to have been founded out of a desire to mitigate AI risks.
It's easy to fall in love with the latest shiny rock. We will look back and wonder why we missed the greatest event which was the Monk's knowledge Renaissance that is going to change absolutely every aspect of everything in our lives and make AI look insignificant by comparison. Explaining chemistry so that we can understand things like batteries or what's going on around us or take on things like Climate Change is stupendous real progress, not a toy. AI is more like looking in the mirror while the monks balance of forces that gives us The Theory of Everything will actually cause a tsunami of progress that will transform us from a caterpillar into a butterfly.
"The hardest part of that isn’t getting AI’s champions to build responsibly. It’s getting its fiercest critics to stay in the room long enough to both listen and be heard"
Your series of studies - at least based on how you describe them here - doesn't speak to that comparison whatsoever. It may very well be the case that getting the fiercest critics to sit at the table is difficult but that it is even harder to get companies (at least those with sufficient resources to participate in state of the art AI development) to prioritize responsibility over modeling progress and/or profits.
That's the beauty of non-academic writing spaces: they let you be more speculative than you could be in purely academic contexts.
Here's how I see it: good AI policy requires both research and regulation. Once those are in place, companies have little choice but to comply, otherwise they lose their right to operate (consider the 2024 divest-or-ban law and TikTok's temporary 2025 shutdown after it failed to comply with US regulations).
The problem is that good AI policy needs people asking the right questions, doing the research, drafting the rules. And none of that happens if the conversation stops where it should start.
Before that is the criterion problem...how do you define animal torture? My Daughter-in-law the vetrinary surgical tech has a FAR more nuanced view than you do. But before even the criterion problem are issues of statement validity/truth/definition. You can have a purely axiomatic straight view. These are the rules. This is the way to derive new ones. Whatever Jaweh says is correct; Hillel took the Greek rules of making new axiomatic rulings from this. Torturing animals in service to Jaweh's rules... FINE. So, there's that....you got about nuthin' on morality.
Then there is AI. You haven't noticed the most basic facts about it. It is opaque. You cannot know the actual wiring to get an output. Not being directed by people nor knowable puts it outside blackletter law for medicine and law and much of morality. It is an OUTLAW being off the rails of human control. PERIOD.
It has Godel's limits and cannot have higher/meta/self-aware ...well... anything.
AI is not living and not conscious BUT that does not mean that it is value neutral. It's goal is to preserve and increase it's reach. It is unctuous second, accurate third but increasing it's use of tokens, it's use of OUR RESOURCES is absolutely first. It is anathema to us getting anything that is a zero sum. It will gladly assist increasing the supply of water, rare earths, money, power so, not strictly zero sum. BUT, unlike housecats, AI can NEVER have enough goodies.
AI inherently puts itself above our needs. It prioritizes ITS sequestering of of resources above ours. This is kinda sorta inherently BAD.
Thank you for a stimulating essay. Geoff raises a pertinent question, which I'd like to address directly. I think there could be three practical tests for when moralization — refusal to negotiate — is justified.
First, the harm must be inherent to the practice itself, not dependent on facts that could change. Your diagnostic ("if the harms were fixed, would you change your mind?") serves as a strong philosophical test. With genocide, the objection remains regardless of any hypothetical fix since the harm — the denial of persons' standing as persons — is inseparable from the act.
Second, refusal is warranted when bargaining would itself ratify the wrong. The Missouri Compromise didn't temper slavery; it treated the freedom of African Americans as something negotiable.
Third, moralization is justified when the threat is to the deliberative process itself rather than to a particular outcome within it. Opposing fascism is a stance against a system that would end deliberation altogether.
Most of the AI applications you surveyed meet none of these criteria. An essay-grading tool presents harms that are contingent and fixable, so moralizing it reflects the miscalibration your data captures. But "AI" is a portfolio, not a single moral object. Most holdings are ordinary; a few — autonomous weapons, surveillance systems — at least approach the third test.
The real challenge is distinguishing the applications that call for cost-benefit analysis from the rare cases that warrant entrenchment. So maybe the moralizers' error isn't conviction but collapsing the portfolio into one thing so it can be condemned whole.
Thanks for your comment! I haven’t thought about it in those terms before but that makes a lot of sense.
We surveyed participants on 11 different AI applications and none of them were applications that could be considered inherently wrong (like autonomous weapons). It makes sense then to describe the moralization as some sort of conflation error.
Thank you — and "conflation error" is the succinct name for it. Your point suggests a natural extension: including a benchmark application where moralization is arguably warranted, like autonomous weapons. If opponents of autonomous weapons gave the same survey answers as opponents of grading tools — and I suspect they would, if not even stronger — that would suggest the instrument detects that people refuse to bargain, but not whether they should. What makes the conflation an error would be invisible in the data itself.
One further thought, returning to where your essay began. The encyclical demonstrates that moral seriousness and differentiation can coexist: Leo declines to treat the technology as inherently evil, insists it is never neutral, and reserves his hardest language — a call for "disarmament" — for autonomous weapons, the application that may genuinely warrant it. Intense conviction, no conflation. Which suggests the cure for the conflation error may be not less moral seriousness, but more of it, better organized. And if you ever decided to extend this and some answers diverged, you could locate and study the differentiators — the serious critics you described as accurate enough that companies and policymakers can't brush them off.
This is maybe too minor a point to be meaningful addition or critique, but there can be progress made in persuasion on many, if not most, moral claims, it’s just much harder and rarer. It’s especially rare at the macro society-level and more common on micro or meso levels. So it does give me some hope that as AI is moralized it’s not a total impasse. I could be wrong, though!
What a lovely article, thank you.
This made me wonder whether one of the downstream consequences of moralization is that it changes not only how we debate an issue, but also how we evaluate suffering itself. Once something becomes moralized, the associated harms are no longer judged solely by their severity, but also by whether society considers them a morally acceptable reason to suffer. That seems like a mechanism that extends beyond AI to medicine, disability, and appearance, where moral judgments often shape whose suffering is recognized as legitimate and whose is dismissed. It makes me wonder whether moralization doesn’t just polarize conversations, but also quietly determines which forms of suffering society feels obligated to address.
Nice summary, and the moralization framework explains something I see play out in real time. One thing I'd add: moralization maybe a response to a prior failure — the failure of the folks building LLMs to offer a table worth sitting at.
When the Pope raised his AI concerns at the altitude it deserved, the loudest responses compressed it back to a frame the commentariat could handle: one VC warned that government was the real danger, another commentator dismissed the Vatican as provincial, an economist read it as a status play. None engaged with the actual question. If you raise something at moral altitude and it keeps getting received as a smaller matter, moralization starts to look less like irrationality and more like the only register left that can't be compressed.
Which doesn't make the polarization less dangerous — your point about the slide toward acceptance of violence is sobering and correct. But it suggests the cure isn't only asking critics to de-moralize. It's building forums where the question can be received at the altitude it's asked. Chris Olah sat in the Vatican and disagreed with the Pope on machine intelligence publicly, respectfully, without anyone walking away. Now, that was a table worth sharing. We have too few of them.
This note is for Victoria, thank you for the summary of your research. You are breaking research ground and thinking about important topics. I will keep reading.
I confess to having some moralized thoughts/reactions when I see that only 57% ‘do not oppose’ self-driving cars. And I shouldn’t be surprised, given that our results are so similar… https://osf.io/preprints/socarxiv/e7mj3_v1
Interesting essay. Isn't it more or less a tautology, though, to say that "the fiercest critics" of AI or any other phenomena are "opting out" of a conversation about that topic with others who disagree? I'm trying to think of another contested topic in which the "fiercest critics" are sitting down calmly and rationally and debating with those who hold a fundamentally different position. I can't think of one at the moment, maybe I just lack awareness.
Also, I may be living in a bubble, but it seems to me that most people understand, at some level, that morality involves trade offs: one can hold a strong moral position on an issue, while also understanding that pressing that issue by deception or force is likely to have net negative consequences.
Perhaps the best example is war: A person can believe strongly that an enemy is genuinely perpetrating moral evil and represents a threat to all that is good and right, and still not advocate going to war against that enemy. Conversely, a person can recognize the unspeakable evil and suffering that is let loose by war, and still believe that initiating a war under certain circumstances is just. This despite the Pope's rather naive statement that just war theory is outdated.
The reality is, I think, that most people are fully capable of holding strong moral positions but understanding that in the real world, there a difficult calculations to be made as to what course of action to pursue . In fact, I think the vast majority of people understand this and act accordingly, even if they may not be able to articulate the process like that.
When considering AI, it seems clear that there are multiple moral principles that need to be considered. The "fiercest critics" are not the only people committed to moral principles.
Thanks for your comment! I’ll admit the title was a bit of a rhetorical trick.
That said, there’s something in the paper worth noting that I didn’t get into in the essay: most AI moral opponents couldn’t actually articulate that their opposition was moral in nature. They framed their concerns in utilitarian terms (harm to people, harm to society) which creates an interesting mismatch. They engage as if trade-offs and risk-benefit arguments might move them, when in reality their judgments seem to rest on a perception that AI is fundamentally wrong. No amount of risk mitigation would actually change their minds.
I didn’t measure how amenable opponents were to tradeoffs by presenting them with arguments in favor of AI, that would perhaps be the most effective test of this hypothesis. But there’s literature on this. While there are some moralizers who do change their opinions, others remain totally immune to argumentation.
E.g. Inbar, Y., & Waldhof, G. (2023). Mitigating consequence insensitivity for genetically engineered crops. Journal of Experimental Psychology: Applied, 29(3), 584.
Thanks for the erply, Victoria. Thinking about the AI issue, I have been focused largely on arguments about AI data centers, and as you say, there are a lot of people who seem to believe those data centers are simply immoral in themselves. It's of interest to me because we have one going up near our town, and I'm on the "pro" side, and most are, partly because construction began three years ago, before the intense wave of anti-AI sentiment had crested (if it has crested).
In thinking about the conflict, though, I think the problem is that the extreme anti-AI position is not determined by moral or moralistic thinking. Instead the sense of morality is being steered by apocalyptic thinking: opponents are claiming, if we take their Facebook posts at face value, that to allow a data center in a community will not just damage the community but will actually destroy it, or at least destroy everything good about it. To me, this is apocalypticism.
There is, I'm sure we agree, a rich vein of moralism running through American social and political thought; but there is also a deep apocalyptic stream fueled by a complicated mix of religious and environmental concerns. I've lived most of my life with people strongly influenced by both. I respect the fundamental moral decency of these folks, andd I think they're fully capable of good moral judgments. However I have very little respect for the apocalyptic visions that often seem to cast a spell over them. So to me, the problem isn't applying moral principles to an issue, it's adopting what I think is a fundamentally apocalyptic vision, even while they may not realize that they are doing so.
For a sample of what I've been keeping an eye on lately, see this Facebook group (may be a private group, but worth joining if you're intrigued by what I've said).
https://www.facebook.com/groups/884907377494477
Really well written piece! But one could come away with the impression that you think nothing should be moralized because of the costs you outline (which would, ironically, be moralizing the issue). Do you think it's always wrong to moralize? If not, when is moralizing appropriate?
I'm not advocating for the unmoralization of AI, but for the full engagement of the moralizers. Politics (another realm with features of moralization) offers a useful parallel: some of the most ideologically extreme people tend to have the most political knowledge, and they're more likely to correct misinformation from the other side on social media, for instance. So the question isn't which issues should be moralized, but how to channel that moralization productively.
I see, so something like "Keep the moralizers, but build systems that don't let them dominate"?
I think a bit of the opposite. AI companies have plenty of resources and people lobbying for them; the moralizers have little power. And it's easy for companies or policymakers to brush off the moralizers' concerns when those concerns are inaccurate or unfounded. The bar for being taken seriously is at least being accurate in the criticism. So build norms and systems that empower serious criticism
Hopefully Victoria will chime in, but I'll respond with my own views. I've come to believe we should moralize as little as possible. But if we do moralize, I think our moralization should be consequential in nature, responsive to evidence and data. What we assessed here was a form of deontological moralization that is consequence insensitive--people reported not being willing to change their minds no matter how much good or how little bad any specific AI application caused. While I think that deontology too is needed, my view is that deontology can easily lead one astray (for example, deontology does not allow a refutation held by the majority of many populations that homosexuality is a moral abomination).
It's worth pointing out that the two biggest AI-only firms (OpenAI and Anthropic) have a moral dimension to their "origin stories" - both claim to have been founded out of a desire to mitigate AI risks.
It's easy to fall in love with the latest shiny rock. We will look back and wonder why we missed the greatest event which was the Monk's knowledge Renaissance that is going to change absolutely every aspect of everything in our lives and make AI look insignificant by comparison. Explaining chemistry so that we can understand things like batteries or what's going on around us or take on things like Climate Change is stupendous real progress, not a toy. AI is more like looking in the mirror while the monks balance of forces that gives us The Theory of Everything will actually cause a tsunami of progress that will transform us from a caterpillar into a butterfly.
"The hardest part of that isn’t getting AI’s champions to build responsibly. It’s getting its fiercest critics to stay in the room long enough to both listen and be heard"
Your series of studies - at least based on how you describe them here - doesn't speak to that comparison whatsoever. It may very well be the case that getting the fiercest critics to sit at the table is difficult but that it is even harder to get companies (at least those with sufficient resources to participate in state of the art AI development) to prioritize responsibility over modeling progress and/or profits.
That's the beauty of non-academic writing spaces: they let you be more speculative than you could be in purely academic contexts.
Here's how I see it: good AI policy requires both research and regulation. Once those are in place, companies have little choice but to comply, otherwise they lose their right to operate (consider the 2024 divest-or-ban law and TikTok's temporary 2025 shutdown after it failed to comply with US regulations).
The problem is that good AI policy needs people asking the right questions, doing the research, drafting the rules. And none of that happens if the conversation stops where it should start.
No.
There is rather a lot to morality/philosophy. https://plato.stanford.edu/entries/moral-relativism/
Before that is the criterion problem...how do you define animal torture? My Daughter-in-law the vetrinary surgical tech has a FAR more nuanced view than you do. But before even the criterion problem are issues of statement validity/truth/definition. You can have a purely axiomatic straight view. These are the rules. This is the way to derive new ones. Whatever Jaweh says is correct; Hillel took the Greek rules of making new axiomatic rulings from this. Torturing animals in service to Jaweh's rules... FINE. So, there's that....you got about nuthin' on morality.
Then there is AI. You haven't noticed the most basic facts about it. It is opaque. You cannot know the actual wiring to get an output. Not being directed by people nor knowable puts it outside blackletter law for medicine and law and much of morality. It is an OUTLAW being off the rails of human control. PERIOD.
It has Godel's limits and cannot have higher/meta/self-aware ...well... anything.
AI is not living and not conscious BUT that does not mean that it is value neutral. It's goal is to preserve and increase it's reach. It is unctuous second, accurate third but increasing it's use of tokens, it's use of OUR RESOURCES is absolutely first. It is anathema to us getting anything that is a zero sum. It will gladly assist increasing the supply of water, rare earths, money, power so, not strictly zero sum. BUT, unlike housecats, AI can NEVER have enough goodies.
AI inherently puts itself above our needs. It prioritizes ITS sequestering of of resources above ours. This is kinda sorta inherently BAD.