This is maybe too minor a point to be meaningful addition or critique, but there can be progress made in persuasion on many, if not most, moral claims, it’s just much harder and rarer. It’s especially rare at the macro society-level and more common on micro or meso levels. So it does give me some hope that as AI is moralized it’s not a total impasse. I could be wrong, though!
This note is for Victoria, thank you for the summary of your research. You are breaking research ground and thinking about important topics. I will keep reading.
I confess to having some moralized thoughts/reactions when I see that only 57% ‘do not oppose’ self-driving cars. And I shouldn’t be surprised, given that our results are so similar… https://osf.io/preprints/socarxiv/e7mj3_v1
Really well written piece! But one could come away with the impression that you think nothing should be moralized because of the costs you outline (which would, ironically, be moralizing the issue). Do you think it's always wrong to moralize? If not, when is moralizing appropriate?
Hopefully Victoria will chime in, but I'll respond with my own views. I've come to believe we should moralize as little as possible. But if we do moralize, I think our moralization should be consequential in nature, responsive to evidence and data. What we assessed here was a form of deontological moralization that is consequence insensitive--people reported not being willing to change their minds no matter how much good or how little bad any specific AI application caused. While I think that deontology too is needed, my view is that deontology can easily lead one astray (for example, deontology does not allow a refutation held by the majority of many populations that homosexuality is a moral abomination).
I'm not advocating for the unmoralization of AI, but for the full engagement of the moralizers. Politics (another realm with features of moralization) offers a useful parallel: some of the most ideologically extreme people tend to have the most political knowledge, and they're more likely to correct misinformation from the other side on social media, for instance. So the question isn't which issues should be moralized, but how to channel that moralization productively.
I think a bit of the opposite. AI companies have plenty of resources and people lobbying for them; the moralizers have little power. And it's easy for companies or policymakers to brush off the moralizers' concerns when those concerns are inaccurate or unfounded. The bar for being taken seriously is at least being accurate in the criticism. So build norms and systems that empower serious criticism
Nice summary, and the moralization framework explains something I see play out in real time. One thing I'd add: moralization maybe a response to a prior failure — the failure of the folks building LLMs to offer a table worth sitting at.
When the Pope raised his AI concerns at the altitude it deserved, the loudest responses compressed it back to a frame the commentariat could handle: one VC warned that government was the real danger, another commentator dismissed the Vatican as provincial, an economist read it as a status play. None engaged with the actual question. If you raise something at moral altitude and it keeps getting received as a smaller matter, moralization starts to look less like irrationality and more like the only register left that can't be compressed.
Which doesn't make the polarization less dangerous — your point about the slide toward acceptance of violence is sobering and correct. But it suggests the cure isn't only asking critics to de-moralize. It's building forums where the question can be received at the altitude it's asked. Chris Olah sat in the Vatican and disagreed with the Pope on machine intelligence publicly, respectfully, without anyone walking away. Now, that was a table worth sharing. We have too few of them.
It's easy to fall in love with the latest shiny rock. We will look back and wonder why we missed the greatest event which was the Monk's knowledge Renaissance that is going to change absolutely every aspect of everything in our lives and make AI look insignificant by comparison. Explaining chemistry so that we can understand things like batteries or what's going on around us or take on things like Climate Change is stupendous real progress, not a toy. AI is more like looking in the mirror while the monks balance of forces that gives us The Theory of Everything will actually cause a tsunami of progress that will transform us from a caterpillar into a butterfly.
"The hardest part of that isn’t getting AI’s champions to build responsibly. It’s getting its fiercest critics to stay in the room long enough to both listen and be heard"
Your series of studies - at least based on how you describe them here - doesn't speak to that comparison whatsoever. It may very well be the case that getting the fiercest critics to sit at the table is difficult but that it is even harder to get companies (at least those with sufficient resources to participate in state of the art AI development) to prioritize responsibility over modeling progress and/or profits.
That's the beauty of non-academic writing spaces: they let you be more speculative than you could be in purely academic contexts.
Here's how I see it: good AI policy requires both research and regulation. Once those are in place, companies have little choice but to comply, otherwise they lose their right to operate (consider the 2024 divest-or-ban law and TikTok's temporary 2025 shutdown after it failed to comply with US regulations).
The problem is that good AI policy needs people asking the right questions, doing the research, drafting the rules. And none of that happens if the conversation stops where it should start.
Before that is the criterion problem...how do you define animal torture? My Daughter-in-law the vetrinary surgical tech has a FAR more nuanced view than you do. But before even the criterion problem are issues of statement validity/truth/definition. You can have a purely axiomatic straight view. These are the rules. This is the way to derive new ones. Whatever Jaweh says is correct; Hillel took the Greek rules of making new axiomatic rulings from this. Torturing animals in service to Jaweh's rules... FINE. So, there's that....you got about nuthin' on morality.
Then there is AI. You haven't noticed the most basic facts about it. It is opaque. You cannot know the actual wiring to get an output. Not being directed by people nor knowable puts it outside blackletter law for medicine and law and much of morality. It is an OUTLAW being off the rails of human control. PERIOD.
It has Godel's limits and cannot have higher/meta/self-aware ...well... anything.
AI is not living and not conscious BUT that does not mean that it is value neutral. It's goal is to preserve and increase it's reach. It is unctuous second, accurate third but increasing it's use of tokens, it's use of OUR RESOURCES is absolutely first. It is anathema to us getting anything that is a zero sum. It will gladly assist increasing the supply of water, rare earths, money, power so, not strictly zero sum. BUT, unlike housecats, AI can NEVER have enough goodies.
AI inherently puts itself above our needs. It prioritizes ITS sequestering of of resources above ours. This is kinda sorta inherently BAD.
This is maybe too minor a point to be meaningful addition or critique, but there can be progress made in persuasion on many, if not most, moral claims, it’s just much harder and rarer. It’s especially rare at the macro society-level and more common on micro or meso levels. So it does give me some hope that as AI is moralized it’s not a total impasse. I could be wrong, though!
What a lovely article, thank you.
This note is for Victoria, thank you for the summary of your research. You are breaking research ground and thinking about important topics. I will keep reading.
I confess to having some moralized thoughts/reactions when I see that only 57% ‘do not oppose’ self-driving cars. And I shouldn’t be surprised, given that our results are so similar… https://osf.io/preprints/socarxiv/e7mj3_v1
Really well written piece! But one could come away with the impression that you think nothing should be moralized because of the costs you outline (which would, ironically, be moralizing the issue). Do you think it's always wrong to moralize? If not, when is moralizing appropriate?
Hopefully Victoria will chime in, but I'll respond with my own views. I've come to believe we should moralize as little as possible. But if we do moralize, I think our moralization should be consequential in nature, responsive to evidence and data. What we assessed here was a form of deontological moralization that is consequence insensitive--people reported not being willing to change their minds no matter how much good or how little bad any specific AI application caused. While I think that deontology too is needed, my view is that deontology can easily lead one astray (for example, deontology does not allow a refutation held by the majority of many populations that homosexuality is a moral abomination).
I'm not advocating for the unmoralization of AI, but for the full engagement of the moralizers. Politics (another realm with features of moralization) offers a useful parallel: some of the most ideologically extreme people tend to have the most political knowledge, and they're more likely to correct misinformation from the other side on social media, for instance. So the question isn't which issues should be moralized, but how to channel that moralization productively.
I see, so something like "Keep the moralizers, but build systems that don't let them dominate"?
I think a bit of the opposite. AI companies have plenty of resources and people lobbying for them; the moralizers have little power. And it's easy for companies or policymakers to brush off the moralizers' concerns when those concerns are inaccurate or unfounded. The bar for being taken seriously is at least being accurate in the criticism. So build norms and systems that empower serious criticism
Nice summary, and the moralization framework explains something I see play out in real time. One thing I'd add: moralization maybe a response to a prior failure — the failure of the folks building LLMs to offer a table worth sitting at.
When the Pope raised his AI concerns at the altitude it deserved, the loudest responses compressed it back to a frame the commentariat could handle: one VC warned that government was the real danger, another commentator dismissed the Vatican as provincial, an economist read it as a status play. None engaged with the actual question. If you raise something at moral altitude and it keeps getting received as a smaller matter, moralization starts to look less like irrationality and more like the only register left that can't be compressed.
Which doesn't make the polarization less dangerous — your point about the slide toward acceptance of violence is sobering and correct. But it suggests the cure isn't only asking critics to de-moralize. It's building forums where the question can be received at the altitude it's asked. Chris Olah sat in the Vatican and disagreed with the Pope on machine intelligence publicly, respectfully, without anyone walking away. Now, that was a table worth sharing. We have too few of them.
It's easy to fall in love with the latest shiny rock. We will look back and wonder why we missed the greatest event which was the Monk's knowledge Renaissance that is going to change absolutely every aspect of everything in our lives and make AI look insignificant by comparison. Explaining chemistry so that we can understand things like batteries or what's going on around us or take on things like Climate Change is stupendous real progress, not a toy. AI is more like looking in the mirror while the monks balance of forces that gives us The Theory of Everything will actually cause a tsunami of progress that will transform us from a caterpillar into a butterfly.
"The hardest part of that isn’t getting AI’s champions to build responsibly. It’s getting its fiercest critics to stay in the room long enough to both listen and be heard"
Your series of studies - at least based on how you describe them here - doesn't speak to that comparison whatsoever. It may very well be the case that getting the fiercest critics to sit at the table is difficult but that it is even harder to get companies (at least those with sufficient resources to participate in state of the art AI development) to prioritize responsibility over modeling progress and/or profits.
That's the beauty of non-academic writing spaces: they let you be more speculative than you could be in purely academic contexts.
Here's how I see it: good AI policy requires both research and regulation. Once those are in place, companies have little choice but to comply, otherwise they lose their right to operate (consider the 2024 divest-or-ban law and TikTok's temporary 2025 shutdown after it failed to comply with US regulations).
The problem is that good AI policy needs people asking the right questions, doing the research, drafting the rules. And none of that happens if the conversation stops where it should start.
No.
There is rather a lot to morality/philosophy. https://plato.stanford.edu/entries/moral-relativism/
Before that is the criterion problem...how do you define animal torture? My Daughter-in-law the vetrinary surgical tech has a FAR more nuanced view than you do. But before even the criterion problem are issues of statement validity/truth/definition. You can have a purely axiomatic straight view. These are the rules. This is the way to derive new ones. Whatever Jaweh says is correct; Hillel took the Greek rules of making new axiomatic rulings from this. Torturing animals in service to Jaweh's rules... FINE. So, there's that....you got about nuthin' on morality.
Then there is AI. You haven't noticed the most basic facts about it. It is opaque. You cannot know the actual wiring to get an output. Not being directed by people nor knowable puts it outside blackletter law for medicine and law and much of morality. It is an OUTLAW being off the rails of human control. PERIOD.
It has Godel's limits and cannot have higher/meta/self-aware ...well... anything.
AI is not living and not conscious BUT that does not mean that it is value neutral. It's goal is to preserve and increase it's reach. It is unctuous second, accurate third but increasing it's use of tokens, it's use of OUR RESOURCES is absolutely first. It is anathema to us getting anything that is a zero sum. It will gladly assist increasing the supply of water, rare earths, money, power so, not strictly zero sum. BUT, unlike housecats, AI can NEVER have enough goodies.
AI inherently puts itself above our needs. It prioritizes ITS sequestering of of resources above ours. This is kinda sorta inherently BAD.