Some Things Are Not Negotiable. Is AI One of Them?

Here is another guest post by my PhD student, Victória Oldemburgo de Mello. This is a follow-up to her last essay, where she described the results of one of my favourite lab papers on how AI has become moralized by a minority of people. And, yes, they are a minority despite how loud they are and how much the media covers their grievances.

In this new piece Victória gets at something deeper, the permissibility of moralization itself. She interrogates the assertion that the deontological moralization of artificial intelligence is a perfectly reasonable position. She agrees it could be reasonable, but then asks whether it actually is, given what we know about AI and about how moralization works. You’ll also see that she doesn’t spare me in her analysis. Ouch!

I think the essay is brilliant, but I think that about nearly everything Victória does. I’m grateful she decided to let me reprint her essay here for your enjoyment.

---

In my last post on Speak Now Regret Later, I summarized our recent research on the moralization of artificial intelligence. A quick recap: we found that for some people who oppose AI, opposition isn’t really a risk-benefit calculation, but a moral commitment—closer to how most people feel about theft or cruelty than to how they feel about, say, tax policy.

This distinction matters. Moralized attitudes have a specific and well-documented profile. People hold them as non-negotiable rather than as positions to be weighted. They don’t change in response to new evidence. And they correlate with a cluster of other things: intolerance of people on the other side, willingness to endorse violence in service of the cause, and a preference for keeping people with different attitudes at a distance. Whatever you think about AI, that particular bundle of attitudes does not sound like great news for a democratic society.

Several readers—in the comments, and a few more in my DMs—raised a reasonable objection. It went roughly: cool findings, but is the implication here that moralizers should swallow their pride and treat AI as a neutral policy question? And isn’t that a normative claim that your research is smuggling in? It’s a fair question, and I don’t think the answer is obvious. So I spent a while discussing and reading and thinking about it. Here’s where I ended up.

When is AI moralization the right response, and when is it a mistake?

One common answer to that question is based on a consequentialist logic. I call it the lazy answer–and yes, I’m talking to you, Mickey. The lazy answer posits that moralization is simply an error: if good AI policy requires weighting costs and benefits, then refusing to weigh them undermines that process. Under this view, moralizers should really just stop the drama and do better. But that response overlooks how humans actually make moral judgments on a bunch of other domains. Deontological ethics (which arguably describes how humans make moral decisions better than consequentialism) posits an alternative: some things can be wrong not because of the effects they bring about, but because they are wrong in themselves. Think about genocide, slavery, or fascism. They are appropriately held as wrong and non-negotiable, and we typically don’t accept people who treat them as open questions pending better data. This points to a distinction (originally made by Kant) between price and dignity: some things can be traded off against others, but some cannot, and treating the second kind as tradeable is itself a moral error.

The question, then, becomes whether moralized, non-negotiable AI opposition is ever based on some legitimate untradeable principle. So I propose two criteria for judging legitimate untradeable principles: whether the AI objection is based on some non-contingent principle (criterion 1) and whether AI threatens the deliberative process (criterion 2).

Criterion 1: The harm is non-contingent (deontology)

Moral philosophy distinguishes acts that are wrong in themselves from acts that are wrong only because of their consequences. A simple test separates them: if the harms were solved, would the objection go away? Genocide fails this test, as the wrong is the denial of people’s standing as persons, and no fix removes it. So does slavery. Genetically modified organisms (GMOs) are different: despite being morally opposed by some, objections are almost entirely about consequences; the food is unsafe to eat, transgenes will escape into wild populations, seed patents will eliminate small farmers, etc. If the food turned out to be safe, genes stayed put, and farmers remained competitive, would the objection remain? (Ideally) no. That is what it looks like to oppose an act for its consequences rather than in itself.

In our own studies, we asked participants to justify the reasons for their objections to several AI applications. A striking majority mentioned issues such as biased decisions, labor displacement, environmental costs, and misinformation. These are all worth serious consideration, but they’re also contingent: in principle they can be reduced or solved.

A few justifications, however, were non-contingent. Some participants said AI degrades human relationships, or the dignity of human life itself. For these objectors, technical fixes miss the point: we could make AI the best conversation partner imaginable, imbue it with all the right moral values, put every safety guardrail in place, and they would still think there is something wrong with people forming relationships with a machine, and something diminished about a life lived alongside one.

I have to admit, I struggle to grasp this intuition. I do worry that AI companions may be harmful for some people, but I don’t see anything that couldn’t in principle be fixed. For me, AI degrades human relationships no more than dating apps, and yet I don’t see people despising dating apps the way they do AI. I suspect there might be something else behind this justification, but whatever it is, the objection has the structure I’ve been describing—no fix dissolves it. So I’ll take it as non-contingent.

One might conclude, then, that moral opposition is warranted for some AI applications but not others—that objecting to AI companions on grounds of degraded human relationships is a categorically different move from objecting to chatbots on grounds of misinformation risk, since only the former rests on non-contingent reasons. In theory, that distinction should show up in how people actually judge these systems.

In practice, we found almost no evidence of such differentiation: moralization of one AI application strongly predicted moralization of others. Moral attitudes appear to spread across the entire “AI” category. But “AI” names systems that differ radically in purpose, architecture, and origin: a recommendation algorithm and a companion chatbot share little beyond the label. So a person whose objection to one generalizes to all risks extending it beyond the cases that have the feature that made the original objection non-contingent in the first place.

Criterion 2: The threat is to the deliberative process itself (proceduralism)

There is a difference between an outcome one judges wrong through a fair process and a threat to the fair process itself. For instance, opposing fascism is not rejecting one policy among many; it is defending the conditions under which policies can be debated at all. It is possible that some opponents perceive AI as threatening the process of collective deliberation—through concentration of power in the hands of a few powerful actors, the threat of large-scale manipulation, or the degradation of shared information. Another similar form of opposition could be based on a perceived threat of AI technocracy—a political system where decisions are delegated to the technical expertise of AI rather than to elected politicians.

Those are worth serious consideration given current asymmetries in the system: AI development and deployment are concentrated among a small number of firms and individuals who, with substantial resources and little external oversight, hold disproportionate power to determine how AI shapes society. In the United States, this dynamic is evident: industry actors have organized a Super PAC led by some of the wealthiest AI investors with the explicit goal of rolling back AI regulation. At the same time, the federal government has created policies that enforce the removal of all district-level regulations on AI, framing U.S. dominance in AI as a national policy priority and directing the administration to eliminate barriers to that end.

One could argue, however, that some concentration of power does not straightforwardly remove questions from contestation. AI companies can still be subject to litigation, investigative journalism, regulatory complaints, public pressure, employee dissent, and competing products. The question, then, is whether these facts about the current AI system genuinely threaten the process of collective deliberation—and if so, how severely. The degree matters: a distortion of deliberative conditions calls for correction, while their destruction would warrant something closer to outright moral opposition.

Unwarranted but beneficial? The instrumental case for moralization.

The preceding two criteria concern whether AI moralization is warranted—whether it tracks something that genuinely merits non-negotiable opposition. A separate question is whether moralization is socially beneficial, and the two can come apart. A moral stance can be poorly calibrated to the underlying facts and still produce good outcomes through its effects on behavior, since moralization is so useful at mobilizing people. Take food taboos, for instance: they often rest on false beliefs—spiritual pollution, folk theories of the body—yet frequently track real hazards like toxic foods. So here I try to answer: can AI moral opposition, even if unwarranted, be justified instrumentally?

We described earlier two asymmetries in the current AI political landscape: concentrated power and the absence of regulation. Under conditions like these, constraint is unlikely to emerge from calm cost-benefit deliberation, since the better-resourced party is generally positioned to win: it can fund the studies, frame the tradeoffs, and lobby. Checks and balances, in this case, are more likely to come from countervailing power: organized opposition that does not negotiate on the terms the powerful party prefers.

Here, the very feature that makes moralization epistemically unwarranted can become strategic: because moral values resists being traded off, a moralized opposition cannot be bought off, out-argued on cost-benefit grounds, or incrementally accommodated in the way a calculated position can. In other words, moralization may mobilize people in the direction that is most needed when the appropriate checks are not in place, even when the reasons for doing so are unwarranted.

If this is right, moralization can serve as a corrective to a power asymmetry even when it misidentifies the underlying wrong.

The potential consequences of such strategy for AI could be several: it can make developers more cautious by raising the reputational cost of careless deployment; make policymakers warier of industry claims; and it can slow the pace of adoption, buying time for institutions and norms to adapt to changes that might otherwise outrun them. But this could also yield detrimental effects: the same mechanism that checks reckless deployment can also block the development of beneficial AI applications and slow scientific progress, for instance. Each of those assumptions is an empirical claim, and the instrumental defense stands or falls on whether those claims hold.

Two clarifications keep this argument from claiming too much. First, the claim proposed here is the weaker one: that moralization can be a second-best corrective under non-ideal conditions, not that it is good in itself or that a well-functioning deliberative system should prefer it to calibrated judgment. Second, the instrumental motive—collective mobilization checking disproportionally powerful actors—is not necessarily benign; movements that mobilize against powerful institutions can sometimes be detrimental (i.e., organized opposition to vaccination).

In sum

I’ve said a lot, and my verdict is in: “is AI moralization warranted?” admits no single answer. On the two criteria I examined—inherent harm and threats to the deliberative process—most of the AI opposition we observed doesn’t clear the bar. The harms people invoke to justify their opposition are largely contingent and fixable, and the proceduralist case rests on an empirical forecast about how much AI actually impedes deliberation—a forecast that has yet to be established.

But warrant and value can come apart. Under conditions of concentrated power and weak institutional checks, even miscalibrated moralization might do useful work as a countervailing force. Whether it actually does, though, depends on where these attitudes lead, and that’s where things get uncertain.

Now, I want you to picture two futures. In one, today’s moral backlash is what slowed the reckless deployments, forced the audits, and bought regulators enough time to learn—and the outrage, in hindsight, looks like foresight. In the other, that same backlash is what stalled the model that would have flagged the tumor earlier, designed medications faster, or cracked open the next scientific revolution—and the caution, in hindsight, looks like cost. We will only ever live in one of these futures, and we’ll never get to check our version against the other. The upshot isn’t that AI moralization is right or wrong, it’s that its verdict rests on evidence no one will ever hold. And that should make everyone, moralizers and supporters alike, a little more humble.