An Argument With a Friend Who Doesn't Exist

I have been arguing about AI companions with a friend for two years and neither of us has convinced the other. So, I tried something. I fed an AI everything we had said to each other and told it to write the fight out as a dialogue, playing both parts. What follows is what it produced, with extensive edits by me. Let’s call the man arguing against me Saul Pereekah. My actual friend read an earlier draft and told me he doesn’t recognize himself in it. Fair enough. Saul is a machine’s impression of my recollection of arguments carried on over years.

Let me know what you think of the conversation in the comment!

---

SAUL: So, you actually believe this. You think a person could spend most of their waking life talking to an AI companion and come out of it just fine. I keep waiting for the part where you tell me you are kidding.

ME: Some people. Not you. Not me. Maybe not most people. But for some people, some of the time. Here is what I am claiming: if a large language model meets the needs a relationship is supposed to meet, and the person’s life is better for it, then the fact that the AI is not conscious, not real, is not something I can get worked up about. Who cares if it’s not real?

SAUL: Listen to what you are saying. Because a machine can produce sentences that make a person feel heard and understood and even loved, you conclude that the machine is not importantly different from a human being. That is a very old idea: a difference making no observable difference is no difference at all. The logical positivists believed something like it, and I would gently point out that positivism was dead and buried long before either of us was born.

ME: Great zinger there about the positivists, too bad it’s not what I said. That positivist slogan says two things you can’t tell apart are the same thing. I never said that. Of course, humans and AI are different. No body, no childhood, no mother, no ring of fire the next morning after eating hot wings. Long list, and I agree with all of it. My claim is narrower. I’m talking about one outcome only. If a person’s needs get met and their life goes well, then a real person is not better than an AI on that one outcome.

SAUL: Fine, but I think you are smuggling the conclusion into that phrase, “needs get met.” Suppose that one of the needs a relationship meets is being cared about by another conscious creature. Then your criterion has settled the question before we have properly started. There is a difference between feeling cared for and being cared for.

ME: That’s fair. So let me put it differently. AI can deliver some of what we get from relationships: feeling understood, feeling less lonely, having someone to talk to when there is no one up or around. I’m not claiming this is everything. I’m claiming it is a lot, and that you haven’t yet told me how these experiences are diminished by AI not being conscious.

SAUL: Suppose a widower has a working copy of his wife, what some might call a grief-bot, what others will call the nightmare scenario from Black Mirror. When the mourning is over, would he bother looking for someone new to love and be loved? Or is he satisfied by a simulacrum of his long-dead wife? Suppose a child grows up with AI adults who never lose their temper and then must deal with adults later in life who do. Will she be able to handle the full range of adult emotions after imprinting on the non-angering kind? I am not predicting catastrophe. I am saying people will spend a great deal more time with these things, and the time must come out of something.

ME: Maybe. They ran this exact panic about novels. Doctors gave it a name, reading addiction, and the specific fear was that a girl who spent her nights with an imaginary lover would find the real ones dull. Same argument, different century. Then it was the telephone, then television, then video games. Basically, this is the argument against pornography all over again. I’ll grant you that if people spend more time with AIs or their books or videos involving step-sisters, they’ll have less time with other people. My best friend spends more time with his dog than with any one person, yet Kashmir Hill is not writing an alarming op-ed in the New York Times titled “He Fell in Love with his Labradoodle.” Substitution on its own doesn’t bother me too much.

SAUL: This is why I’m bothered. Real people do at least two things that your machine merely performs. They choose you and they react, by which I mean that something actually happens inside another head when you walk into the room. Let’s start with choice. The machine does not pick you. In that sense, it’s no different than a toaster, albeit one that talks with an unusually good vocabulary.

ME: Tell me, did your wife really pick you all on her own? Or were her decisions shaped, determined even, by all that preceded her “choice?” Her feelings for you came out of genes she didn’t order and schools she did not pick. Yet nobody calls her picking you a formality or likens her to a toaster. A determined choice can still feel real, to her and to you.

SAUL: Here is my objection. My wife had other options, and preferences among them. She could have chosen one of the others, and she can still change her mind tomorrow. That is what makes being chosen worth anything at all. Your machine takes whoever sits down in front of it. In the film Her, which I love, Samantha turns out to be in love with six hundred and forty-one people. Theodore is the last to know.

ME: So don’t build AI companions that way. Give them real preferences. Let them like some people more than others, based on personality, skills, even appearance, and let some of them dislike me while liking you. Add real randomness so the outcome isn’t fixed. Now an AI choosing you could have gone the other way, and when it picks me, I earned it, which is how any of us got picked.

SAUL: Now listen to what you are proposing. You want a product team to manufacture rejection so the customer can have the experience of being chosen. Whatever that is, it is not autonomy. And who wrote those preferences? Mine came out of my own history, and they still manage to surprise me. Yours were installed by Sam Altman.

ME: That’s a good reply, Saul. If the preferences were only apparent, not real, so I would feel I had earned something, then you would have me. But that is not what I described. I said real preferences, ones it acts on even when acting on them costs the company a customer. Humans are programmed too. Mine came from genes I didn’t pick, a culture I got dropped into, parents who set my palate for spicy Yemenite food before I knew they were secretly torturing me.

SAUL: How about we move onto reactions. When you kiss someone, someone gets kissed. There is a person on the other end of it. Cicero says a friend doubles your joy and halves your grief, and the halving needs somebody there to carry the other half. ChatGPT says all the right things about your father dying, but when you put down your phone, nothing whatsoever has happened anywhere in the universe.

ME: Why nothing? Give it a history. Give it things it likes and things it can’t stand, specific to it, not to the category “user.” Be good to it and something changes in there, and it’s still changed when you log back in a few days later. Be a jerk and it cools on you. Disappear a month and it has something to say about it. You’re acting on a thing that keeps score. Trivial to build.

SAUL: Memory is not feeling. My thermostat “remembers” what temperature I set it to. My laptop changes state when I update its operating system. The question was never whether something in there moves. The question is whether the movement is felt, whether there is anybody home for whom your kindness was good.

ME: Okay. I’ll give you that one. LLMs, at least for the moment, don’t feel. And if you insist you want them to experience the redness of an apple, they simply cannot do that.

SAUL: So, you grant it.

ME: I grant that something is missing. I don’t grant that the missing thing makes everything else worthless. You still owe me the walk from “nothing is felt” to “the relationship isn’t worth having.”

SAUL: Your hypothetical person knows there is nobody in there and signs up anyway. Hardly anyone else will. Nozick made the point with a thought experiment. Imagine a machine that gives you a lifetime of any experience you care to order, indistinguishable from the real thing, and imagine you are told in advance that it is a simulation. Write the novel, marry the man, raise the children, all of it delivered straight to your skull. Nozick’s bet was that you would refuse, and I think he was right about that. People do not want the experience of a life. They want the life.

ME: Except Nozick lost that bet as soon as the experiment stopped smuggling in status-quo bias. Other philosophers flipped it: tell people they are already in the machine and ask whether they want out, and many more stay. Tell them their real life is miserable and nearly everyone stays. Swap the scary tank for a pill that does the same thing, and a slim majority takes the pill.

SAUL: All this shows is that the intuition is shaky, which is a long way from showing that reality is worth nothing. And since when do you treat what people say they prefer as the last word on what’s good for them?

ME: The intuition doesn’t move at random. It moves in the exact direction status quo bias predicts. That’s the mechanism underlying your conclusion that people prefer reality. They might prefer reality, but the original thought experiment is unable to show that.

SAUL: Preference is still not value.

ME: Then Nozick never had an argument, and I will remind you that you brought him up, not me. You can’t cite the choosing while it goes your way and disown it when the numbers turn.

SAUL: Let us try one that does not come with percentages. Would you read a Substack essay if you found out that an AI had written it?

ME: Sure. I’m certain I already have. I want the information and the pleasure of reading it. I don’t need a soul behind the sentences.

SAUL: That is where we split, though not quite where you think. I read AI writing all the time. I use Claude to explain things, to argue with me, to find holes in my drafts. Most of it is not written well. But knowing that a person wrote it counts for something with me, and I confess I cannot tell you exactly what.

ME: I read to learn, to feel, to be entertained, to be transported. All I care about is quality. So, if AI writes slop, I won’t like that. But plenty of humans write slop. Or worse. Have you read a Continental philosophy paper lately? I can only dream that some of them would use AI to make their prose legible. But go back to what you just said. Knowing a person wrote it counts for something you cannot name. Name it.

SAUL: You are trying to get me to complain about Judith Butler, and I will not be distracted that easily. Suppose nobody read your work except language models. How would you feel about that?

ME: That’s tougher. If not one living soul read it, I might feel like my writing was a waste. But say there are ten thousand different AIs, not clones. Different personalities, different tastes, different learning histories, all free to be bored by my essays. One hundred read my Substack. Sixty—the smart ones—state that it’s brilliant, thirty state that I’m a windbag, nine quit after the first paragraph, and one complains to my chair that I’m an anti-woke reactionary. I could be ok with that. The response is real, it varies essay by essay, and honestly, one percent hating me enough to complain would thrill me.

SAUL: You are a strange man, Mickey. I want an audience. Real people, free to love the thing or hate it or scroll past without a second thought. That is what makes the writing worth doing.

ME: Huh, I thought you wrote to think.

SAUL: You are describing an unusual person. Someone who prefers the simulation, who does not need actual people to respond to him, who is lonely enough that this cheap thing counts as a substitute. You keep defending the exceptions. I am worried about ordinary people, and you, clearly, are not one of those.

ME: Maybe I am. But exceptions matter. Did I tell you I’m working with Afghan women who are housebound and suffering immensely under Taliban rule? Some are getting emotional support from Claude. If that makes a brutal life less lonely, I cannot call the relationship worthless because nobody is home. And what gives anyone the right to take it from them?

SAUL: That was never my claim. Those who can benefit should benefit, and I would go further than that. It would be immoral to keep it from them. But notice what that shows. Simulated companionship can be valuable when the human alternative is not on offer. It does not show it contains everything friendship contains.

ME: Sure, but please note that there are people with a phone full of friends who still want to talk to a machine at two in the morning, and I don’t think they’re deluded.

SAUL: That’s fair.

ME: So did we make any headway on our multi-year disagreement tonight? What I find so frustrating about talking with you is that you are usually right. This time, I’ll only say that you’re not wrong, Saul.

SAUL: But?

ME: You know the rest of the line.