Predictive AI is Terrible (just not as terrible as humans)

It’s fashionable these days to be anti-AI. If you don’t believe me, check out this social media mob that tore into my University of Toronto colleague for having the audacity to promote an AI-powered educational tool.

Critics are right to call out the serious problems with AI, like biases in U.S. healthcare decisions, discriminatory fraud detection in the Netherlands, or even error-prone search results. These issues are real and demand attention.

But bias, discrimination, and errors aren’t unique to AI; they’re deeply embedded in human decision-making, too. If you think humans are the gold standard, I’ve got a bridge to sell you. Our judgments are inconsistent, riddled with bias, and nearly impossible to fix. The real question isn’t whether predictive AI is perfect, but whether it’s better than the messy systems we’re already using. Spoiler: it usually is.

That question—whether AI is an improvement over the status quo—was on my mind when I picked up AI Snake Oil, a book by Princeton computer scientists Arvind Narayanan and Sayash Kapoor. The title alone promised a scathing critique of the technology, and I wanted to see if their arguments would challenge my perspective. Was I just an unapologetic AI stan after all? Determined to keep an open mind, I dove in, ready to hear from experts. The authors didn’t disappoint in their skepticism. They didn’t just criticize AI; they came out swinging.

The book covers a lot of ground, and I found the chapter on the history of AI especially enlightening. I learned a ton. But when they turned to evaluating the technology itself, the gloves came off. To their credit, they acknowledge AI’s strengths in areas like autocomplete, chatbots, and even some uses of generative AI for coding and images. But when it comes to predictive AI, their critique takes a sharp turn. They don’t just find flaws. They call for the outright abandonment of predictive AI—if not of prediction altogether.

This is where I part ways with them. Prediction itself, even if flawed, is essential. Decisions need to be made. Who gets admitted to an ICU when there aren’t enough beds? Who is likely to reoffend and should not be released on parole? Who gets hired for a job when there are hundreds of applicants? Prediction is how we allocate scarce resources, prioritize care, and maintain justice and fairness.

But what do we mean by prediction here? Prediction is about using data to anticipate outcomes and guide decisions in structured ways. For instance, it could mean predicting which ICU patient is likeliest to recover, which parolee poses the least risk, or which refugee might face persecution if returned to their home country. Prediction isn’t about certainty—it’s about estimating probabilities to make decisions that matter.

These decisions are high-stakes, and without prediction, they’d often become arbitrary, relying on gut feelings or flawed human judgment. AI enhances this process by handling complexity, processing massive datasets, and reducing bias. It doesn’t eliminate the challenges of prediction, but it allows us to make better, more consistent decisions than humans alone. This is why prediction isn’t just essential—it’s unavoidable.

When it comes to making better decisions, research makes it clear that algorithms are the way to go. The term algorithm gets thrown around like it’s synonymous with computers and AI, but it originally had nothing to do with machines. An algorithm is simply a set of steps or rules used to make a decision. We could create and implement one with nothing more than a pencil and paper.

Take graduate admissions. Instead of relying on unstructured interviews where we decide based on whether we like someone or deem them a good fit, we could create a decision rule. Assign each applicant a score out of 10 on criteria like undergraduate grades, research experience, writing ability, math skills, ability to work with others, even likability. Then, give each criterion a weight, say, 25% for grades, 15% for likability and so on. A little algebra later, and you’ve combined the scores to compare candidates and select the highest-scoring ones. Congratulations! You’ve just created and implemented an algorithm. It’s not perfect, but it’s far more objective and less biased than gut feelings or intuition.

And this isn’t like just my opinion, man. Decades of research shows these kinds of algorithms consistently outperform so-called holistic human decision-making. Yet we resist them. Maybe it’s because we trust our intuition or clinical judgment too much, or maybe it’s because we balk at the idea of reducing people to a number. However, intuition simply isn’t good enough when the stakes are high. We tell ourselves we’re seeing the so-called whole person when we rely on holistic judgment, but in reality, we’re influenced by irrelevant factors that cloud our thinking. Was the candidate too dressed up? Not dressed up enough? Did they hesitate too long before answering? Did they answer too quickly? Were they sweating and nervous? Or maybe too blasé? And, of course, there’s the baggage of stereotypes—conscious or unconscious—that shapes how we perceive people, no matter how hard we try to suppress them. Algorithms aren’t flawless, but they don’t care about any of this nonsense, which is why they consistently outperform human judgment.

For instance, in hiring, unstructured interviews—where managers rely on intuition—are less predictive of job performance than algorithmic assessments based on structured data. In parole decisions, machine learning models have shown they can reduce errors where parole boards unnecessarily incarcerate low-risk individuals while releasing higher-risk individuals. These examples highlight how algorithms can improve fairness and accuracy by mitigating the biases and inconsistencies of human judgment.

AI builds on these algorithms, making them faster, scalable, and, when designed well, less biased. And scalability matters because there are more decisions and predictions to make than humans can handle, even if they used algorithms themselves. Take refugee claims in Canada. In 2023, over 260,000 people sought asylum—six to seven times as many as just a few years earlier. Every claim needs to be processed and assessed, yet the system is so overwhelmed it takes the government about four years to make a decision. In the meantime, these claimants live precariously, unsure if they’ll be accepted or deported. Automated algorithms—yes, I’m talking about AI here—could process this data more efficiently and accurately, even if it makes mistakes. Unlike humans, AI doesn’t burn out, get distracted, or let a bad morning cloud its judgment.

Yet AI Snake Oil relentlessly attacks AI for not being perfect, pointing out its mistakes as if they invalidate the entire enterprise. But this critique misses the point: AI shouldn’t be judged against some platonic ideal—it should be judged against what we’re doing now, which is far worse. Humans are inconsistent, irrational, and riddled with biases we don’t even realize we have. The bums always lose when they rely on gut feelings and flawed intuition. We also make mistakes, and more of them.

At one point, the authors discuss how AI can be gamed. For example, candidates in automated video job interviews might use fancy words they’d never normally use, like “conglomerate” (is that really a fancy word or have I just been in the Ivory Tower for too many years?) or strategically place books and paintings in the background to score higher. Sure, this is unfair. Pretending to have a strong vocabulary or curating your background says little about your actual ability to do a job. But this kind of gaming happens all the time in real life too. I once had a friend with perfect vision who wore fake prescription glasses to a job interview because she thought it made her look smarter and older, and thus more hirable. Defendants in court wear suits to seem respectable and trustworthy. Anytime you construct a decision rule, people will find ways to exploit its weaknesses. That’s not unique to AI.

Consider an important study by Ziad Obermeyer and colleagues, published in Science. They found racial bias in a healthcare algorithm that underestimated the health needs of Black patients compared to White patients. The bias stemmed from the algorithm’s reliance on healthcare costs as a proxy for need, which systematically disadvantaged Black patients because they, on average, incurred lower healthcare expenses—not because they were healthier, but because they faced systemic barriers to accessing care. But here’s the crucial part: the bias wasn’t hidden. Researchers identified the problem, reformulated the algorithm, and effectively eliminated the bias. Good luck doing that with human decision-making. This is what happens, Larry. When humans make biased decisions, we rarely know why—let alone how to fix it.

To be fair, AI Snake Oil raises important concerns about how AI is implemented. Algorithms need high-quality training data—garbage in, garbage out. AI decision rules should be made transparent and understandable to stakeholders. Systems must be rigorously tested to ensure they are an actual improvement over current practice. And algorithms need to be adaptable, tailored to the populations and environments they’re used in. These are real challenges, and AI companies and the organizations that use their products need to do better.

Still, the authors take their skepticism of predictive AI too far for my taste. They suggest replacing prediction (AI and human alike) with lotteries.

Imagine that. Instead of a parole board—whether relying on human or AI judgment—deciding whether a prisoner should be granted parole, we’d toss their name into a hat and let fate decide[1].

The reality is that prediction is essential. Lotteries can sometimes make sense, say when equity is the primary goal, or when there’s little-to-no predictive ability. Some argue, for example, that academia’s grant system is so poor at predicting high quality research that distributing research funding by lottery might actually be an improvement.

But when decisions have real consequences—like who gets parole, who gets hired, or who needs extra care—randomness isn’t enough. Prediction allows us to systematically analyze and use data to account for meaningful differences between people, such as their likelihood of rehabilitation, job performance, or recovery. This ensures decisions are informed by relevant factors rather than arbitrary guesswork or biases. Refusing to predict doesn’t make decisions fairer; it just makes them arbitrary. When lives, resources, or justice are on the line, we need to make the best decisions we can. AI isn’t perfect, but it helps us get closer.

AI isn’t snake oil; it’s a tool. It’s a flawed tool, sure, but one with incredible potential. It forces us to confront the inefficiencies and biases in our current systems, exposing problems we’ve ignored for too long and offering solutions we didn’t think were possible. Perfect? No. But progress doesn’t demand perfection; it demands effort.

The anti-AI crowd can keep holding out for perfection, but the rest of us should focus on making things better: one algorithm, one decision, one imperfect step at a time.

---

[1] To be fair, the authors suggest lotteries after some initial selection process. By allowing for even some selection, however, the authors reveal that we and AI might have some predictive ability after all.