Does Data Matter in Psychology?

Psychology presents itself as deeply empirical and quantitative. But is it?

I ask because psychologists will regularly ignore data when it goes against their own intuitions and personal experiences. This is perhaps clearest when we look at the various casualties of the replication crisis—and people’s responses to these failures.

Let’s start with the topic of one of my Substack posts, stereotype threat. Stereotype threat is the idea that people from negatively stereotyped groups choke under pressure when those stereotypes become salient. So, when high-performing Black students are reminded they're Black, they feel unnerved by negative academic-ability stereotypes and supposedly underperform on academic tests. Similarly, women become anxious when faced with the stereotype that “girls aren’t good at math” and are more likely to bomb math exams. It was psychology's darling theory for explaining and remedying achievement gaps that left progressives feeling good about themselves.

When I started voicing tentative doubts about stereotype threat in a blogpost that went viral, some of the responses pushed back. I’ll never forget how a prominent African American psychology professor struggled to make sense of my doubts because she had experienced stereotype threat personally. She didn’t expand, but it’s not hard to imagine how negative stereotypes might make her feel any time she entered a room. For her, it was clear as day that stereotype threat was real no matter what the data suggested.

A similar thing has occurred to my favourite replication crisis victim, ego depletion. You’re probably sick of me talking about it but let me quickly explain for the newbs. Ego depletion is the idea that self-control works like a fuel tank: If you use self-control in one domain, you'll have less available for another. So if you spend your morning biting your tongue while your boss acts like he’s the Malibu Police Chief, you might find yourself stress-eating butter tarts by lunch or binge-watching Love Island USA that night. For two decades, psychology got drunk on this theory and got busy spawning hundreds of studies, but then the buzz-kill data police turned the music off and told all the drunk revellers to go home.

While most revellers eventually agreed that the replication police were right, many weren't willing to say ego depletion isn't real. Even if the data aren't there, the concept is still solid. This is especially true if you water down ego depletion to mental fatigue. Everyone knows that after a full day of work, you feel tired, and certain tasks seem harder. The data say one thing, but common sense says something else.

Finally, I recently wrote about the death of implicit bias when describing a new paper on the utter and complete failure of measures of implicit bias to deliver on their vast promise. Implicit bias is the idea that many of us harbour prejudices that we are unaware of, that we cannot control, and that automatically influence our behaviour. Scholars have been trying to revive the concept of implicit bias, but this time the scholars doing the reviving are the same ones burying the corpse.

In a series of excellent papers, my buddy Bertram Gawronski makes the point that measures of implicit bias are not the same as the concept of implicit bias. Actual implicit bias occurs when people genuinely don't know their behaviour is influenced by race or gender and this is distinct from how people perform on computer tasks like the implicit association test. The data say that implicit bias is not so implicit after all, and we respond by saying our concepts were always right, it’s just that our tools are broken.

To recap, here are three hotly contested areas that have each been found wanting, empirically. Our response each and every time is that the concepts are still valid—we get tired, dummy; stereotypes influence how I feel, how dare you say otherwise; and of course we're all secretly racist, our computers are just too dumb to catch us in the act.

Can we just keep these beloved theories around without supporting data? I mean, this is not ’Nam. There are rules. So, again I ask: Does data matter in psychology?