My P-Hacking Felt Like Truth

I became a social psychologist because I wanted to make the world a better place.

After taking Introduction to Social Psychology with the legendary McGill professor Donald Taylor, I knew I had found my calling. Taylor—who sadly passed away during the pandemic—was a natural performer and storyteller who inspired countless young Canadian undergraduates to pursue careers in social psychology. For me, Taylor’s material came most alive when he talked about the psychology of prejudice.

Before Taylor’s class, I had not realized there was a big part of an entire discipline devoted to studying discrimination in the hopes of not just understanding it but remedying it. As a Jewish kid, with Holocaust survivors on one side of my family and Yemeni refugees on the other, I had a keen eye for injustice. My family had been on the wrong side of hate more than once, including, I later learned, at the hands of pale-skinned Ashkenazi Jews. My formal education in Jewish history taught me that judging people by their identity rather than the content of their character was a sin, a moral stain. When I learned there was an entire discipline focused on this, I knew I had found my intellectual home.

I dove in headfirst. At Brown and NYU, where I did my PhD and postdoc, I conducted research on stereotype threat. At the University of Toronto, I created the university’s first course on the psychology of prejudice and taught it for nearly fifteen years. It felt like I was changing the world, one small step at a time. I could not have been prouder.

Then the replication crisis happened. I came to realize that much of my past work was the product of questionable research practices, and I no longer trusted it. I’ve recounted this story many times. But what I haven’t told you is the part I only see now, in hindsight. At the time, none of it felt like cheating. And I can’t help but wonder if it felt fine because the cause was good.

My work was uplifting people, or so I told myself. Looking back, I suspect that feeling like I was on the right side of history was a kind of moral cover. Maybe I felt less wrong about the p-hacking because I was promoting social justice. The conclusions I drew were what the good guys already believed, after all.

However, a scientist who wants to change the world in a particular direction is no longer beholden to the truth. They are beholden to a cause. I was. My question stopped being “What is true here?” and slid toward “How do I show that what I believe is true?” This is why I have come to believe that science and activism cannot mix. It is a plain conflict of interest.

I was never an activist, but I knew which answer I was hoping for in my research. That is all it takes. What I did quietly, some in psychology now argue the whole field should do on purpose: make scholarship serve the cause of justice. To see where that leads, let’s look at a discipline that has already done this. Consider anthropology, which once aspired to science.

Last month a group of scholars convened by the chancellors of Vanderbilt and Washington University in St. Louis reviewed several fields and found scholarly standards giving way to political ones, activism crowding out inquiry. They named anthropology the worst case, a discipline that had traded the search for truth for the advancement of a cause. Now one can argue about the report’s methods and conclusions, but a recent defense of anthropology by the president of the American Anthropological Association only confirms the report’s conclusions—and the public’s worst fears.

Carolyn Rouse, who once delivered a lecture titled “F%*# Free Speech,” sat for an interview to defend her field. Shown a 2022 survey in which anthropologists split almost evenly on whether biological sex is binary, she waved it off: “I don’t believe in opinion research.” Say what!? A survey of practitioners in her own discipline turns worthless the moment it returns an inconvenient number. She compares teaching that sex is binary to turning an astronomy department into astrology.

Set aside whether Rouse is right. Pay attention to what she is doing: Evidence that agrees with her is fact; evidence that disagrees with her is contested. Has the whole world gone crazy? Am I the only one who gives a shit about the rules?

This is what I mean when I say science and activism should not mix. It is not that activists are dishonest or that justice does not matter. It is that science can only be loyal to one thing: the truth. When truth comes first, a survey that cuts against you is data you must accommodate. When justice comes first, that survey is noise to explain away, and the field slowly loses touch with reality. Rouse has a cause, and the cause now decides what counts as evidence.

Anthropology shows an entire discipline sorting evidence by conclusion. The next case is smaller, a single episode, but it happened in philosophy, a field I love and which is supposed to be best at following an argument wherever it goes.

In 2017, Rebecca Tuvel, then an untenured professor at Rhodes College, published “In Defense of Transracialism” in the feminist journal Hypatia, arguing that, if we accept the reasoning that supports transgender identities, and she does, that same reasoning seems to support transracial identities too. It didn’t matter that her argument was logically sound: more than 800 academics signed an open letter demanding retraction and a majority of Hypatia‘s associate editors apologized for publishing it. The stated objections were about vocabulary, citations, and the author’s identity. But everyone understood the real offense was the conclusion: if we accept Caitlyn Jenner, the same logic should extend to Rachel Dolezal.

Ten years later, in a recent interview, Tuvel reflected on what she learned from this incident. She still cares about justice, she says. What changed is the order. “I think philosophy serves justice best by pursuing the truth about justice. And the pursuit of truth about justice requires exposure to a wide range of views on contested moral and political topics.” Get to the truth first and let justice follow. She describes the world she had been working in as an echo chamber, a place where dissenting views rarely reached her and certain conclusions were treated as settled.

This is what worries me about where social psychology is heading. What captured Tuvel’s corner of philosophy is creeping into my field: scholars who know the acceptable conclusion before the evidence is in. I felt the pull myself. Now I see more of it, as more academics refer to themselves as scholar-activists.

Don’t believe me? About five years ago, justice and truth were placed in direct conflict in my own corner of academia, and guess what happened?

In 2020, Bedoor AlShebli, Kinga Makovi, and Talal Rahwan published a large bibliometric study in Nature Communications finding that junior researchers mentored by women went on to be less successful in their careers. Twitter erupted. Within two days an open letter was circulating, hundreds of academics were calling the paper harmful to women in STEM, and the journal slapped a warning note on it. The stated objections were methodological: citations are a poor proxy for success, co-authorship a garbage proxy for mentorship. Fair enough. Except that two years earlier, some of the same authors had used the same bibliometric machinery, the same citation proxy, in the same journal, to show that ethnically diverse teams produce more impactful science. Nobody batted an eye at the method then. The method was fine when it flattered the cause and fatally flawed when it didn’t. Within weeks the authors retracted the female mentorship paper.

Now imagine a world where that paper stands and we ask why the citation gap exists. Is it discrimination, reviewers and citers undervaluing work that carries a woman’s name? Are women saddled with heavier teaching and service loads that starve their trainees of attention? Do women get less funding and smaller collaborative networks, so their labs produce less? Do women, on average, gravitate toward safer, more incremental questions, while men chase the riskier and more citable bets? Each answer implies a different fix, and several of those fixes would help the students of female mentors. That world is better for the very people the retraction claimed to protect. We chose the world where nobody gets to know.

I am not arguing for value-free science. Values can and do shape what we study. However, they cannot determine what you find. Let injustice draw your attention to a question; then let the evidence answer it, even when the answer is inconvenient, embarrassing, or useless to your side. I know how hard that is. The replication crisis forced me to give up findings I loved, findings that flattered everything I believed. Giving them up was the most useful thing I have ever done as a scientist.

I still want to make the world a better place. I just no longer believe a scientist does it by deciding first what the world should look like. The job is to find out what is true, follow it wherever it goes, and only then decide what to do about it. Fight for whatever cause you want once the finding is in. But if justice comes first, truth is what gets sacrificed.