Since you mentioned "sample sizes going up", if you are curious, I recently collected some stats on this as part of a paper that is in press... you can see this graph here (https://imgur.com/a/XDFflW9) of my estimates for the typical sample size in each field. These numbers are based on scanning ~250k papers for t-values, taking the degrees of freedom associated with each t-value and just adding +1. These numbers reflect the median sample size of the median paper. Social psychology has gone up the most among the different psychology subfields (from N = 70 in 2004 to N = 250 people today)
Very good essay! As a final year PhD student and a lecturer in research methods who is, as you say, "grappling with questions that threatened to unravel everything I thought I knew about my field", I have decided to start blogging on Substack to practice writing and clarify my thoughts, but I do fear the costs that may come when my colleagues/college discovers this. I already suspect I don't get opportunities and invitations to collaborate because of some of the perspectives I have shared (and opportunities like that are already few down here in New Zealand). So I don't know about my future in the field.
I hear you. I hope you don’t give up before trying, though. I was at SPSP last weekend, and a student just casually mentioned the replication crisis, with most stuff needing to be reevaluated, and no one batted an eye. This, despite there being a few senior people in the audience who, a decade earlier, had a public fit when replicability issues were first mentioned. So we’ve come along way, and your opinions might resonate with more people than you think.
This was great, and I appreciated how you described the limitations of experiments. I am an associate professor in ecology. I think our field is years behind social psychology in terms of coming to grips with the issues you have outlined here and in other writings/podcasts. However, we don't have something like priming studies as a primary culprit. In ecology, there has been a big move away from theory towards more applied work. However, the applied work in ecology is so context-dependent and liable to researcher degrees of freedom that I am not sure how informative it is. Your writing about descriptive work is interesting as there is a long history of this in ecology (as noted by Darwin). But it is difficult to justify spending millions of dollars describing esoteric natural history (as much as I love that work.) There are also an infinite number of phenomena to describe and I am not sure how to go about that descriptive work without some theory to direct it.
I am rambling, but I have been a big fan of yours and listened to every episode of 2 Psychologists 4 Beers. I wanted to you know that your work and ideas are reaching beyond psychology.
I find it fascinating how much pushback there still is in social psych towards those who are critical of the field's history and current state. Yes social psychology has made progress, but from my view (which is largely that of an outsider) I still see a dearth of proper theory. And the theories which do exist rarely extend beyond the bounds of arrows, boxes, and circles. While open science solved some issues, I still think the bigger issue for social psych (and psychological research at large) is the development of robust theories with clear empirical predictions rather than the reliance on vague analogies. All theories needn't be fancy maths, but in general there's seemingly an aversion to anything 'computational' in social psych (with obvious exceptions!)
I generally agree with you, Jake. But I want to push back a bit too. I hear this a lot: our problem is bad theory. But I wonder if that's true. Yes, our theories are terrible, but is that the reason we are not making sufficient progress? I'm not so sure any more; I'm not sure our field is even ready for good theory. I would be satisfied right now with reliable observations that are true not only in a lab but in the real world. And once we have way more of these reliable observations, then maybe we'll be ready for good theory. These thoughts are still ill-formed, but maybe I'll write about it one day.
Yes, I have recently changed my mind on the "psychology needs more theory" movement. Maybe if you do strict formal theorising in the manner that highly advanced scientists like Boorsboom, then it might be worthwhile. But really, in the incredibly messy domain of human emotion and behaviour, in which everything correlates with everything else, and there are unbelievable methodological issues and researcher degrees of freedom, all theorising does is allow researchers to build in their own biases and ideology into theories for human behaviour which can basically be made to comport with any and all evidence. Crucially, scientific theories (especially in social science) are underdetermined, which, again, means any number of theories can be made to fit the data and make accurate predictions.
Combine underdetermination with the major issues in psych research (which I review here: https://backcountrypsych.substack.com/p/reign-of-error-the-failure-of-social), and you basically give an ideologically committed field the ability to construct propagandistic theories which are used to advance a particular political/ideological project rather than advance science and knowledge. System justification theory is one of these.
Not that I need more to do but ... we should do a paper, a review. Themed, maybe titled, "Skeptics' Views of Valid, Important Discoveries in Social Psych over the Last 50 Years." Here is a draft outline.
Intro I: Replication crisis and long list of handwringing/validity crises.
Intro II: What does it mean to be a scientific skeptic (Lilienfeld has written on this; Mayo's severe testing as standards; I've written on what it takes to establish some research-based claim as a new fact -- it takes a lot). Lay out the standards we skeptics require for giving something credibility. (This would be a nice yet actionable philosophy of sci section).
Meat of Paper
And yet, even we, skeptics tho we may be, believe that social psych and its adjacent fields have discovered lots of things.
Paul Bloom has a list on his Substack, as do I. Not saying either list is comprehensive. We'd have to agree of course. And, if we do it, we might want to take on a few more skeptics.
Conclusion
Proponents of various ideas will soon start screeching that we left them out. But proponents do not count because this paper is a review of phenomena/findings that even the skeptics give credence to.
We may have missed stuff. Scientists are imperfect; so are we; get over it. But if you want to propose something that would convince us, you need to meet the standards described in Intro II. If you think you can do it, go for it.
Social Psych has a glorious yet checkered history -- kinda like humans in general. Nonetheless, our view is that it actually has made important disoveries, many of which can be used by people to make the world a better place.
-----------
How's that for an uncharacteristically benevolent and optimistic tone? My racist mule was feeling spunky this morning. Be careful tho -- if we do this, you'd be racist-mule-adjacent. You know I've quoted you in a pub? Here's the pub:
https://www.mdpi.com/2075-4698/10/4/82 (on academic radicalization, outrage mobs, demonization and socially distributed censorship/suppression), long before I hooked up with my racist mule. Anyway, here is the section quoting you:
This dynamic (self-censorship) was captured beautifully in a podcast by social psychologists Michael Inzlicht and Yoel Inbar (2018). Inzlicht, at about 25 minutes in:
“What if I felt that overemphasis on oppression is a terrible idea, hurts alleged victims of oppression, and is bad for everyone? What if I was outspoken about this? I suspect I would face a lot more opposition. Even though not much could happen to my job security, I’d have a lot of people screaming at me, making my life uncomfortable. And, truly, I wouldn’t do it, because I’d be scared. I wouldn’t do it because I’m a coward.”
My racist mule insisted that I remind you of all this before you agree to anything.
"In this essay, I explain why, if the only thing you know is that something is published in a psychology peer reviewed journal, or book or book chapter, or presented at a conference, you should simply disbelieve it, pending confirmation by multiple independent researchers in the future."
That final phrase is the bulwark against skepticism going full cynic.
Here are the key ideas:
Equation 1:
%False Claims = (%Unreplicable Findings) + (%References to Unreplicable Findings as If They are True) + (%False Claims Based on Replicable Findings) + (%False Claims Based on Ignoring Contrary Evidence) + (%False Claims Due to Censorship) + (%False Claims Based on Making Shit Up Completely)
The replication run rate in social is below 50%. And it goes downhill from there.
Since you mentioned "sample sizes going up", if you are curious, I recently collected some stats on this as part of a paper that is in press... you can see this graph here (https://imgur.com/a/XDFflW9) of my estimates for the typical sample size in each field. These numbers are based on scanning ~250k papers for t-values, taking the degrees of freedom associated with each t-value and just adding +1. These numbers reflect the median sample size of the median paper. Social psychology has gone up the most among the different psychology subfields (from N = 70 in 2004 to N = 250 people today)
Wow! Huge change indeed!
Very good essay! As a final year PhD student and a lecturer in research methods who is, as you say, "grappling with questions that threatened to unravel everything I thought I knew about my field", I have decided to start blogging on Substack to practice writing and clarify my thoughts, but I do fear the costs that may come when my colleagues/college discovers this. I already suspect I don't get opportunities and invitations to collaborate because of some of the perspectives I have shared (and opportunities like that are already few down here in New Zealand). So I don't know about my future in the field.
I hear you. I hope you don’t give up before trying, though. I was at SPSP last weekend, and a student just casually mentioned the replication crisis, with most stuff needing to be reevaluated, and no one batted an eye. This, despite there being a few senior people in the audience who, a decade earlier, had a public fit when replicability issues were first mentioned. So we’ve come along way, and your opinions might resonate with more people than you think.
This was great, and I appreciated how you described the limitations of experiments. I am an associate professor in ecology. I think our field is years behind social psychology in terms of coming to grips with the issues you have outlined here and in other writings/podcasts. However, we don't have something like priming studies as a primary culprit. In ecology, there has been a big move away from theory towards more applied work. However, the applied work in ecology is so context-dependent and liable to researcher degrees of freedom that I am not sure how informative it is. Your writing about descriptive work is interesting as there is a long history of this in ecology (as noted by Darwin). But it is difficult to justify spending millions of dollars describing esoteric natural history (as much as I love that work.) There are also an infinite number of phenomena to describe and I am not sure how to go about that descriptive work without some theory to direct it.
I am rambling, but I have been a big fan of yours and listened to every episode of 2 Psychologists 4 Beers. I wanted to you know that your work and ideas are reaching beyond psychology.
Thank you!
I find it fascinating how much pushback there still is in social psych towards those who are critical of the field's history and current state. Yes social psychology has made progress, but from my view (which is largely that of an outsider) I still see a dearth of proper theory. And the theories which do exist rarely extend beyond the bounds of arrows, boxes, and circles. While open science solved some issues, I still think the bigger issue for social psych (and psychological research at large) is the development of robust theories with clear empirical predictions rather than the reliance on vague analogies. All theories needn't be fancy maths, but in general there's seemingly an aversion to anything 'computational' in social psych (with obvious exceptions!)
I generally agree with you, Jake. But I want to push back a bit too. I hear this a lot: our problem is bad theory. But I wonder if that's true. Yes, our theories are terrible, but is that the reason we are not making sufficient progress? I'm not so sure any more; I'm not sure our field is even ready for good theory. I would be satisfied right now with reliable observations that are true not only in a lab but in the real world. And once we have way more of these reliable observations, then maybe we'll be ready for good theory. These thoughts are still ill-formed, but maybe I'll write about it one day.
Yes, I have recently changed my mind on the "psychology needs more theory" movement. Maybe if you do strict formal theorising in the manner that highly advanced scientists like Boorsboom, then it might be worthwhile. But really, in the incredibly messy domain of human emotion and behaviour, in which everything correlates with everything else, and there are unbelievable methodological issues and researcher degrees of freedom, all theorising does is allow researchers to build in their own biases and ideology into theories for human behaviour which can basically be made to comport with any and all evidence. Crucially, scientific theories (especially in social science) are underdetermined, which, again, means any number of theories can be made to fit the data and make accurate predictions.
Combine underdetermination with the major issues in psych research (which I review here: https://backcountrypsych.substack.com/p/reign-of-error-the-failure-of-social), and you basically give an ideologically committed field the ability to construct propagandistic theories which are used to advance a particular political/ideological project rather than advance science and knowledge. System justification theory is one of these.
Not that I need more to do but ... we should do a paper, a review. Themed, maybe titled, "Skeptics' Views of Valid, Important Discoveries in Social Psych over the Last 50 Years." Here is a draft outline.
Intro I: Replication crisis and long list of handwringing/validity crises.
Intro II: What does it mean to be a scientific skeptic (Lilienfeld has written on this; Mayo's severe testing as standards; I've written on what it takes to establish some research-based claim as a new fact -- it takes a lot). Lay out the standards we skeptics require for giving something credibility. (This would be a nice yet actionable philosophy of sci section).
Meat of Paper
And yet, even we, skeptics tho we may be, believe that social psych and its adjacent fields have discovered lots of things.
Paul Bloom has a list on his Substack, as do I. Not saying either list is comprehensive. We'd have to agree of course. And, if we do it, we might want to take on a few more skeptics.
Conclusion
Proponents of various ideas will soon start screeching that we left them out. But proponents do not count because this paper is a review of phenomena/findings that even the skeptics give credence to.
We may have missed stuff. Scientists are imperfect; so are we; get over it. But if you want to propose something that would convince us, you need to meet the standards described in Intro II. If you think you can do it, go for it.
Social Psych has a glorious yet checkered history -- kinda like humans in general. Nonetheless, our view is that it actually has made important disoveries, many of which can be used by people to make the world a better place.
-----------
How's that for an uncharacteristically benevolent and optimistic tone? My racist mule was feeling spunky this morning. Be careful tho -- if we do this, you'd be racist-mule-adjacent. You know I've quoted you in a pub? Here's the pub:
https://www.mdpi.com/2075-4698/10/4/82 (on academic radicalization, outrage mobs, demonization and socially distributed censorship/suppression), long before I hooked up with my racist mule. Anyway, here is the section quoting you:
This dynamic (self-censorship) was captured beautifully in a podcast by social psychologists Michael Inzlicht and Yoel Inbar (2018). Inzlicht, at about 25 minutes in:
“What if I felt that overemphasis on oppression is a terrible idea, hurts alleged victims of oppression, and is bad for everyone? What if I was outspoken about this? I suspect I would face a lot more opposition. Even though not much could happen to my job security, I’d have a lot of people screaming at me, making my life uncomfortable. And, truly, I wouldn’t do it, because I’d be scared. I wouldn’t do it because I’m a coward.”
My racist mule insisted that I remind you of all this before you agree to anything.
Nice post, again. Skepticism is entirely warranted. ICYMI:
~75% of Psychology Claims are False
https://unsafescience.substack.com/p/75-of-psychology-claims-are-false
"In this essay, I explain why, if the only thing you know is that something is published in a psychology peer reviewed journal, or book or book chapter, or presented at a conference, you should simply disbelieve it, pending confirmation by multiple independent researchers in the future."
That final phrase is the bulwark against skepticism going full cynic.
Here are the key ideas:
Equation 1:
%False Claims = (%Unreplicable Findings) + (%References to Unreplicable Findings as If They are True) + (%False Claims Based on Replicable Findings) + (%False Claims Based on Ignoring Contrary Evidence) + (%False Claims Due to Censorship) + (%False Claims Based on Making Shit Up Completely)
The replication run rate in social is below 50%. And it goes downhill from there.
P.S. I heard my racist mule made an appearance at SPSP. https://x.com/saltypsych/status/1893853141885403575