For a project I'm working on, I have a dataset of psychology papers. I just looked, and in the dataset, I have 763 sentences reporting a significant result that contain the word "depletion". Among those sentences, 41% of results report a p-value of .01 < p < .05 (i.e., just barely falling under the significance threshold). I figure that you're familiar with p-curve stuff... for reference, a study with 80% power will produce 26% of its p-values in that range, and the mean study over the past decade across all of psychology is about at 29%. The "depletion" percentage puts it very close to the rate associated with the word "priming" (rate of 40% across 3820 sentences)
I'm sorry for your tough time over this (I am a retired research psychologist myself), but really appreciate your putting it out here for everyone to see. This is indeed a shocking example for everyone's attention (!).
Thanks for this article—really helped me understand the issues with ego depletion! I got the sense that researchers moved past this debate years ago, but the narrative of decision fatigue still seems strong (e.g., the Zuckerberg/Jobs outfit choices). Is decision fatigue fundamentally different, or is it just a rebranded version of the same idea? I’m studying for a master’s in applied behavioral science and still getting familiar with discussions on decision-making and self-control, so I’d love to hear your thoughts!
Decision fatigue is the same as ego depletion. Just old wine in a new bottle. I once tried to track the etymology of the decision fatigue term, and the best I could figure it is was a term coined by the science journalist John Tierney, when describing...you guessed it...eho depletion. He later went on to co-author a book with Baumeister on depletion.
1. ego depletions. how do we explain those 600 studies? I'm cool with assuming 60% are fake (p-hacking or worse). but if 40% were legitimate in terms of reach individual study. reasonable prior? "file drawer"? wouldn't suffice. because we will need 19*240=4560 studies to have been left hidden. and more of those have 0.01 etc.
I can see that you came from the sophisticated formulas inferring direction. but what's the explanation?
2. from reading the fine print in various "failed replication" studies, quite some of them were faulty in various obvious ways.
when arguing online, it was painfully obvious that most people simply trust those studies with similar superficiality that they trusted every study 15 years ago. not generalising. but lots of fake "failed replications". I'm not referring to borderline cases!
Yes, I published a paper estimating how many file drawered studies would need to be out there and that seems unlikely. Yet, if there was a signal, why can't we find it reliably. As for your second point, we'll need to disagree on that one. I do not see any fake replications, just imperfect attempted to ferret out the truth.
can't heterogeneity (well known before the replicated crisis, for me at least!) explain a lot of it? and if you accept some of Roy's issues with various replications, the question is less jarring.
"fake" was somewhat strong. eg the first famous "failed replication" of old age priming (with Bargh blog etc) missed a basic detail which I have known about from a popular book. and everyone rush to attack Bargh on three legitimate claims in this blog made me feel untrusting. multiple similar stories.
lots more to hash out here about those grievances of my. I live in Thailand... 3am .......
I felt that lots of replication supporters mix up "precise replication" with robustness checks, conceptual replication, etc
the more worrisome is the derision towards replication critics, where every counter argument has been recieved as if it's a mafia suspect giving lawyer concocted excuses, rather than genuine concerns about details.
In 1-2 cases I corresponded with an rrr coordinator, and got a semi dismissive "yeah, precise replications aren't possible. we try our best" which felt too uncaring rather than "we actually try to maximise precision"
it saddened me, because I'm a skeptic in heart, and I never trusted single studies at all. I feel the replication movement is a huge improvement. but often it's as unreliable as any published study.
the anti replication camp did themselves no favours at times, of course.
but how can I even treat with respect or trust people who read Roy's paper (2017?) that basically said "there are trade-offs. if you promote students that are good at replications, you'll lose other useful positive researcher traits", and went "haha this guy is an idiot". this was so depressing to witness.
Thanks for being so honest about all this. How do you account for Baumeister's intransigence? Do you think most of his other work (e.g on self) holds up, or is much of it tied to ego depletion assumptions and thus questionable? For some reason I'm fascinated by his particular story because at least some of his stuff has struck me as more philosophical and thoughtful, and more out of the box, than most mainstream social psych.
He's covered such a wide range of topics - why die on this hill?
I think his work on ego depletion is separate from the rest and don't want to cast aspersions on other areas. But pre-replication crisis work should be treated with caution and skepticism, and I put my own work in that category too.
Any good sources that summarize which findings have successfully replicated and which key theories have been validated in the post-replication crisis era?
You mean of Baumeister or in general? I am not familiar with any taxonomy for Baumeister or the rest of the field. Though, many of us are familiar with stuff that we know this is sus: terror management theory, "social priming", stereotype threat, but others that we're suspicious of but have not been rigorously vetted, like construal level theory.
What if apart from self-control, some observations can be attributed to working memory being limited? As a person with usually healthy self-control (I've been vegan for over 10 years, with a good academic record etc.), I still tend to forget intentions that I wanted to follow (e.g. "I will not eat chocolate today", 5 hours later: oops, ate chocolate, forgot my intention from this morning). This can easily get confused with a lack of willpower, but in my experience, it's different.
Couldn't there be a general confusion between what people attribute to willpower, and working memory? I believe that working memory is indeed limited - the more things I try not to forget on a given day, the less likely it is that I will actually keep up with all of them. Maybe this can explain at least part of why the theory seems so appealing at first sight. There could be more factors than fatigue explaining phenomena that would traditionally be attributed to "ego-depletion".
Excellent comments, not naive at all. I agree with you that there is something about the theory that feels right, even beyond fatigue. You point to working memory; I have pointed to shifting priorities and motivations. I think all of this is possible. But...and this is the important part...if we want to examine these ideas scientifically, we need a way to SEE these effects empirically, in a lab. Once we can do this--assuming people have the inclination--then we can start digging deeper to understand it. Right now, though, we're in a place where we cannot even see a basic effect; we cannot reliably produce it in a lab. And without that, we're left with wild speculations.
To be fair, shouldn't there be productive tension around what exactly counts as fatigue, and consequently (and overly simply), what causes and is caused by it?
For example, is fatigue caused by any one or many systems being "overworked" relative to its maximum and extended capacities over time? Working memory would seem like a prime culprit on account of its low capacity.
What fatigue might then "cause" might then diminish the capacity any given system or sets of systems that may play a role in compensation. However, this then becomes an immediate tangle from seemingly simple parts. For example, when feeling ambivalence or cognitive dissonance, are they upstream, downstream, or independent of fatigue?
Absolutely! Still lots of work to be done in the fatigue space. Matthew Apps of U of Birmingham, in my opinion, is doing some of the best work in this space. So, yes to more work on fatigue! But, please let's not do that work, call it ego depletion, and use it to vindicate the undead depletion literature.
Couldn't ego depletion be downgraded to a potentially useful heuristic in the sense of being strictly untrue, but not proven to be maladaptive as a rule of thumb?
And might the height of the fall, from celebrated researcher to publicly undifferentiated from some generic fraudster, contribute to an understandable defensiveness and empirical hedging? Are there no "soft landings" in "soft sciences?"
I know it would be easier for me if I could say "oops, I was wrong. Let's see if I was at least helpful."
Oh I understand the defensiveness and wonder if I would act any differently if the positions were reversed. I think it is a very human response.
But the idea of saying this is a useful heuristic is not what I'm in the business of--ferreting out the truth. Depletion, as originally conceived, and tested is simply not true. The much broader concept of fatigue, though--but please know there are important differences in how depletion was originally formulated and fatigue, which is a broadband effect not limited to self-control.
Going down the wrong rabbit hole and freely admitting it is an enormous gift to science. Without such heroes we would see bogus “theories” perpetuate even longer, wasting incalculable time, energy and money, and polluting clinical practice in the meantime.
This is why even if we end up wrong, as long as we openly admit that we can still be grateful we’ve played an important part in achieving progress, saving many others from the red herring we were once so keen on and freeing them up to discover new pathways.
For a project I'm working on, I have a dataset of psychology papers. I just looked, and in the dataset, I have 763 sentences reporting a significant result that contain the word "depletion". Among those sentences, 41% of results report a p-value of .01 < p < .05 (i.e., just barely falling under the significance threshold). I figure that you're familiar with p-curve stuff... for reference, a study with 80% power will produce 26% of its p-values in that range, and the mean study over the past decade across all of psychology is about at 29%. The "depletion" percentage puts it very close to the rate associated with the word "priming" (rate of 40% across 3820 sentences)
"New information about meta-analysis came to light, man"
"Say what you will about the tenets of ego depletion theory, at least it had an ethos."
🙂
Way to go, Pauly! Now, mark it zero cuz there are rules and I'm not playing that game today. (Also, there are other references in there, sir).
"White russian"
ding ding ding. Another urban achiever I see. What do you think about the Eagles?
I hate the fucking eagles, man
You said it, man.
I'm sorry for your tough time over this (I am a retired research psychologist myself), but really appreciate your putting it out here for everyone to see. This is indeed a shocking example for everyone's attention (!).
Ha! That’s an illustration from AoM when went all in on ego depletion over a decade ago.
I wondred where that illustration came from...it's been floating around for years.
Thanks for this article—really helped me understand the issues with ego depletion! I got the sense that researchers moved past this debate years ago, but the narrative of decision fatigue still seems strong (e.g., the Zuckerberg/Jobs outfit choices). Is decision fatigue fundamentally different, or is it just a rebranded version of the same idea? I’m studying for a master’s in applied behavioral science and still getting familiar with discussions on decision-making and self-control, so I’d love to hear your thoughts!
Decision fatigue is the same as ego depletion. Just old wine in a new bottle. I once tried to track the etymology of the decision fatigue term, and the best I could figure it is was a term coined by the science journalist John Tierney, when describing...you guessed it...eho depletion. He later went on to co-author a book with Baumeister on depletion.
two points.
1. ego depletions. how do we explain those 600 studies? I'm cool with assuming 60% are fake (p-hacking or worse). but if 40% were legitimate in terms of reach individual study. reasonable prior? "file drawer"? wouldn't suffice. because we will need 19*240=4560 studies to have been left hidden. and more of those have 0.01 etc.
I can see that you came from the sophisticated formulas inferring direction. but what's the explanation?
2. from reading the fine print in various "failed replication" studies, quite some of them were faulty in various obvious ways.
when arguing online, it was painfully obvious that most people simply trust those studies with similar superficiality that they trusted every study 15 years ago. not generalising. but lots of fake "failed replications". I'm not referring to borderline cases!
Yes, I published a paper estimating how many file drawered studies would need to be out there and that seems unlikely. Yet, if there was a signal, why can't we find it reliably. As for your second point, we'll need to disagree on that one. I do not see any fake replications, just imperfect attempted to ferret out the truth.
can't heterogeneity (well known before the replicated crisis, for me at least!) explain a lot of it? and if you accept some of Roy's issues with various replications, the question is less jarring.
"fake" was somewhat strong. eg the first famous "failed replication" of old age priming (with Bargh blog etc) missed a basic detail which I have known about from a popular book. and everyone rush to attack Bargh on three legitimate claims in this blog made me feel untrusting. multiple similar stories.
lots more to hash out here about those grievances of my. I live in Thailand... 3am .......
I felt that lots of replication supporters mix up "precise replication" with robustness checks, conceptual replication, etc
the more worrisome is the derision towards replication critics, where every counter argument has been recieved as if it's a mafia suspect giving lawyer concocted excuses, rather than genuine concerns about details.
In 1-2 cases I corresponded with an rrr coordinator, and got a semi dismissive "yeah, precise replications aren't possible. we try our best" which felt too uncaring rather than "we actually try to maximise precision"
it saddened me, because I'm a skeptic in heart, and I never trusted single studies at all. I feel the replication movement is a huge improvement. but often it's as unreliable as any published study.
the anti replication camp did themselves no favours at times, of course.
but how can I even treat with respect or trust people who read Roy's paper (2017?) that basically said "there are trade-offs. if you promote students that are good at replications, you'll lose other useful positive researcher traits", and went "haha this guy is an idiot". this was so depressing to witness.
I hope you understand me feeling here....
Thanks for being so honest about all this. How do you account for Baumeister's intransigence? Do you think most of his other work (e.g on self) holds up, or is much of it tied to ego depletion assumptions and thus questionable? For some reason I'm fascinated by his particular story because at least some of his stuff has struck me as more philosophical and thoughtful, and more out of the box, than most mainstream social psych.
He's covered such a wide range of topics - why die on this hill?
I think his work on ego depletion is separate from the rest and don't want to cast aspersions on other areas. But pre-replication crisis work should be treated with caution and skepticism, and I put my own work in that category too.
Any good sources that summarize which findings have successfully replicated and which key theories have been validated in the post-replication crisis era?
You mean of Baumeister or in general? I am not familiar with any taxonomy for Baumeister or the rest of the field. Though, many of us are familiar with stuff that we know this is sus: terror management theory, "social priming", stereotype threat, but others that we're suspicious of but have not been rigorously vetted, like construal level theory.
I meant in general. Those were good examples. Thanks!
I'll make some naive comments, hope that's ok.
What if apart from self-control, some observations can be attributed to working memory being limited? As a person with usually healthy self-control (I've been vegan for over 10 years, with a good academic record etc.), I still tend to forget intentions that I wanted to follow (e.g. "I will not eat chocolate today", 5 hours later: oops, ate chocolate, forgot my intention from this morning). This can easily get confused with a lack of willpower, but in my experience, it's different.
Couldn't there be a general confusion between what people attribute to willpower, and working memory? I believe that working memory is indeed limited - the more things I try not to forget on a given day, the less likely it is that I will actually keep up with all of them. Maybe this can explain at least part of why the theory seems so appealing at first sight. There could be more factors than fatigue explaining phenomena that would traditionally be attributed to "ego-depletion".
Excellent comments, not naive at all. I agree with you that there is something about the theory that feels right, even beyond fatigue. You point to working memory; I have pointed to shifting priorities and motivations. I think all of this is possible. But...and this is the important part...if we want to examine these ideas scientifically, we need a way to SEE these effects empirically, in a lab. Once we can do this--assuming people have the inclination--then we can start digging deeper to understand it. Right now, though, we're in a place where we cannot even see a basic effect; we cannot reliably produce it in a lab. And without that, we're left with wild speculations.
To be fair, shouldn't there be productive tension around what exactly counts as fatigue, and consequently (and overly simply), what causes and is caused by it?
For example, is fatigue caused by any one or many systems being "overworked" relative to its maximum and extended capacities over time? Working memory would seem like a prime culprit on account of its low capacity.
What fatigue might then "cause" might then diminish the capacity any given system or sets of systems that may play a role in compensation. However, this then becomes an immediate tangle from seemingly simple parts. For example, when feeling ambivalence or cognitive dissonance, are they upstream, downstream, or independent of fatigue?
Absolutely! Still lots of work to be done in the fatigue space. Matthew Apps of U of Birmingham, in my opinion, is doing some of the best work in this space. So, yes to more work on fatigue! But, please let's not do that work, call it ego depletion, and use it to vindicate the undead depletion literature.
Couldn't ego depletion be downgraded to a potentially useful heuristic in the sense of being strictly untrue, but not proven to be maladaptive as a rule of thumb?
And might the height of the fall, from celebrated researcher to publicly undifferentiated from some generic fraudster, contribute to an understandable defensiveness and empirical hedging? Are there no "soft landings" in "soft sciences?"
I know it would be easier for me if I could say "oops, I was wrong. Let's see if I was at least helpful."
Oh I understand the defensiveness and wonder if I would act any differently if the positions were reversed. I think it is a very human response.
But the idea of saying this is a useful heuristic is not what I'm in the business of--ferreting out the truth. Depletion, as originally conceived, and tested is simply not true. The much broader concept of fatigue, though--but please know there are important differences in how depletion was originally formulated and fatigue, which is a broadband effect not limited to self-control.
Going down the wrong rabbit hole and freely admitting it is an enormous gift to science. Without such heroes we would see bogus “theories” perpetuate even longer, wasting incalculable time, energy and money, and polluting clinical practice in the meantime.
This is why even if we end up wrong, as long as we openly admit that we can still be grateful we’ve played an important part in achieving progress, saving many others from the red herring we were once so keen on and freeing them up to discover new pathways.