Back in 2010, Joe Henrich, Steve Heine, and Ara Norenzayan dropped a bomb[1] on psychology with their paper "The WEIRDest people in the world?" They argued that most psychology studies rely on Western, Educated, Industrial, Rich, Democratic (WEIRD) populations.
Oh wow. I sometimes laugh to myself about the 2000-2010 psychology community believing in cute unintuitive phenomena that were later shown to not replicate. I figure that I'm good at applying common sense... but that cute WEIRD result on illusions totally fooled me. It was so cool (and Henrich is so smooth) that I honestly didn't even consider it wouldn't replicate
I feel this is a bit different... Not really that it doesn't replicate but is not quite so simple as conveyed in that article (although maybe failed replication is a fair gloss). The new article seems exhaustive and reveals the nuances that the paragraph in Henrich et al did not.
I’m quite sympathetic to Heinrich’s argument but I think this is a reasonable and well-argued perspective!
I suppose it might come down to whether our starting assumption should be that a given behavior or phenomenon generalizes or doesn’t. Ideally there would be a clear set of theoretical principles to determine this, as well as a good set of heuristics to navigate the trade offs. Lower level behaviors like visual perception seem like good candidates for “assume more likely to generalize” than something like ethical values, as you point out.
FWIW, there’s a very similar problem in research on LLMs right now, and my view is that generalization is too often assumed when what’s studied is a small sample of models trained on English data.
Another excellent post. I had not seen the Amir & FIrestone paper, but Firestone has done this type of thing before, debunking supposed motivational influences on basic perception. IDK all his stuff, but to have done this type of thing 2x in ten years is a helluvan accomplishment.
OTOH, most of what we study in most of the social sciences is higher level stuff, and, as you wrote, culture often matters quite a lot there,.
Insightful and informative article, Mickey. Another major issue I have with "WEIRD" is that, as a category, it's very inconsistent and hard to define...if not largely incoherent. How does one determine which populations fit that label and which don't? How many of the 5 WEIRD criteria does the population from which a sample is drawn have to meet in order to count as WEIRD?
For example, how is "rich" (R) determined? How about "industrialized?" If we use the UN "Human Development Index", the top countries are a mix of both "eastern" and "western" cultures, most of them highly educated, industrialized, and democratic. That pretty much cancels out the importance of the EIRD parts of WEIRD. And what determines eastern/western? Is it based merely on the geopolitical origin of that culture's mainstream political and philosophical thought, or are there more modern influences that should be considered? Take Japan, for example. Usually considered "eastern," but many aspects of its modern culture are indistinguishable from western culture (e.g., music, entertainment, capitalism, technology, fast food, etc.). It's also rich and democratic. So, is Japan a 4.5 out of 5.0 in terms of WEIRD? What about African nations? Culturally, most of them are not considered either "western" or "eastern."
My way-too-long, bloviating point is mostly that I strongly dislike the (over)application of WEIRD as a criticism of social/behavioural research. The purported bias problems that some researchers automatically assume that WEIRD samples have are, in many cases, far outweighed by the bias from the myriad conceptual and definitional problems inherent in the largely incoherent, overly broad WEIRD classification itself. Sure, it's an important consideration for studying certain phenomena, but the way it's often used as some sort of one-size-fits-all accusation of bias is intellectually lazy, imho.
Let's not throw out the baby with the bathwater when we can't even coherently explain why the water's bad in the first place.
Bright undergrad and grad students love iconoclastic papers. I remember delighting in a classic paper by Harry Harlow (“On mice, monkeys, men and motives”) in which he pointed out that the same psychologists who promulgated the behaviorist model (and its underlying tabula rasa assumptions) with near religious zeal went home to their children with their sometimes vastly different temperaments yet never interrogated the role biology might play in personality development. Rosenhan’s “On being sane in insane places” was another “gotcha” favorite. So the WEIRD paper was bound to be popular- and if the thesis was correct, it did have serious implications. But as always in science, the evidence is key, and you’ve done a lovely job here summarizing the evidence and making the case for a more nuanced conclusion. Great piece!
I think a lot about the role of culture in higher cognitive phenomena, especially in cases where poor performance is usually interpreted as a deficit. This argument by Henrich was compelling to me back in the day and I was also surprised/impressed by the new paper. But I think we should step back for a moment and think carefully about the implications: the argument was used to suggest if even basic psych phenomena are shaped by culture, than more complex psych phenomena must be. (Or is that just my idiosyncratic interpretation?) So since we now have less evidence to support the premise that basic psych phenomena are shaped by culture, we feel somewhat less persuaded that complex psych phenomena are necessarily shaped by culture. However, there is also direct evidence that complex psych phenomena are shaped by culture.
I am not seeing the strong implication that we should keep psychology WEIRD, but I agree with you that we need to be realistic and pursue non-WEIRD samples where it makes the most sense to do so (although it's tricky when we make assumptions about "where it makes sense..." and we might be wrong in thinking that the phenom does not vary by culture when it actually does). I wrote a paper on this that I can send you.
Thanks for engaging, Sam ;-)! My argument was more about cognitive psychology and lower-level processes than higher-level processes, for example in social psych. This is clearly shaped, at least to an extent, by culture. So while my fun bumper sticker implied we should keep all of psychology WEIRD, that mostly because the bumper sticker didn't look good with the "cognitive" qualifier. But hopefully the text clairifies my position enough.
I'm not surprised. Unfortunately I've mentioned that cross-cultural illusion finding numerous times, so if it's mistaken, then I've done people a disservice. I do worry that findings like these might mislead people into thinking we don't need cross-cultural research for more complex phenomena (like morality) even if this clearly isn't an implication and even if people commenting on this explicitly say this isn't the case.
This is really, really interesting. I suspect all these animals have frames of reference which lead to the same illusion, and that’s the commonality. Are you familiar with _How Minds Change_ by David McRaney? There’s a really fascinating experiment he references, involving kittens. They were raised initially without normal frames of reference, and it took them a little while to sort things out. You may already be well aware…
What about colour perception? I'm staying to suspect I've been tricked into thinking language influences that more than it does. Obviously trichromatic colour perception is a lot older than human language, but one could still imagine that the perception/discrimination is quite influenced by language...
Oh wow. I sometimes laugh to myself about the 2000-2010 psychology community believing in cute unintuitive phenomena that were later shown to not replicate. I figure that I'm good at applying common sense... but that cute WEIRD result on illusions totally fooled me. It was so cool (and Henrich is so smooth) that I honestly didn't even consider it wouldn't replicate
I feel this is a bit different... Not really that it doesn't replicate but is not quite so simple as conveyed in that article (although maybe failed replication is a fair gloss). The new article seems exhaustive and reveals the nuances that the paragraph in Henrich et al did not.
Same here.
I’m quite sympathetic to Heinrich’s argument but I think this is a reasonable and well-argued perspective!
I suppose it might come down to whether our starting assumption should be that a given behavior or phenomenon generalizes or doesn’t. Ideally there would be a clear set of theoretical principles to determine this, as well as a good set of heuristics to navigate the trade offs. Lower level behaviors like visual perception seem like good candidates for “assume more likely to generalize” than something like ethical values, as you point out.
FWIW, there’s a very similar problem in research on LLMs right now, and my view is that generalization is too often assumed when what’s studied is a small sample of models trained on English data.
We now need to pour resources into a ManyFish project that will include fish exposed to different cultural environments!!
I'm game!
Another excellent post. I had not seen the Amir & FIrestone paper, but Firestone has done this type of thing before, debunking supposed motivational influences on basic perception. IDK all his stuff, but to have done this type of thing 2x in ten years is a helluvan accomplishment.
OTOH, most of what we study in most of the social sciences is higher level stuff, and, as you wrote, culture often matters quite a lot there,.
Insightful and informative article, Mickey. Another major issue I have with "WEIRD" is that, as a category, it's very inconsistent and hard to define...if not largely incoherent. How does one determine which populations fit that label and which don't? How many of the 5 WEIRD criteria does the population from which a sample is drawn have to meet in order to count as WEIRD?
For example, how is "rich" (R) determined? How about "industrialized?" If we use the UN "Human Development Index", the top countries are a mix of both "eastern" and "western" cultures, most of them highly educated, industrialized, and democratic. That pretty much cancels out the importance of the EIRD parts of WEIRD. And what determines eastern/western? Is it based merely on the geopolitical origin of that culture's mainstream political and philosophical thought, or are there more modern influences that should be considered? Take Japan, for example. Usually considered "eastern," but many aspects of its modern culture are indistinguishable from western culture (e.g., music, entertainment, capitalism, technology, fast food, etc.). It's also rich and democratic. So, is Japan a 4.5 out of 5.0 in terms of WEIRD? What about African nations? Culturally, most of them are not considered either "western" or "eastern."
My way-too-long, bloviating point is mostly that I strongly dislike the (over)application of WEIRD as a criticism of social/behavioural research. The purported bias problems that some researchers automatically assume that WEIRD samples have are, in many cases, far outweighed by the bias from the myriad conceptual and definitional problems inherent in the largely incoherent, overly broad WEIRD classification itself. Sure, it's an important consideration for studying certain phenomena, but the way it's often used as some sort of one-size-fits-all accusation of bias is intellectually lazy, imho.
Let's not throw out the baby with the bathwater when we can't even coherently explain why the water's bad in the first place.
Well said
Bright undergrad and grad students love iconoclastic papers. I remember delighting in a classic paper by Harry Harlow (“On mice, monkeys, men and motives”) in which he pointed out that the same psychologists who promulgated the behaviorist model (and its underlying tabula rasa assumptions) with near religious zeal went home to their children with their sometimes vastly different temperaments yet never interrogated the role biology might play in personality development. Rosenhan’s “On being sane in insane places” was another “gotcha” favorite. So the WEIRD paper was bound to be popular- and if the thesis was correct, it did have serious implications. But as always in science, the evidence is key, and you’ve done a lovely job here summarizing the evidence and making the case for a more nuanced conclusion. Great piece!
I think a lot about the role of culture in higher cognitive phenomena, especially in cases where poor performance is usually interpreted as a deficit. This argument by Henrich was compelling to me back in the day and I was also surprised/impressed by the new paper. But I think we should step back for a moment and think carefully about the implications: the argument was used to suggest if even basic psych phenomena are shaped by culture, than more complex psych phenomena must be. (Or is that just my idiosyncratic interpretation?) So since we now have less evidence to support the premise that basic psych phenomena are shaped by culture, we feel somewhat less persuaded that complex psych phenomena are necessarily shaped by culture. However, there is also direct evidence that complex psych phenomena are shaped by culture.
I am not seeing the strong implication that we should keep psychology WEIRD, but I agree with you that we need to be realistic and pursue non-WEIRD samples where it makes the most sense to do so (although it's tricky when we make assumptions about "where it makes sense..." and we might be wrong in thinking that the phenom does not vary by culture when it actually does). I wrote a paper on this that I can send you.
Thanks for engaging, Sam ;-)! My argument was more about cognitive psychology and lower-level processes than higher-level processes, for example in social psych. This is clearly shaped, at least to an extent, by culture. So while my fun bumper sticker implied we should keep all of psychology WEIRD, that mostly because the bumper sticker didn't look good with the "cognitive" qualifier. But hopefully the text clairifies my position enough.
"Aaaargh, not again.........!" - Springs to mind.
Calmer than you are.
What are you, some park ranger now?
I'm not surprised. Unfortunately I've mentioned that cross-cultural illusion finding numerous times, so if it's mistaken, then I've done people a disservice. I do worry that findings like these might mislead people into thinking we don't need cross-cultural research for more complex phenomena (like morality) even if this clearly isn't an implication and even if people commenting on this explicitly say this isn't the case.
This is really, really interesting. I suspect all these animals have frames of reference which lead to the same illusion, and that’s the commonality. Are you familiar with _How Minds Change_ by David McRaney? There’s a really fascinating experiment he references, involving kittens. They were raised initially without normal frames of reference, and it took them a little while to sort things out. You may already be well aware…
Dude!!
What about colour perception? I'm staying to suspect I've been tricked into thinking language influences that more than it does. Obviously trichromatic colour perception is a lot older than human language, but one could still imagine that the perception/discrimination is quite influenced by language...