An important consideration when looking at psychology studies.

PZ Meyers recently posted about a paper "The Weirdest People in the World" (PDF) by Joseph Henrich, Steven J, Heine, and Ara Norenzayen. In this paper the authors warn about avoiding hasty generalizations when looking at the results of psychological research.The main thesis of the paper is that there is a serious problem regarding subject samples in contemporary psychology research. In particular, too much of that research is focused on a group the authors call the WEIRD (Western, Educated, Industrialized, Rich and Democratic).  Research is done on this WEIRD group and is then erroneously generalized to cover the entire human population.

Who are the people studied in behavioral science research? A recent analysis of the top journals in six sub‐disciplines of Psychology from 2003‐2007 revealed that 68% of subjects came from the US, and a full 96% of subjects were from Western industrialized countries, specifically North America, Europe, Australia, and Israel (Arnett 2008). The make‐up of these samples appears to largely reflect the country of residence of the authors, as 73% of first authors were at American universities, and 99% were at universities in Western countries. This means that 96% of psychological samples come from countries with only 12% of the world’s population. Put another way, a randomly selected American is 300 times more likely to be a research participant in a study in one of these journals than is a randomly selected person from outside of the West.

The authors go on to note that the problem is even worse in that "67% of the American samples (and 80% of the samples from other countries) were composed solely of undergraduates in psychology courses." This means that an enormous amount of psychological research is apparently biased in favor undergraduate students at American universities. But, this need not be a problem if these results are generalizable. However, the authors look at a number of important cross-cultural surveys and find that many of the results for American undergraduates cannot be generalized to the rest of the world.

The most striking example of this (in my opinion) is the discussion of the Meuller-Lyer Illusion. 

The idea here is that line segments (a) and (b) are identical in length, and the question then is how much bigger does line segment (a) need to be in order to look equal in length to line segment (b). When done on American undergraduates, line segment (a) needs to be about 1/5 as long to look equal to line segment (b), but in other populations the amount varies. This chart nicely summarizes the results:

 

As the chart indicates, the American undergraduates (from Evanston, all the way on the right) required segment (a) to be the longest of any of the groups surveyed while the illusion didn't even work for the San people of the Kalahari. This means that American undergraduates are anything but a representative sample of human's generally, and research that is based solely on them will be seriously in error.

The major take-away from all this is that we need to be careful when drawing generalizations. We need to ensure that our sample size is robust enough and unbiased enough that we can legitimately generalize to the population as a whole. This also means those of us living in America need to take extra care to remember that most people in the world do not live like us, and that our experiences and beliefs cannot be generalized to the whole world. As this study indicates, because we are WEIRD we are weird.