An important consideration when looking at psychology studies.

PZ Meyers recently posted about a paper "The Weirdest People in the World" (PDF) by Joseph Henrich, Steven J, Heine, and Ara Norenzayen. In this paper the authors warn about avoiding hasty generalizations when looking at the results of psychological research.The main thesis of the paper is that there is a serious problem regarding subject samples in contemporary psychology research. In particular, too much of that research is focused on a group the authors call the WEIRD (Western, Educated, Industrialized, Rich and Democratic).  Research is done on this WEIRD group and is then erroneously generalized to cover the entire human population.

Who are the people studied in behavioral science research? A recent analysis of the top journals in six sub‐disciplines of Psychology from 2003‐2007 revealed that 68% of subjects came from the US, and a full 96% of subjects were from Western industrialized countries, specifically North America, Europe, Australia, and Israel (Arnett 2008). The make‐up of these samples appears to largely reflect the country of residence of the authors, as 73% of first authors were at American universities, and 99% were at universities in Western countries. This means that 96% of psychological samples come from countries with only 12% of the world’s population. Put another way, a randomly selected American is 300 times more likely to be a research participant in a study in one of these journals than is a randomly selected person from outside of the West.

The authors go on to note that the problem is even worse in that "67% of the American samples (and 80% of the samples from other countries) were composed solely of undergraduates in psychology courses." This means that an enormous amount of psychological research is apparently biased in favor undergraduate students at American universities. But, this need not be a problem if these results are generalizable. However, the authors look at a number of important cross-cultural surveys and find that many of the results for American undergraduates cannot be generalized to the rest of the world.

The most striking example of this (in my opinion) is the discussion of the Meuller-Lyer Illusion. 

The idea here is that line segments (a) and (b) are identical in length, and the question then is how much bigger does line segment (a) need to be in order to look equal in length to line segment (b). When done on American undergraduates, line segment (a) needs to be about 1/5 as long to look equal to line segment (b), but in other populations the amount varies. This chart nicely summarizes the results:

 

As the chart indicates, the American undergraduates (from Evanston, all the way on the right) required segment (a) to be the longest of any of the groups surveyed while the illusion didn't even work for the San people of the Kalahari. This means that American undergraduates are anything but a representative sample of human's generally, and research that is based solely on them will be seriously in error.

The major take-away from all this is that we need to be careful when drawing generalizations. We need to ensure that our sample size is robust enough and unbiased enough that we can legitimately generalize to the population as a whole. This also means those of us living in America need to take extra care to remember that most people in the world do not live like us, and that our experiences and beliefs cannot be generalized to the whole world. As this study indicates, because we are WEIRD we are weird.

It's that time of the month: Hasty Generalizations and Post Hoc reasoning.

A recent press release from Concordia University in Montreal has announced the publication of a paper by lead author Gad Saad linking shopping habits among women to their menstrual cycle. In particular, the study seemed to indicate that women were more likely to buy clothing and beauty related products and spend more time on appearance related activities (manicures, tanning salons, sun-bathing etc.) during the fertile phase of their menstrual cycle, and more likely to consume high-caloric foods during the infertile phase of the menstrual cycle. Of particular interest for me is the linking of these behaviors to evolution, with Saad essentially arguing that women are biologically determined to engage in these sorts of behaviors.

"In ancestral times,” he explains, “women had to focus more time on mating-related activities during the fertile phase of the menstrual cycle, when the likelihood of conception was highest. Those same psychological and physiological mechanisms now lead women to engage in greater consumption of products relevant to reproductive drives during the fertile phase of their cycle."

Now I don't want to get into too much detail about the science behind this article since I want to focus on the press-release, but I do want to highlight one element of the research conducted:

Working with his doctoral student and co-author Eric Stenstrom, Saad recruited hundreds of participants by canvassing classes at Concordia university. Through a careful selection process, 59 female participants were chosen. Over a period of 35 days, the women kept detailed diaries that chronicled beautification behaviours, clothing choices, calorie consumption, and purchases.

So what we have here is a very limited sample size. The study is only looking at 59 women over a period of 35 days. All these women are college students, and given that Concordia is a public university they are all likely Canadian citizens. In addition, Montreal is about 75% Caucasian, so it is likely that a majority of the participants in this study are college-age white women (Let me be clear that I am just making estimates based on the press release and some quick Google searches). Given this small, limited sample size, it is clear that the conclusions drawn about the research by the author of the press release is a good example of a Hasty Generalization. A Hasty Generalization occurs when one draws a general or universal conclusion from a small or limited set of data, and this is exactly what we see here in drawing a conclusion about the evolutionary origins of female behavior from a sample size of 59. There are roughly 7 billion people on earth and about half of them are women. That amounts to about 3.5 billion women in the world, so a sample size of 59 would equal about .00000000842% of the population (please check my math and leave a comment if I got ti wrong!). This is really too small a percentage to draw a general conclusion about the evolutionary origins of this behavior.

Beyond this, the authors also appear to be guilty of a Post Hoc fallacy. Here I am less confident about whether or not a fallacy has been committed. Having not read the original paper, perhaps the authors of the study do correct for this, but given the limited information in the press release, I think we can reasonably question whether or not there is a causal link between the behaviors identified and the menstrual cycle. First, the study had a very short duration, just 35 days. The average menstrual cycle is about 28 days, but there is enormous variation among women, and even the same woman can have significant differences between the lengths of her menstrual cycles. Given this, a 35 day study is far too short to draw a causal inference between the cycle and the behavior. Furthermore, given that all of these women were in essentially the same environment (college students at a public university) it seems that there could be numerous other factors causing or contributing to the behavior of the women. Given a study performed under these constraints, the authors have at best identified a correlation, but one can't assume this correlation to be equivalent to causation. This is a Post
Hoc fallacy.

Finally, even if the authors have discovered a causal connection between menstruation and consumption behaviors, it doesn't seem that one can assume that the latter causes the former. For example, it might well be the case that cultural factors play a much larger role in determining behavior than genetic or biological factors. It might be that women are told by their cultures to engage in certain behaviors at certain times of their menstrual cycle. If this were true, then it wouldn't be biology causing behavior, but rather culture causing behavior during certain times of a woman's menstrual cycle. Given that all the participants in the study were members of a similar culture group (women at a large, Canadian public university), this hypothesis seems as reasonable as the conclusion drawn by the authors. I am not claiming that this hypothesis is correct, but merely noting that it is another possible explanation that appears to be neglected, and when one assumes a particular causal relation beyond the available evidence, one is is committing a Post Hoc or False Cause fallacy.

For more on the relation between genetics, culture and behavior generally, I recommend this podcast.

h/t to Pharyngula