Of course the world's greatest detective knows his fallacies

Here is a nice example of a Hasty Generalization. I don't really need to explain it since the great Adam West as Batman makes the fallacy explicit.

An important consideration when looking at psychology studies.

PZ Meyers recently posted about a paper "The Weirdest People in the World" (PDF) by Joseph Henrich, Steven J, Heine, and Ara Norenzayen. In this paper the authors warn about avoiding hasty generalizations when looking at the results of psychological research.The main thesis of the paper is that there is a serious problem regarding subject samples in contemporary psychology research. In particular, too much of that research is focused on a group the authors call the WEIRD (Western, Educated, Industrialized, Rich and Democratic).  Research is done on this WEIRD group and is then erroneously generalized to cover the entire human population.

Who are the people studied in behavioral science research? A recent analysis of the top journals in six sub‐disciplines of Psychology from 2003‐2007 revealed that 68% of subjects came from the US, and a full 96% of subjects were from Western industrialized countries, specifically North America, Europe, Australia, and Israel (Arnett 2008). The make‐up of these samples appears to largely reflect the country of residence of the authors, as 73% of first authors were at American universities, and 99% were at universities in Western countries. This means that 96% of psychological samples come from countries with only 12% of the world’s population. Put another way, a randomly selected American is 300 times more likely to be a research participant in a study in one of these journals than is a randomly selected person from outside of the West.

The authors go on to note that the problem is even worse in that "67% of the American samples (and 80% of the samples from other countries) were composed solely of undergraduates in psychology courses." This means that an enormous amount of psychological research is apparently biased in favor undergraduate students at American universities. But, this need not be a problem if these results are generalizable. However, the authors look at a number of important cross-cultural surveys and find that many of the results for American undergraduates cannot be generalized to the rest of the world.

The most striking example of this (in my opinion) is the discussion of the Meuller-Lyer Illusion. 

The idea here is that line segments (a) and (b) are identical in length, and the question then is how much bigger does line segment (a) need to be in order to look equal in length to line segment (b). When done on American undergraduates, line segment (a) needs to be about 1/5 as long to look equal to line segment (b), but in other populations the amount varies. This chart nicely summarizes the results:

 

As the chart indicates, the American undergraduates (from Evanston, all the way on the right) required segment (a) to be the longest of any of the groups surveyed while the illusion didn't even work for the San people of the Kalahari. This means that American undergraduates are anything but a representative sample of human's generally, and research that is based solely on them will be seriously in error.

The major take-away from all this is that we need to be careful when drawing generalizations. We need to ensure that our sample size is robust enough and unbiased enough that we can legitimately generalize to the population as a whole. This also means those of us living in America need to take extra care to remember that most people in the world do not live like us, and that our experiences and beliefs cannot be generalized to the whole world. As this study indicates, because we are WEIRD we are weird.

Big Pharma Hides the Truth

The science columnist, author, and academic Dr. Ben Goldacre, has just written a new book about the pharmaceutical industry, Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients. In connection with the publication of this book, The Guardian has published an excerpt. In this excerpt, Goldacre discusses a number of fallacies regularly committed by the pharmaceutical companies in their efforts to get their drugs approved for use by various government regulatory agencies. We can see these fallacies clearly exemplified in Dr. Goldacre's discussion of the anti-depressant reboxetine.

Through the looking glass

It is with some reluctance that I write this post as I have generally been avoiding news reports concerning the massacare in Aurora, Colorado. As is often the case with tragedies that get a great deal of traction in the media, people come out of the woodwork exploiting that tragedy as a way to garner support for their particular political cause. There has obviously been a great deal of discussion about gun control (though apparently no one's opinion is changing), but I want to focus on a slightly different angle, promoted most recently by Jon Rappoport.

In an article published today on NaturalNews, and reprinted from his blog Rappoport suggests that the accused Aurora, Colorado shooter, James Holmes, was either (a) the patsy for some large conspiracy designed to change public opinion on gun control or (b) driven crazy by psychiatric drugs or (c) driven crazy by psychiatric drugs so that he could be used as the patsy or perpetrator of a mass killing designed to change people's opinions on gun control. The article itself is poorly written and reads more like a rant than any kind of thoughtful analysis, but I will do my best to unpack some of the fallacies lurking in it.

The first and most obvious is the post hoc fallacy. Rappoport doesn't quite come out and say that Holmes was driven violently insane by pharmaceuticals (though he does make that claim in his blog), but that does appear to be the major point of the article.  After making a number of unsubstantiated claims about the pressure being exerted by pharmaceutical manufacturers on law enforcement and the judicial system to suppress the medical records of Holmes in order to prevent tarnishing the reputation of the drugs Holmes was taking thereby causing people to refuse to take them thus hurting the profits of Big Pharma, Rappoport then launches into a lengthy (but more substantiated, at least there are references though almost all of them are over 20 years old) discussion of the potential psychotic side effects of Prozac. The clear inference here is supposed to be that Holmes took Prozac (even though there is no evidence that this was a drug he was taking, we will probably know more when his medical records are released during the court proceedings), this made him psychotic, and this psychosis led him to commit the crimes for which he is accused. This is a clear example of a post hoc fallacy in that the author is assuming a causal relation when all we have evidence for is a correlation (a correlation Rappoport invented for the sake of his article). Assuming that Holmes was taking some sort of psychiatric medicine, it seems far more likely that his underlying mental condition caused him to commit his crimes, rather than the medication he was taking to treat that crime. As a quick note, I have no evidence concerning Holmes' psychiatric condition or medication he may have been taking, I am merely trying to unpack and evaluate the argument Rappoport makes. For all I know, Holmes may have been perfectly sane and was not prescribed nor taking any medication.

Moving on, Rappoport then commits a rather large Hasty Generalization when he suggests that because there is some evidence of people becoming psychotic when they take certain classes of anti-depressants that therefore everybody who takes them will become psychotic. Again, there is no direct quote which shows a clear example, but the article as a whole is clearly trying to make the case that this generalization is true.

Lastly, Rappoport's article is filled with what can best be described as Bullshit. Bullshit occurs when one has no concern for the truth and just makes up whatever facts or stories will support one's position. For example:

So right now, in Aurora, there are pharmaceutical people on the scene. Not just low-level goofballs, but competent investigators. They want to know what drugs James Holmes was prescribed. They need to know. And behind the scenes, people with clout are making phone calls. These pharma types are talking to government agents and it's crazy time and damage-control time, and nobody is laughing. This is a high-stakes game. WHAT DRUGS WAS HOLMES TAKING?

Rappoport provides no evidence whatsoever that any of the things he describes in this paragraph are actually taking place. Similiarly, he writes:

[If] they could get to Holmes in his cell, they'd erase him. They'd make it look like a suicide. Today. What do you think "lone shooter" is all about? Yes, the covert op that very well may have used Holmes as the patsy, to push the government into banning guns, is a major piece here. But that work is done. Now it's "lone shooter" because getting rid of Holmes by killing him or warehousing him for the rest of his life in a mental prison, with brain-hammer drugs making him into a vegetable, means that the names of the psychiatric drugs he was taking before the massacre will be lost to history, and no one will take the criminal investigation any further.

Again this appears to have no basis in reality, and no evidence beyond Rappoport's wild speculations. This is, quite simply, bullshit.

Why aromatherapists are not (and shouldn't be) doctors.

In a guest editorial in the MansfieldNewsJournal, George Cox, an aromatherapist, attacks the status of current health legislation in the US. In particular, he objects to those provisions of US law that prohibit unaccredited individuals (people who aren't MD's) from making unverified health claims (making claims about products when there is insufficient evidence of efficacy). In laying out his argument Cox commits several fallacies: a version of the Ad Hominem and several Hasty Generalizations.

I will begin by looking at the ad hominem. In his article Cox uses a particular version of the ad hominem known as poisoning the well. This is when one introduces negative claims about a person or group as a way to discredit anything that this individual or group might claim in the future. In a sense it is a sort of preemptive ad hominem. We can see this at work when Cox writes:

Traditional doctors and the AMA are not infallible, despite what TV shows have shown us over the years. All you have to do is think of a couple of instances to prove that. Remember, butter is the worst thing you can eat, eat only margarine. Then a few years later, margarine is bad for you, eat only butter. Huh?
Eggs were the worst thing you could eat for a while, or so they told us. Your cholesterol would go sky high and you would die. After a bunch of poor chicken farmers lost everything, they came back and said eggs aren't so bad after all.

Basically, Cox is arguing that because the medical community has made mistakes in the past, we can't trust anything they say in the future. This is a classic rhetorical technique of turning an opponent's greatest strength against hm and treating it as a problem.  One of the greatest benefits of medical science (and science generally) is that it is evidence based. When a doctor makes a medical recommendation, that recommendation is generally based on the best evidence available at the time. However, because the recommendation is evidence based, when new evidence comes in this sometimes forces a revision in the current understanding, leading to new recommendations. If science thinks something is going to work and then it doesn't, the scientist doesn't stick his head in the sand, he bites the bullet and revises his opinion. This is exactly what we would hope for from any medical practitioner: a recommendation based on the best evidence currently available. This is in sharp contrast to many alternative medical modalities in which the recommendation stays the same regardless of evidence. To try to suggest that this responsiveness to evidence is a defect rather than an advantage is simply poisoning the well.

Cox then goes on to commit a hasty generalization when he writes:

To the people who say that aromatherapy -- my business -- is just a bunch of hooey, I would point out that dentists still use clove bud essential oil for dry socket and root canal pain. They have access to all of the narcotics that MDs do, but they still use clove bud oil because it works. It stops the pain.
So if it stops the pain in a tooth, wouldn't it, shouldn't it be helpful for arthritis pain? Both are technically bone pain.

First, it is worth noting that Cox has greatly over-simplified the dental uses of clove oil. Orac has a great deal more to say on this topic, but suffice to say that the picture is significantly more complex than Cox makes it out to be.  Furthermore, dentists use clove oil because there is strong evidence that it serves the purposes for which the dentist uses it. Despite what many alternative medicine practitioners claim, doctor are not opposed to natural or nature derived-remedies so long as there is adequate evidence of efficacy. Regarding the hasty generalization, Cox's argument seems to run like this:

1. Clove bud oil treats dry socket and root canal pain.
2. Dry socket and root canal pain are forms of tooth pain.
3. Teeth are bones.
4. Therefore, clove bud oil can treat bone pain.
5. Arthritis is a form of bone pain.
6. Therefore, clove bud oil can treat arthritis.

Again, ignoring the factual errors discussed by Orac in the above link, this is clearly a hasty generalization. Assuming that clove oil does work as an analgesic, it is being applied directly to the bone (or in the case of a root canal, directly to the nerve). This is very different from a topical application to the skin. To assume that because it works in one very specific situation involving teeth that it will apply to all situations involving bone is a clear hasty generalization.

Cox, however, does not stop there and goes one further, writing:

I need to watch my language and say things like, "it's been shown to" or "historically, it has been used to." Even though there are small clinical studies that prove it and we have tons of anecdotal evidence, it doesn't matter.

In effect, Cox is arguing that because there are small, preliminary clinical studies and anecdotal evidence, he should be able to make the same claims of efficacy for his products that doctors are only allowed to make for products for which there have been large-scale, double-blind, placebo controlled studies. This is a very common hasty generalization that alternative medicine practitioners make when they assume that anecdotes have value. The simple truth is that, for a variety of reasons that I can't go into here, anecdotes are worthless. To draw any conclusion about efficacy for a product based on anecdotes is as clear cut an example of a hasty generalization as one can possibly find.

h/t to Orac

It's that time of the month: Hasty Generalizations and Post Hoc reasoning.

A recent press release from Concordia University in Montreal has announced the publication of a paper by lead author Gad Saad linking shopping habits among women to their menstrual cycle. In particular, the study seemed to indicate that women were more likely to buy clothing and beauty related products and spend more time on appearance related activities (manicures, tanning salons, sun-bathing etc.) during the fertile phase of their menstrual cycle, and more likely to consume high-caloric foods during the infertile phase of the menstrual cycle. Of particular interest for me is the linking of these behaviors to evolution, with Saad essentially arguing that women are biologically determined to engage in these sorts of behaviors.

"In ancestral times,” he explains, “women had to focus more time on mating-related activities during the fertile phase of the menstrual cycle, when the likelihood of conception was highest. Those same psychological and physiological mechanisms now lead women to engage in greater consumption of products relevant to reproductive drives during the fertile phase of their cycle."

Now I don't want to get into too much detail about the science behind this article since I want to focus on the press-release, but I do want to highlight one element of the research conducted:

Working with his doctoral student and co-author Eric Stenstrom, Saad recruited hundreds of participants by canvassing classes at Concordia university. Through a careful selection process, 59 female participants were chosen. Over a period of 35 days, the women kept detailed diaries that chronicled beautification behaviours, clothing choices, calorie consumption, and purchases.

So what we have here is a very limited sample size. The study is only looking at 59 women over a period of 35 days. All these women are college students, and given that Concordia is a public university they are all likely Canadian citizens. In addition, Montreal is about 75% Caucasian, so it is likely that a majority of the participants in this study are college-age white women (Let me be clear that I am just making estimates based on the press release and some quick Google searches). Given this small, limited sample size, it is clear that the conclusions drawn about the research by the author of the press release is a good example of a Hasty Generalization. A Hasty Generalization occurs when one draws a general or universal conclusion from a small or limited set of data, and this is exactly what we see here in drawing a conclusion about the evolutionary origins of female behavior from a sample size of 59. There are roughly 7 billion people on earth and about half of them are women. That amounts to about 3.5 billion women in the world, so a sample size of 59 would equal about .00000000842% of the population (please check my math and leave a comment if I got ti wrong!). This is really too small a percentage to draw a general conclusion about the evolutionary origins of this behavior.

Beyond this, the authors also appear to be guilty of a Post Hoc fallacy. Here I am less confident about whether or not a fallacy has been committed. Having not read the original paper, perhaps the authors of the study do correct for this, but given the limited information in the press release, I think we can reasonably question whether or not there is a causal link between the behaviors identified and the menstrual cycle. First, the study had a very short duration, just 35 days. The average menstrual cycle is about 28 days, but there is enormous variation among women, and even the same woman can have significant differences between the lengths of her menstrual cycles. Given this, a 35 day study is far too short to draw a causal inference between the cycle and the behavior. Furthermore, given that all of these women were in essentially the same environment (college students at a public university) it seems that there could be numerous other factors causing or contributing to the behavior of the women. Given a study performed under these constraints, the authors have at best identified a correlation, but one can't assume this correlation to be equivalent to causation. This is a Post
Hoc fallacy.

Finally, even if the authors have discovered a causal connection between menstruation and consumption behaviors, it doesn't seem that one can assume that the latter causes the former. For example, it might well be the case that cultural factors play a much larger role in determining behavior than genetic or biological factors. It might be that women are told by their cultures to engage in certain behaviors at certain times of their menstrual cycle. If this were true, then it wouldn't be biology causing behavior, but rather culture causing behavior during certain times of a woman's menstrual cycle. Given that all the participants in the study were members of a similar culture group (women at a large, Canadian public university), this hypothesis seems as reasonable as the conclusion drawn by the authors. I am not claiming that this hypothesis is correct, but merely noting that it is another possible explanation that appears to be neglected, and when one assumes a particular causal relation beyond the available evidence, one is is committing a Post Hoc or False Cause fallacy.

For more on the relation between genetics, culture and behavior generally, I recommend this podcast.

h/t to Pharyngula