The Null Hypothesis is not Null

Here's a riddle.  What is one of the most misunderstood, unappreciated, and frequently unpublished facts in science?  The answer: the null hypothesis.  Now, don't stop reading.  Fight off the temptation to roll your eyes and click to the next page.  And whatever you do, DON"T let those traumatic memories from your grad level psychological methods course cause you to avoid the material that follows.  It's important stuff.  Generally, the null hypothesis is considered supported whenever a study finds that the difference between the means of two groups does not exceed what would be expected by chance.  Why is it so important?  Because null is generally thought of as unimportant--as a finding of  "nothing."   More troubling, however, is that it doesn't sell books, help establish careers on the speaking circuit or sell new pharaceutical products.  Consider this: historically, research journals are less likely to publish studies that find...nothing (or, in other words, support the null hypothesis).  Think about that finding the next time you hear about the latest "evidence-based practice."  When confronted by such claims, the qusetion is, "How many studies are stuffed in file drawers around the country that were not published because their sexiness was NULL-ified by their findings?"  The problem is made worse by the current requirement that treatments need only prove superior to a placebo or treatment as usual comparison in two studies in order to make the list of official, "empirically supported" or evidence-based practices.  If you don't think this is a problem, you probably haven't been keeping up with the latest skirmishes between the field of Medicine and the pharmaceutical companies.  In fact, the risk to science and knowledge was considered so great, that the AMA recently set up a database where all clinical trials of new medications have to be registered prior to initiation in order to be published in an official AMA journal.  An article from the June 16th, 2004 New York Times notes that the AMA was taking this unprecedented step because the "drug industry...was affecting quality and because companies as well as medical journals tend to spotlight tests with positive findings compared with those having negative or inconclusive results."

And the evidence shows the problem is not limited to medicine.  Consider Multisystemic Therapy--currently the rage for treating troubled youth.  However, according to professor of social work Julia Littrell, Ph.D., "The biggest and best studies show it doesn't work."  How could Dr. Littrell come to such a startling conclusion?  In part, by having access to a database of clinical trials that never saw the light of day--in other words, studies that were never published.  Says Littrell, "a lot of experiments that have been held up as evidence for MST have problems with their design."  Oy vey, if only the governments of Denmark, Sweden, and Norway had known this before dedicating tremendous professional and financial resources to the adoption of MST for children's services. 

Dingfelder, S.F. (December, 2004).  The significance of null.  Monitor on Psychology, 24-25.
Meier, B. (June 16, 2004).  AMA urges disclosure on drug trials.  New York Times, C1, 1, 14.

More Baloney...