Argh argh argh

The AVI left me an interesting link on my last post on famous social psychology studies that have not been replicated.  It’s good reading….they include the famous study that found that teacher’s expectations being self fulfilling (ie kids achievement went up or down based on how smart the teacher thought they were).  That was interesting to me, as I’ve heard that study quoted many times, and never heard that larger studies had failed to replicate it.

Anyway, as I was reading that article, a headline for another article floated across the top of the screen “Sleeping more than 7 hours or less than 5 1/2 hours has been found to decrease longevity”.

No.

No.

No.

I don’t even have to read the article to tell you no study found any such thing.

The only way you could actually prove that is to randomize three groups, force one to sleep more than 7 hours, one to sleep between 7 and 5.5 hours, and one less than 5.5 hours per night (for the rest of their lives) and then see how long they lived.  No one did that.  We know no one did this.

Sure enough I clicked on the article and found that people who reported getting more than 7 hours of sleep/night were 12% more likely to do within 6 years than those who got slightly less (again, with the raw numbers the 12% increase might not be that impressive….how many otherwise healthy people died in the 6 year time period to begin with?).  So there is a correlation, but no one proved what caused it.  The most obvious caveat is that people who are sick might sleep more.

Why oh why do people still write headlines like this?  I can see it when it’s on the front page of Yahoo.com or something, but shouldn’t Psychology Today have slightly higher standards?

Sigh.

Lord of the Rings Statistics

Four posts in two days?  This is what happens when the little one starts sleeping in 7 hour stretches.

Anyway, this one was too good to pass up….a statistical breakdown of various aspects of Lord of the Rings.

More thoughts on voting and non publication bias

The more I think about the study I commented on yesterday, the more irritated I am they didn’t include a control group (either women over 50 or women on hormonal birth control) to give some context to their claims.

Of course then the results might not have been as stark, and this means they either would have chosen not to publish, or it wouldn’t have been accepted for publication.  It’s crucial to keep in mind that study authors are under no compulsion to publish any results they don’t like.  Obviously, this can skew what gets out there.  Apparently there are laws that actually require this reporting for drug trials, but an audit found only 20% compliance in the US.
Ben Goldacre is currently waging quite the campaign trying to get pharmaceutical companies to live up to the laws that require them to publish info on ALL of their clinical trials, not just the ones that produce flattering results.  This comes in conjunction with his new book Bad Pharma that has apparently caused quite a stir (it’s not out yet in the US….but it will be in January…in case you wondered what to get me for Christmas).
I suggest reading some of his blog posts if you want a crash course in publication bias and why it’s so harmful to us.  The quick example of course is the study on hormones and voting….do you really think if a study came out showing that women’s menstrual cycles did not effect their voting that it would be published?  Journals wouldn’t find it interesting, and researchers who base their careers on finding ovulation/behavior links would likely not even submit it.  
In the last chapter of his book Bad Science, Goldacre takes the media to task for this.  He documents how the most sensational science stories are almost never given to science writers in the interest of making a better story.  He then calls out journalists (by name) in the UK who published stories calling for more research on vaccine/autism links, while subsequently failing to report when such research was done (and came up with no link).  
If you haven’t read anything by him, I highly recommend it.

Technical Clarification

I was feeling a bit ranty in my last post about the women/hormones study, but I decided it needs a slightly more academic treatment.  Despite CNN yanking the story, I managed to find the original study and read the whole thing.

A few points:

  1. All the participants were paid via Mechanical Turk for their participation.  This gave me pause.  Depending on how this was set up, I was curious how they verified that people didn’t give some of their answers just to qualify to get paid.  
  2. The study did not follow individual women and show them to be fluctuating.  The study compared groups of women at high and low fertility times and reported their differences.
  3. The measured political attitudes excluded all fiscal views (because those didn’t change much) and focused only on social views.  
  4. The single women assessed for political affiliation had a median income of $15000-25000/year, whereas the married women had incomes of $35000-$50000/year.  Interestingly, in the discussion section, this difference is considered relatively small and inconsequential.
  5. While the study (and articles) mention that they surveyed 275 women for the first experiment, they later clarify that they tossed out nearly half of them because they couldn’t reasonably determine where they were in their cycle.  The second study started at around 500 and got whittled down the 300.  This means the groups being compared were about 75 people each in the first study and 150 each in the second.  
  6. The groups were not controlled for anything.  Those income ranges are so big you could drive a truck through them, and nothing was said about what states people came from. 
  7. No woman under 44 was counted, nor were any of them asked if they planned on voting. 
Overall, I was less weirded out by this study when I saw the authors.  They are all pretty hard core evolutionary psych folks, and pretty much believe everything people do is hooked to mating opportunity (interesting, this includes religion.  Apparently women become religious to either stop themselves from cheating or to attempt to impose a social order on others that will keep their mates faithful).  Take a look at Kristina Durante’s publishing history and you’ll see why they never even looked at how any other variables might influence anything.  Truthfully, they saw a link where they had already decided there was a link.  
With sample sizes as small as they were from a very specific group (people seeking out paid work on the internet), a control for region or income would have been helpful.  Additionally, the group studied (18-44) is the least likely group to vote.  Even beyond their reproductive years, women still tend to vote Democrat…so there’s that.  I also thought it interesting that there was no control for historic voting behavior…if women who voted for Obama in 2008 were more likely to change their vote in conjunction with specific times of month, it might have been more interesting.  As is though, we have no idea if there’s a real shift in individuals or if it’s just the groups they picked (a interesting number of their p values did not reach the level of statistical significance, the results were reported in the article without this caveat).  

Data so bad even CNN took it down

After living in New Hampshire for my entire upbringing, moving to Massachusetts when I was 18 was a bit of a surprise.  Why you ask?  Because my goodness are election years more peaceful here.

For those readers who aren’t from New England, New Hampshire residents are some of the most harassed people in the nation when it comes to presidential elections.  Between the first in the nation primary and swing state status, the amount of effort people put in to trying to find out what New Hampshirites are going to do on election day is staggering.  Massachusetts on the other hand is reliably blue, so everyone pretty much leaves us alone (Exception: the Scott Brown/Elizabeth Warren face off is really harshing my mellow this year).

Anyway, as a woman who both strongly believes it’s her civic duty to vote and who puts a lot of thought in to her vote every 4 years, I was a bit surprised to see a story on CNN yesterday about how women voted with their hormones.  The link actually goes to Jezebel there because the “science” was so bad that CNN actually took the story down. 

Essentially, the research claimed that during “that time of the month” women felt sexier.  This led single women to want more social services (because they apparently were worried they wouldn’t be able to help but get pregnant with a random partner).  Married women on the other hand apparently overcompensated and wanted to vote Republican because they….I don’t know.  I really couldn’t follow the convoluted reasoning of how feeling sexy or not influenced your vote.

To note, this was an internet survey done by a marketing research person.  It also apparently found that women’s level of religiousness varied based on monthly cycle.

The sheer weirdness of saying political party and religious affiliation, two of the deepest and most profound beliefs people have, is based on a few fluctuating hormones (of course only in women….I mean, have you ever heard of testosterone influencing men?  I don’t think so) is just so reductionist it’s bizarre.

It also of course leaves out post menopausal women, women who are on hormone regulating birth control, and ignores better research that shows women in committed relationships are already more likely to be conservative.  Oh, and it totally leaves out anyone voting for a third party candidate.

I bring this up not just because it was a bad story and because it actually got taken down, but also because it’s part of a larger phenomena of journalists inflating the effect of small differences to write a better story.  I am really stunned how many times in the past week I’ve seen stories about “why Obama/Romney isn’t getting as much support as he should”.  The author then goes on to talk about some line of reasoning that supposedly explains why their candidate would be creaming the other guy if it weren’t for the influence of the small factor that they and only they are acknowledging.

News flash to the media:  most people not voting for your candidate are voting the way they are because they don’t agree with him, or your party, or because they like the other candidate or party better.  Stop belittling large portions of the population while trying to prove otherwise.

*Gets off soapbox*
Thank you for your time.