The Carlisle Method: One Year Later

Every once in a while the traffic on an old post goes through the roof, and I realize something interesting must have happened. This week it was my Carlisle Method post, which I appreciated because I had been meaning to check back in on that whole situation.

For those of you who didn’t read the initial post, here’s the overview:
An anesthesiologist came up with a neat statistical way of checking for data fraud. Rather than focus on the flashy results and conclusions of papers, he focused on the characteristics of the test group vs control group and looked for evidence that they weren’t random. The theory is that people committing fraud would mostly focus on their conclusions, but might get sloppy when reporting things like subject age and such.  He publicly named the papers he thought were most questionable, and investigations were launched.

A year later, some of the results are in. The New England Journal of Medicine announced their results, and they actually are a bit encouraging. Out of the 11 papers they analyzed, 5 had mislabeled “standard error” as “standard deviation” so the Carlisle Method was picking up on the error. 5 papers were reviewed and found to be okay, and they suspect some limitations in the Carlisle Method itself had wrongly flagged them.

One paper however, is getting lots of attention. It turns out that a major paper on heart health and the Mediterranean Diet wasn’t quite done as reported. While it had been reported as a randomized trial, a bit of digging showed that about 20% of its participants  were actually “cluster randomized”.  The authors reanalyzed the data using the remaining 80% of people, and noted that this didn’t change the conclusions, but it did make them less statistically compelling. The paper was retracted and replaced with a new version that correctly covers the methods.

Now all of these errors may appear to be small, but they do raise a bit of a cause for concern. There is an assumption that all statistics will be checked thoroughly before papers go to press, but I think this highlights that sometimes errors get through. Particularly if a paper has a big splashy finding, it’s possible that errors will not be reviewed. The NEJM shared this concern, and is implementing a statistics course for their editors and giving extra scrutiny to published papers.

More concerning than the NEJM’s findings however, is that the other 7 journals who had papers on the list haven’t completed their investigations yet, said there were no problems with any of the papers, or didn’t respond. It seems a little unlikely that only the most prestigious journal involved had any errors, so it’s unfortunate the other journals aren’t doing more.

More on the varied reactions to the Mediterranean diet study here and here.

One thought on “The Carlisle Method: One Year Later

  1. I am still liking your idea that the percentage of Mediterraneans who are following Orthodox (or Roman Catholic, to a lesser extent) feast-fast schedules might have skewed the initial promise of the Mediterranean Diet. Any suggestion of that in the discussions? It also occurs to me that places that have had farming a long, long time, such as the Mediterranean, might have genetic differences in responding to different foods. We already know there are natural tendencies toward this with alcohol, in that long-time farmers have some body responses that discourage them from drinking too much, while those who have never farmed, such as Eskimos, don’t have those.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s