Every once in a while the traffic on an old post goes through the roof, and I realize something interesting must have happened. This week it was my Carlisle Method post, which I appreciated because I had been meaning to check back in on that whole situation.

For those of you who didn’t read the initial post, here’s the overview:
An anesthesiologist came up with a neat statistical way of checking for data fraud. Rather than focus on the flashy results and conclusions of papers, he focused on the characteristics of the test group vs control group and looked for evidence that they weren’t random. The theory is that people committing fraud would mostly focus on their conclusions, but might get sloppy when reporting things like subject age and such.  He publicly named the papers he thought were most questionable, and investigations were launched.

A year later, some of the results are in. The New England Journal of Medicine announced their results, and they actually are a bit encouraging. Out of the 11 papers they analyzed, 5 had mislabeled “standard error” as “standard deviation” so the Carlisle Method was picking up on the error. 5 papers were reviewed and found to be okay, and they suspect some limitations in the Carlisle Method itself had wrongly flagged them.

One paper however, is getting lots of attention. It turns out that a major paper on heart health and the Mediterranean Diet wasn’t quite done as reported. While it had been reported as a randomized trial, a bit of digging showed that about 20% of its participants  were actually “cluster randomized”.  The authors reanalyzed the data using the remaining 80% of people, and noted that this didn’t change the conclusions, but it did make them less statistically compelling. The paper was retracted and replaced with a new version that correctly covers the methods.

Now all of these errors may appear to be small, but they do raise a bit of a cause for concern. There is an assumption that all statistics will be checked thoroughly before papers go to press, but I think this highlights that sometimes errors get through. Particularly if a paper has a big splashy finding, it’s possible that errors will not be reviewed. The NEJM shared this concern, and is implementing a statistics course for their editors and giving extra scrutiny to published papers.

More concerning than the NEJM’s findings however, is that the other 7 journals who had papers on the list haven’t completed their investigations yet, said there were no problems with any of the papers, or didn’t respond. It seems a little unlikely that only the most prestigious journal involved had any errors, so it’s unfortunate the other journals aren’t doing more.

More on the varied reactions to the Mediterranean diet study here and here.