The AVI passed on an interesting post about a new study that replicates the finding that many psychological studies don’t replicate. Using 21 fairly randomly selected studies (chosen specifically to avoid being too sensational…these were supposed to be run of the mill), replication efforts showed that about 60% of studies held up while almost 40% could not be replicated.

This is a good an interesting finding, but what’s even more interesting is that they allowed people to place bets ahead of time on exactly which studies they thought would fail and which ones would bear out. Some of the people were other psych researchers, and some were placing bets for money. Turns out that everyone was pretty darn good at guessing which findings would replicate:

Consistently, studies that failed to replicate had fewer people guessing they would replicate. In fact, most people were able to guess correctly on at least 17 or 18 out of the 21.

Want to try your hand at it? The 80,000 hours blog put together a quiz so you can do just that! It gives you the overview of the study finding with an option to read more. about exactly what they found. Since I’m packing up for a work trip this week, I decided not to read any details and just go with my knee jerk guess from the description. I got 18 out of 21:

I encourage you to try it out!

Anyway, this is an interesting finding because quite often when studies fail to replicate, there are outcries of “methodological terrorism” or that the replication efforts “weren’t fair”. As the Put A Number on It blog post points out though, if people can pretty accurately predict which studies are going to fail to replicate, then those complaints are much less valid.

Going forward, I think this would be an interesting addendum to all replication effort studies. It would be an interesting follow up to particularly focus on the studies that were borderline….those that just over 50% of people thought might replicate, but that didn’t end up replicating. It seems like those studies might have the best claim to change the methodology and repeat.

Now go take the quiz, and share your score if you do! The only complaint I had was that the results don’t specifically tell you (I should have written it down) if you were more likely to say a study would replicate when it didn’t or vice versa. It would be an interesting personal data point to know if you’re more prone to Type 1 or Type 2 errors.