Well we’ve reached the end of the road here folks, and it’s time to wrap things up with some conclusions and consequences. As I mentioned in the first post, I’ve been loosely following the Wikipedia entry on the replication crisis, and I’d like to point out the first paragraph of it’s consequences section (bolding mine):

When effects are wrongly stated as relevant in the literature, failure to detect this by replication will lead to the canonization of such false facts.^[195]

A 2021 study found that papers in leading general interest, psychology and economics journals with findings that could not be replicated tend to be cited more over time than reproducible research papers, likely because these results are surprising or interesting. The trend is not affected by publication of failed reproductions, after which only 12% of papers that cite the original research will mention the failed replication.^[196][197] Further, experts are able to predict which studies will be replicable, leading the authors of the 2021 study, Marta Serra-Garcia and Uri Gneezy, to conclude that experts apply lower standards to interesting results when deciding whether to publish them.^[197]

So overall we find that in science, with highly educated PhDs with professional reputations and institutional affiliations built on truth we find that:

• False facts end up being canonized
• Less reliable studies get more attention
• Even when findings are formally challenged, they will continue to be repeated as true with almost no one mentioning they were called in to question
• Standards are lower for anything surprising or interesting

Do we really believe that Youtubers and TikTokers are actually more reliable than this, while they compete for nothing but attention? I hate to beat a dead horse, but papers can get retracted, colleges can investigate you, and you can sink a career in academia. Maybe not often, but the odds are certainly better than even a mainstream journalist actually losing a defamation case. Science is set up to self police, maybe not as well as it should be, but there are mechanisms. True crime documentaries and podcasts are set up to entertain, and there are no mechanisms to self correct outside of a person getting aggravated enough to file a lawsuit against you. So it is very likely that:

• Some portion of what you believe you know about popular cases is flat out false
• The most popular cases will have more incorrect facts floating around than the “boring” cases
• Even when things are proven to be incorrect, they will not stop circulating as fact
• Standards are lower for anything surprising or interesting

So what do we do?

Well, it’s actually not straightforward. Because of the apparatus around science, it’s been straightforward to propose changes. Change hasn’t always come fast, but it has been progressing. True crime has no such oversight, so any change will be a challenge. However, I think the things I used to bring up in my Intro to Internet Science Course still all apply here. I broke down the things to watch for in to 4 categories: Presentation: How They Reel You In, Pictures: Trying to Distract You, Proof: Using Number to Deceive, and People: Our Own Worst Enemy. I think those still all apply here, with just a few tweaks.

1. Presentation: How They Reel You In A high production value documentary is not the same as an honest documentary, and a lengthy series on a topic does not mean people didn’t leave anything out. Be skeptical of things, no matter how glossy or voluminous.
2. Pictures: Trying to Distract You In the stats and data world, graphs are often used to catch people’s eye and give them the immediate visual impression something is happening before they’ve had a chance to read anything. In true crime, this is often what the victims or the perpetrator look like, immediately playing on tropes of who we think commits crimes or which victims get our sympathy. Be skeptical of anything that focuses on the good looking, wealthy or college educated to the exclusion of others. Additionally, watch any attempt to immediately invoke another case or movie in the current case, which will prime you to skip actual facts in favor of an “I know this type of person, they do X”. When our local case hit national media, one of the first things one of the main people did was to start citing a popular movie filmed in the area almost 20 year ago, based somewhat on events that had occurred 20-30 years prior to that. The attempt to evoke specific imagery was clear.
3. Proof: Using Number to Deceive While numbers aren’t always at play in the true crime world, evidence certainly gets kicked around pretty often. But just like numbers, out of context evidence is often worse than useless and extremely misleading.
4. People: Our Own Worst Enemy We bring our biases to every case, and some narratives will be more palatable to us than others. Be careful with people who bring cases in to make a “bigger point” or anything that seems a little too outrageous or focuses on extremely unusual types of crime. It’s also good to look back on early reporting and see if what got you in to the case held up, and to actually take it in to account if it didn’t.

To all of this, I’d add two more points. The first is that a surprising number of people tell me that true crime is fine playing fast and loose with the facts as long as it challenges the police, because there the state has more power. This is of course how our whole justice system is set up, but I think it falls rather flat. In science we are taught that there are both type 1 errors (false positives) and type 2 errors (false negatives) and that both carry consequences. This is also true in the criminal justice system. Blackstone’s principal says that it’s better that ten guilty men go free than one innocent man hang, and that is what we build our system around. But this doesn’t mean there’s no consequences to a guilty person going free. The obvious first issue is that they offend again, and that we will then also be upset that nobody stopped them. But this is a natural consequence of “it’s never bad to let the accused go”, and we can’t have it both ways. A recent Twitter thread highlighted this from a victims perspective, as she recounted both the emotional toll of testifying against a stranger who assaulted her and then watching him get let go repeatedly just to watch him continue to assault other women. The other issue of course is that if you have a justice system that never finds anyone guilty, people take things in to their own hands. It’s commonly noted that the mafia initially gained power with immigrant Italian communities because the police wouldn’t investigate crimes against them, and the same is true of newer gangs. Likewise, the Old Testament is riddled with references to the sin of denying justice. Even if you’re not religious, it’s good to flag that unpaid for crimes have been considered a socially destabilizing force for thousands of years. Playing fast and loose with the truth about government actions is not a victimless crime just because they have power, as people typically find when their particular group falls out of favor in the court of public opinion.

And finally, I want to give a mini rant about why this topic bothers me so much. Watching a case up close and personal like this, I was stunned and appalled how many people seemed to completely miss that this case was for many people, one of the darkest moments of their life even before the internet was involved. Watching people turn that in to their own personal whodunit/reality TV show was horrifying. People talked about the various people like they were merely characters in a movie, like you could say horrifying things about them with no consequences. I didn’t know these people but I do see many of them frequently, and the pain on their faces was visible. None of this was fun. None of this was asked for. We’re in a time when we have blockbuster documentaries about how exploitive reality TV show was, so it’s bizarre to me so many people are excited to tune in to stories about people who never volunteered for this. While errors in scientific publishing can erroneously impact how we view the world, errors in true crime reporting can irreparably ruin lives. The first one may sound worse, unless you’re the target of the second. Power posing failing to replicate hurt a few self help gurus talks, thousands of people falsely accusing someone of murder is something you probably never recover from. Consume media that reminds you that everyone involved, whether accused or victim, is a human.

Thanks for reading folks.