Hi friends! Last week we covered some problems with the publication system that helped cause the replication crisis in science, and this week we’re continuing in the same vein with three more topics. Ready? Good, let’s dive in.

Standards of Reporting

When people started getting interested in the replication crisis, one of the first things everyone wanted to do was figure out how bad it was. Before this could even be tested, an immediate problem was noticed: most papers don’t describe what they did well enough for anyone to even try to replicate it. And this was in cancer biology. Yikes. Now having actually written some papers in oncology and also having written work instructions for how people should do their job, I will state this is almost certainly for two reasons: it is not easy to write out what you did well enough for someone else to truly follow, and it’s very boring to try to do so. If you’ve ever tried to teach a kid how to do a multi step home chore, you’ve probably seen this. “Ok now put the soap in the dishwasher” quickly makes you realize you did not specify there is special dishwasher soap and a specific spot in the dishwasher for said soap. So basically this is not necessarily sketchy, but also could seriously impede replication efforts.

So how does this relate to true crime? Well, the biggest content delivery systems for true crime right now are podcasts and documentaries, which just so happen to be the hardest mediums to include any sourcing in. Depending on venue, court documents can be really inaccessible, police don’t tend to release a detailed timeline of their investigation until the trial, and even then they keep it pretty narrow to the specific case. So figuring out the big picture of how an investigation played out can actually be super hard and it’s extremely hard to find a source document to check if your podcaster/documentary film maker of choice is being honest or even just reading the facts the way you would have. I ran in to this recently when someone Tweeted out an “everything you think you know about Amanda Knox is wrong” type thread, and I decided I’d check a fact or two to see if this person was trustworthy. The problem? All the stuff necessary to do that is in Italian. I did eventually find a fan maintained document repository that has some translations, but it’s still wasn’t quite sure how to check a quick fact. I gave up.

This is not great because most of how we sort through which podcasts to listen to on history or politics or other topics come from a quick assessment of how honest people are, but with true crime it’s almost impossible to do this easily. Even when I talk to people about my local case it’s often very hard to send them sources for corrections, often the source is buried in the middle of 5 hours of testimony that has no transcripts, so you’d have to watch through hours of footage to link to the spot. So you’ve got a case where you are telling a story, but it is extremely hard for anyone to check the specifics of what you’re saying. That kind of set up has never once bred honesty. The only advice I can give here is to see if there are podcasts/documentaries from two different sides and try to consume both of them. At least then you’ll see what people leave in and what they leave out. And honestly? The crazier the story sounds for people’s behavior especially over long periods of time? Question it harder. Some interesting studies were pretty rightfully called in to question when people started pointing out they had very brief methods sections for very elaborate study set ups. “Dozens of people acted in insane ways for a period of several years” is a claim you should always be skeptical of. It’s not impossible, but always good to see if there is any nuance being left out. After all, if it happens in science where you’re required to write up everything and cite sources, it’s almost certainly happening even more often in podcasts that are required to do neither of those things.

Procedural Bias

So in that last section we covered some issues that can arise with science even when everyone has the best intentions, and in this section we are going to cover another one. Procedural bias concerns arose from a thing called the Duhem-Quine hypothesis, that talks about how most scientific research actually rests on a bunch of different assumptions, including that your instruments are actually working correctly. In psychology research, there’s concerns that people could end up only testing their instruments/procedures if their tests show nothing, but could assume any positive result is proof their thesis is correct. This is a pretty human tendency right? If I ask you to show me how far you can hit a baseball and you don’t do well, you’ll probably ask for another try because your finger slipped/there was a loud noise/the wind blew the ball/etc/etc. But if the wind blows the ball in your favor, you would almost certainly accept the extra few feet. We tend to look for what went wrong when we don’t get a particular result we want.

In science? This is not a great tendency. But in the justice system? This is a feature not a bug! When someone is accused of a crime, they get a lawyer whose actual job is to sit and nitpick every single thing that was done to get to the point of indictment. This is good and how the system is supposed to work, a defense attorney who showed up and said “gosh your honor, it actually looks like the police did everything they could, guess we gotta take the L here” would be grossly negligent and probably lose their license. If the police searched a two block radius? Well why didn’t they search a 3 block radius? If the local police handled it? Why didn’t they call in the state police? If they called the state police? Why didn’t they call earlier. If they called early, what were they so worried about? Etc etc etc. Again, this is a good and proper design of the system and anyone who doesn’t get this type of representation should. But I think true crime has taken this tendency a little too far in a few different ways.

First, as trust in police declines, I’m seeing people put a surprising amount of trust in defense attorneys, as though they are not paid to question everything. If the police went right instead of left first, the defense isn’t saying “oh well everyone knows you go right first” because they necessarily believe that to be true, they are saying it because it is absolutely their job. You’d think this would be obvious, but especially with good defense attorneys I see a surprising number of people quote them as though they are authorities on the topic. This works in both directions, btw, with people claiming offense that a defense attorney claimed a rape victim actually consented (not a lotta defenses left if you don’t use that) or people saying the police were obviously wrong because the defense attorney said so.

None of this is to say that defense attorneys can’t cross ethical lines, and indeed, I’ve been discovering there are surprisingly few ways of reigning rogue defense attorneys in. However, the point is just because a defense attorney claims something does not mean it’s true or even what they would be claiming in a non-professional setting. One of the more interesting points I read while looking in to this is that while defense attorneys have done an excellent job branding themselves as defenders of our constitutional rights, it should be noted that defense attorneys at work are only defending their clients constitutional rights. They will absolutely argue that the police could have or should have violated other people’s constitutional rights if it will help their client. In the case I’m familiar with, the lawyers actually argued multiple times for warrantless searches for other people in a way many critics pointed out they’d be infuriated with if it was done to any of their clients.

So this is all fine for defense attorneys, who are doing their job. But I think this tendency has snuck in to true crime, particularly amongst people who fancy themselves civil rights defenders. If your answer to how one persons rights should have been preserved is to suggest violating someone else’s rights, then you’re not a civil rights advocate, you’re a fangirl. While courts attach certain rights to those on trial, it is absolutely insane to act like only those accused of crime have rights. A defense attorney doesn’t get the contents of my phone just because he wants it, he also has to offer evidence just like what had to be offered to get his or her clients phone. The constitution applies to all of us at all times, not just individual people at specific times.

All of this points to a slightly different problem I’ve noticed with a lot of true crime media and fans: they want it both ways using legal standards. Some time ago, I had a heated discussion with someone who felt differently than I did about our local case. She dismissed multiple things I said as “irrelevant to the court case” and declared she wanted to just follow the case like the jury would. Ok, fair enough! But less than 5 minutes later when she wanted to counter a different point, she promptly mentioned several things that had also not been allowed in to court. I’ve known this person for years and truly believe she was not trying to be manipulative, I think she actually didn’t notice what she was doing. I didn’t even notice what she was doing until I thought about it later, but since then I’ve proactively brought it up to people when I see it. We can either talk about everything from a strict legal perspective, or we can talk about it from a colloquial “do we think they’re guilty standard” but we can’t have two different standards depending on which part of the case we’re talking about. Make sure you’re machine’s working when you get results you like and when you get results you don’t like. Having two different standards is human nature, but it’s a recipe for disaster when it comes to truth.

Cultural Evolution

Ok, this is a fun one, based on a paper with a great name “The Natural Selection of Bad Science“. In it, the authors use some fun statistical methods to show that if scientists primarily get promoted based on publications, and there are no particular penalties for your study failing to replicate, the quality of science will, over time, optimize towards a high volume of low quality publications. They explain it this way:

An incentive structure that rewards publication quantity will, in the absence of countervailing forces, select for methods that produce the greatest number of publishable results. This, in turn, will lead to the natural selection of poor methods and increasingly high false discovery rates. Although we have focused on false discoveries, there are additional negative repercussions of this kind of incentive structure. Scrupulous research on difficult problems may require years of intense work before yielding coherent, publishable results. If shallower work generating more publications is favoured, then researchers interested in pursuing complex questions may find themselves without jobs, perhaps to the detriment of the scientific community more broadly.

Yuck.

So how does this apply to true crime? Well, as we covered in the publish or perish section, true crime is a highly competitive space and making sure you have a steady stream of content is more important than having an entirely accurate retelling of the story. Currently, there are almost no ramifications for those who are inaccurate, so one assumes the same dynamics will come in to play.

This is actually one spot where I think true crime may be in a slightly better spot than science, as there are some podcasters who literally make “we are going to do a ton of research and be moderate and careful” their brand. At least some of these have gotten a pretty dedicated following, so it is possible for consumers to demand more of this. With science, sadly, most of us never know people who toiled away in obscurity and then didn’t succeed in their field.

Ok, that’s it for this week! Next up we’ll start getting in to questionable research practices. Some of these are field specific so may not entirely apply, but we’ll see what we can draw out. Thanks for reading, and stay safe out there.

To go straight to part 4, click here.