Ok folks, so we’re nearing the end of our Wikipedia list of issues, so I’m at the point where I don’t know what to call this one. We have a bunch of random issues I’ll run through in order. Ready? Let’s go!

Context Sensitivity

In scientific study, context sensitivity refers to the idea that the same study performed under two different sets of circumstances might yield different results in ways people didn’t expect. This seems somewhat obvious when you say it directly, but often isn’t actually on people’s minds when they are reading a study. I have actually covered this a LOT on my blog over the years, as often people will make huge claims about how men or women (in general) view marriage, and you’ll find out the whole study was done on a group of 18 year old psychology students who are almost certainly not married or getting married any time soon. Zooming out, there’s a big criticism that most psychological research is done on “WEIRD” people, meaning Western, Educated, Industrialized, Rich and Democratic. What we consider settled science around human behavior may not be so settled if you include people from wildly different countries and contexts.

So how does this apply to true crime? Well, just like when I look up a paper the first thing I do is go to the methods section to understand the context in which the data was collected, I think the most important thing in a true crime story is to understand the big picture of where and how things happened. As I mentioned previously, true crime cases are often really unusual cases, so it’s important to flag any abnormalities will be heightened substantially. A few questions: how much crime is in the area in general? Were there any unusual events challenging people’s behavior? True crime often goes over this stuff, but I’ve noticed some cases breeze through contextualizing things or not acknowledging that unusual circumstances might change people’s behavior.

The other odd context thing is that a lot of people seem to think that because a case became well known later, the initial investigators should have been thinking from the get go how things would look on Dateline. Unfortunately most investigators/witnesses/defendants don’t have the luxury of knowing in the first 24 hours that people will be reviewing their actions for decades to come. If the case is OJ Simpson? Well yes, you should be prepared for that. If the case is Jon Benet Ramsey? You should give them some grace for not predicting the firestorm. Context matters.

Bayesian Explanation

This is similar to some of the statistical concerns I mentioned last week, but basically if you have a “surprising” result and a low powered study, Bayes theorem suggests you will have a high failure to replicate rate. Bayesian statistics can be powerful to help think through this, because they force you to consider how likely you thought something was before you ran your study, which can help you put your subsequent results in context.

So what’s the true crime equivalent? Well, I think it’s actually a good reminder to put all the evidence in context. Here’s an example: imagine a police department (or podcaster) believes a suspect is guilty mainly because they failed a polygraph. The polygraph has a low ability to detect real guilt (low power) and many innocent people fail it (high false-positive rate), and the prior likelihood that this particular person committed the crime is low. Even though the polygraph result says “guilty,” it does not mean there is a 95% chance they did it. Just like a weak psychological study, a “positive” polygraph doesn’t reliably tell you whether the hypothesis is true or whether the result will replicate.

This can be reapplied to all sorts of evidence, and should be, particularly when you have one piece of evidence that flies in the face of the rest of them. We even have a legal standard for this: circumstantial evidence, which can only be let in under certain circumstances. However in true crime reporting, a lot of circumstantial evidence is treated as extremely weighty, regardless of how discordant it is with everything else. You have to be honest about the prior probability or all your subsequent calculations are going to be skewed.

The Problem With Null Hypothesis Testing

This is a somewhat interesting theory, based on the idea that null hypothesis testing may not be appropriate for every field. For example, if you are testing whether or not a new drug helps cure cancer, you want to know if it has an effect or not. Pretty simple. But with a field like social psychology, human behavior may be too nuanced to have a true yes or no question. Running statistical tests that suggest there is a clear yes/no might end up with unreliable results because the whole set up was inappropriate for the question asked.

In true crime, this reminds me of people using legal standards as though they are moral standards or everyday standards we might use. For example, a person accused of rape may not be convicted under a reasonable doubt standard, but that doesn’t mean that you’d be ok with them dating your daughter/sister/friend. In murder cases, even when the police get things wrong they often had a good reason to start believing people were guilty. Drug or alcohol use can make people looks suspicious, lying up front to the police can make you look suspicious, prior similar convictions can make you look suspicious etc etc. I’ve seen a strong tendency for people to decide that whoever they favor is blameless (null hypothesis = absolutely nothing wrong), but as we covered last week a lot of people mixed up with legal trouble have something working against them.

Base Rate Fallacy

I’ve written about the base rate fallacy before, and it can be a tricky thing to overcome. In short, the base rate fallacy happens when something is extremely uncommon and you use an imperfect method to try to find it. For example, if you use an HIV test to test a thousand random people in the US for HIV, we know that 3-4 might have it. If you are using a test that is 99% accurate but has a 1% false positive rate, that actually means more people (10) will get a false positive result than a true positive result. When the frequency of something is low, false positives become a much bigger problem. In publishing, the theory is that previously unnoticed phenomena are getting rare, so surprising findings are increasingly likely to be false positives.

So how does this apply to true crime? Well, it’s a little hard to make a clear comparison, because so many crimes have unusual things happening by default. To take OJ Simpson as an example, it’s unusual for a celebrity of his stature to be accused of a crime. However, it’s also pretty unusual for a celebrity’s ex wife to end up dead like his did. Our base rate doesn’t totally work because we actually know something weird has happened. This is where we have to get back to judging people by evidence, not statistics.

However, in the broader scheme of true crime content, I think it’s good to note that the demand for new cases is currently exceeding the supply. As we’ve continued to cover, people want attractive articulate defendants with “interesting” cases, and we just don’t have that many of them. This creates a vacuum where people are very incentivized to make their cases “interesting” enough for true crime podcasters to pick up on. This is challenging because overall the murder rate in the US is down substantially from the 80s and 90s, so we have fewer current cases to draw from.

Alright, that’s all I have for this week. I’ll be looking to wrap up next week with a few lessons learned and thoughts. Thanks all!

To go to part 8, click here.

Welcome back folks! This week we’re going to be diving in to questionable research practices, and their impact on the replication crisis.

There’s a few specific questionable research practices that we should cover here, but I want to start with the umbrella concept of “researcher degrees of freedom”. This is a concept that basically means that any time you want to do any experiment, there are a bunch of different ways to do things. Two people locked in separate room asked to create an experiment will almost certainly come up with two different ways of doing things, just because there is often not one right way to do anything. This is fine. What’s less fine is when people start making choices that we know have a high tendency to produce false positive results. Some of these include data dredging, selective reporting of outcomes, HARKing (hypothesizing after results are known), PARKing (pre-registering after results are known). So what are these things and how do they impact research? Let’s get in to it!

Ok, before we talk about the questionable research practices, I want to take a step back and remind everyone that at it’s core, science is largely about trying to deal with the problem of coincidences. We as humans have a tendency to see patterns that may or may not actually exist, and most scientific methods were developed with the idea that you need to separate out coincidences from true causal effect. For example, if a lady claims she can taste a difference in tea based on when the milk was added, you can design a randomized experiment to actually test her prowess. This helps us separate a few lucky guesses from a real effect. Now to reiterate, we almost exclusively have to use this to assess coincidences. If a woman says she could identify the strategy used to make and then promptly misidentified how her tea was made, we’d all move on. It’s mostly when something starts to look plausible that we have to really dig down in the scientific method to figure out what’s going on.

Enter data dredging. Often researchers get large data sets at one time, and may start to run something called exploratory data analyses, just to see if any interesting correlations emerge. This is fine! The problem is that it wasn’t always well communicated that this is what was being done. I’ve actually written about this before when covering some of Brian Wansink’s ongoing scandals and comparing it to my approach for my masters thesis. If you are looking at hundreds of data points, the chance of uncovering a coincidence goes way up, and there are actually statistical methods created just to solve for this problem. Data dredging often leads to spurious correlations, which are fine and relatively easy to deal with if you admit this might be a problem and that you’re going to have to investigate more to figure out if the correlation is real or not. The problem is that even very smart researchers can trick themselves in to thinking that a coincidental finding is more meaningful than it actually is. Andrew Gelman has done excellent work explaining this in his paper “The garden of forking paths: Why multiple comparisons can be a problem, even when there is no fishing expedition or p-hacking and the research hypothesis was posited ahead of time“. In it, he points out that no one has to do this on purpose:

In a recent article, we spoke of shing expeditions, with a willingness to look hard for patterns and report any comparisons that happen to be statistically significant (Gelman, 2013a). But we are starting to feel that the term fishing was unfortunate, in that it invokes an image of a researcher trying out comparison after comparison, throwing the line into the lake repeatedly until a fish is snagged. We have no reason to think that researchers regularly do that. We think the real story is that researchers can perform a reasonable analysis given their assumptions and their data, but had the data turned out differently, they could have done other analyses that were just as reasonable in those circumstances.

The papers go on to explain several examples, but the basics issue is that when you are looking hard for something, you will start unintentionally start expanding your definitions until you are very very likely to find something that is coincidental but you believe it fits in with your initial hypothesis. This can quickly lead to some of the other issues I mentioned earlier: failing to report all endpoints (because you only report the meaningful ones, leaving people in the dark that you actually tested other things), HARKing, basically stating after the fact that you “knew it all along”, and pre-registering after the fact (PARKing) where you take it a step further and state that this is what you were always looking for. Gelman has great examples, but XKCD perhaps put it best:

So how does this apply to true crime? Oh you sweet summer child, let me tell you the ways.

Remember a paragraph or two ago when I said “the basic issue is that when you are looking hard for something, you will start unintentionally start expanding your definitions until you are very very likely to find something that is coincidental but you believe it fits in with your initial hypothesis.“? This is so rampant in true crime you wouldn’t even believe it, and there’s literally no statistical test waiting in the wings to help sort things out.

For years, people have been aware that police can do this. Once they fixate on a suspect, everything that person does can look suspicious. Did the suspect get extremely upset when they heard their wife was dead? Suspicious, probably an act. Did they fail to get upset? Also suspicious! I would argue a large part of our justice system is set up specifically to guard against this issue, and it’s why we have a standard of “beyond a reasonable doubt”. People are fallible, our justice system is imperfect but acknowledges this exists. But as we’ve covered, the disparate and competitive true crime ecosystem takes very little time for self reflection and often breathlessly reports coincidences with no particular acknowledgement that sometimes a coincidence is just a coincidence.

This gets complicated by the fact that (as we covered earlier in this series), true crime type cases happen disproportionately in suburbs or smaller towns vs large cities. I actually have wondered if part of the reason is because it’s easier to have “coincidences” in places with smaller more stable populations. If a police officer gets called to a random murder in the projects, it’s very unlikely they will have a connection to whoever the victim/suspect is. A police officer in a town of 25,000 though? You’ve got a much higher chance of knowing someone who knows someone. Now if there are three officers who respond to a place where 4 people live? You have almost guaranteed someone has a connection. Suspicious!!!! You may think I’m exaggerating, but the case in my town involved DOZENS of things like this. It’s actually what first got me thinking about the connection with the replication crisis. I have had so many people say things like “you REALLY believe that it’s a coincidence that <one of the ten first responders> had a connection with <a relative of one of the four people there>.” Yes, actually I do. It would actually be pretty bizarre if there were literally no connections. I can’t run errands on a Saturday morning without running in to someone I know and you think a bunch of random people from the same town would have absolutely no connections to each other?

The problem here is that people way underestimate the probability of a coincidence. One of my favorite examples of this is “the birthday problem“, a classic stats problem that asks people what the chances are that two people in a randomly selected group of people have the same birthday. Most people are surprised to find out that you only need 23 people before the chances are 50/50 that you’ll get a match, and by 30 people you have a 70% chance of a match. The issue is that people forget you are not trying to find one particular birthday match, but any birthday match. This wildly increases the chance you’ll find something. For my problem above, let’s say you have 6 first responders. Each of them has some combination of parents, siblings, inlaws, children, neighbors and other “suspiciously close” people in their lives. Let’s give them 15-20 each. Now the 5 people at the scene who each have a similar number of people are compared to this. So in the end we have 100+ people compared to 60-80 people, all living in a similar area, and we are looking for ANY connection. What are the chances you think we find one? I’d say quite high! But in true crime land, stuff like this is stated like a stunning development.

Ok now, I’m going to take a step back and be fair for a second: if you’re trying to figure out who did a crime, coincidences may be something you have to look at. I’ve often been annoyed when people start yelling “correlation does not imply causation“, because of course it does. That’s the whole problem. Two items that are correlated may not be causal, but they are much more likely to be causal than two uncorrelated items. But just like with all coincidences, you have to test it against other things to figure out if it’s a coincidence or if you’re really on to something. Otherwise you’re just data dredging, desperately looking for anything that connects to jump off of.

And overinterpretation of data around crimes can cause a LOT of problems. One of the saddest parts of the book In Cold Blood (a classic by Truman Capote that kicked off the modern true crime genre) is when the killers were caught six weeks after murdering a family in a quiet town, Capote mentions many locals had trouble accepting the killers (who ended up confessing) because various coincidences had convinced them others were involved. Those types of issues can stick with you for YEARS, with people vaguely feeling you must have done something.

Ok, so you may be thinking I’m just calling out random individuals here, surely no big time groups report on coincidences like they are meaningful? Au contraire! Recently I saw a Charles Manson documentary that makes the charge that Manson was actually a CIA sleeper agent, based largely on the fact that a CIA member was operating near Manson for years. The guy pushing the theory goes in to great detail about this CIA agent, and how many places were linked to both Manson and this CIA operative. I was already feeling skeptical, when about three quarters of the way through the documentary, the main guy drops “it’s the perfect theory, I just can’t put them in the same room at the same time”. Wait what? Record scratch. So everything we’ve been talking about is just to explain that these two men lived in the same city for some period of time, but you can’t prove they ever met? That’s sort of a key piece of evidence! And this documentary was based on a whole book that was a NYTs best seller and is considered an “Editor’s pick” on Amazon. I don’t see how you consider any of this any more than data dredging, looking for some coincidence, any coincidence, you can use to prop up your theory.

And don’t even get me started on only reporting certain outcomes, this is endemic. One of the weirder examples I’ve come across is one of the most viewed Netflix true crime documentaries of all time The Keepers. This documentary looks in to the murder of a young nun in 1969, and correlates it with sex abuse allegations a woman (then a student) came forward with against a priest at the same school. The heroes are the now adult women who went to the school at the time, and it holds a 97% positive review on Rotten Tomatoes. Surely it can’t be a coincidence that a young nun turned up dead at a school where girls were being molested, can it? Well, it may not be that simple. The problem? The viewers are never told about the background of the primary accuser, which I got suspicious about when I heard she’d been offered a very low settlement. I went looking and discovered that the primary accuser admits that the memories of her abuse was all “recovered memories” that she had no idea about until decades later shortly after she started visiting a new therapist. She actually initially accused her uncle and dozens of strangers, then after a decade of accusations moved on to accusing dozens of school employees. All her initial accusations were actually documented pretty early on, as she filed a 1992 lawsuit that ended up enumerating all of them, including her own admission that many of her reported memories were verifiably false and (in her words) “bull crap”:

The documents go on to point out that even once she settled on the school, she first accused a different priest, then after finding out he was deceased, recanted and moved on to a new priest. And literally none of this is in the documentary. I don’t know what happened here and it sounds like the priest may have been creepy to some people, but it does feel relevant to know someone actually accused dozens of other people before they got to the person in question. There is another accuser, but interestingly the documentary does actually make clear she didn’t come forward until the first woman sent out a mass mailing to all her classmates. I’d really encourage you to read the whole article if you want a sense of how badly a smash hit true crime documentary can shape a narrative through omission.

So where does this leave us? Well, much like with correlated data points, I don’t think it’s wrong to point out coincidences, as long as they are properly contextualized. But a few things to keep in mind:

1. Watch how big you’re making your data set. As mentioned, the more people you look at, the more likely you are to find odd coincidences. Expanding your timeline to everything dozens of people have ever done or to include all of their family members/friends/neighbors/acquaintances wildly expands your data pool. The bigger the data pool, the less meaningful every individual coincidence.
2. Don’t discard data that doesn’t fit the narrative. It’s interesting to watch some coincidence finders totally discard certain coincidences with “well of course that person behaved strangely, they were in shock” while jumping all over other coincidences. I understand the temptation, but it’s always good to admit when your own side has holes.
3. Be aware other people might have already discarded data before you got there. Most documentaries/podcast series have limited space and are going to discard some pieces of information and some of that information could have been important. My new suggestion for everyone is that if you must watch a true crime documentary, google the name + criticism before you watch it. Once someone gives you a slick narrative you are much less likely to care about information that could have shaped your conclusions had you known it beforehand. At least you’ll know when they’re breezing by something that could have been important.
4. Compare coincidences to your own life/baseline probability. Some coincidences are more likely than others. For example, if you hear that someone bought a knife the day before someone got stabbed, but that someone else did laundry the day after, those are two weird coincidences right? Well yes! But also no. Just in my own life, I am several thousand times more likely to do laundry than I am to buy a knife, and I’d imagine most people are. We can certainly look at both people, but remember how frequently you yourself do the “suspicious” action. This also applies to relationships between people. I once had someone ask me how I felt about all the “close relationships” in our local case, and I pointed out the one they were talking about was somebody’s brother’s wife’s sister’s friend. Knowing a bit about their family, I asked them how close they felt to all their brothers wife’s sisters friends, and they admitted that actually did feel rather more distant when they mapped it out using their own brother/sister-in-law/sister of sister-in-laws friend. Again this is fine! But if film makers can make a benign scene feel ominous with scary music, so can true crimers make somewhat distant relationships feel close by saying them ominously.
5. Understand the modern environment for information finding. One thing that is very new since the time of In Cold Blood or even JonBenet Ramsey is social media. Nowadays if a new crime occurs, it is trivially easy to go online and find who people might be connected to through Facebook. Suddenly, that guy who ran the trivia night you used to go to 20 years ago can be connected to you just as easily as your actual best friends from college or actual family members, making coincidences even easier to find. Additionally, there are even bigger groups of online sleuths desperate to track down leads, and they scour the internet finding even smaller and smaller discrepancies. I recently saw someone mention they believed JonBenet’s mother was complicit in her murder because of weird phrasing she used in an interview 10+ years after the fact. At that point your data set has become absolutely massive and you may need to take a break.
6. Prioritize coincidences backed by other evidence. You’d think this would be obvious, but a coincidence that precisely fits the theory of the crime as backed by physical evidence is a lot more meaningful than “this person did something weird elsewhere”.

That’s all I can think of for now! So I’ll close with a quote from Frederick Mosteller “it is easy to lie with statistics, but easier without them.” If scientists attempting to adhere to good statistical methods can make these mistakes, those not even trying to watch their work are several times more likely to fall in to these errors.

To go straight to part 5, click here.

Hi friends! Last week we covered some problems with the publication system that helped cause the replication crisis in science, and this week we’re continuing in the same vein with three more topics. Ready? Good, let’s dive in.

Standards of Reporting

When people started getting interested in the replication crisis, one of the first things everyone wanted to do was figure out how bad it was. Before this could even be tested, an immediate problem was noticed: most papers don’t describe what they did well enough for anyone to even try to replicate it. And this was in cancer biology. Yikes. Now having actually written some papers in oncology and also having written work instructions for how people should do their job, I will state this is almost certainly for two reasons: it is not easy to write out what you did well enough for someone else to truly follow, and it’s very boring to try to do so. If you’ve ever tried to teach a kid how to do a multi step home chore, you’ve probably seen this. “Ok now put the soap in the dishwasher” quickly makes you realize you did not specify there is special dishwasher soap and a specific spot in the dishwasher for said soap. So basically this is not necessarily sketchy, but also could seriously impede replication efforts.

So how does this relate to true crime? Well, the biggest content delivery systems for true crime right now are podcasts and documentaries, which just so happen to be the hardest mediums to include any sourcing in. Depending on venue, court documents can be really inaccessible, police don’t tend to release a detailed timeline of their investigation until the trial, and even then they keep it pretty narrow to the specific case. So figuring out the big picture of how an investigation played out can actually be super hard and it’s extremely hard to find a source document to check if your podcaster/documentary film maker of choice is being honest or even just reading the facts the way you would have. I ran in to this recently when someone Tweeted out an “everything you think you know about Amanda Knox is wrong” type thread, and I decided I’d check a fact or two to see if this person was trustworthy. The problem? All the stuff necessary to do that is in Italian. I did eventually find a fan maintained document repository that has some translations, but it’s still wasn’t quite sure how to check a quick fact. I gave up.

This is not great because most of how we sort through which podcasts to listen to on history or politics or other topics come from a quick assessment of how honest people are, but with true crime it’s almost impossible to do this easily. Even when I talk to people about my local case it’s often very hard to send them sources for corrections, often the source is buried in the middle of 5 hours of testimony that has no transcripts, so you’d have to watch through hours of footage to link to the spot. So you’ve got a case where you are telling a story, but it is extremely hard for anyone to check the specifics of what you’re saying. That kind of set up has never once bred honesty. The only advice I can give here is to see if there are podcasts/documentaries from two different sides and try to consume both of them. At least then you’ll see what people leave in and what they leave out. And honestly? The crazier the story sounds for people’s behavior especially over long periods of time? Question it harder. Some interesting studies were pretty rightfully called in to question when people started pointing out they had very brief methods sections for very elaborate study set ups. “Dozens of people acted in insane ways for a period of several years” is a claim you should always be skeptical of. It’s not impossible, but always good to see if there is any nuance being left out. After all, if it happens in science where you’re required to write up everything and cite sources, it’s almost certainly happening even more often in podcasts that are required to do neither of those things.

Procedural Bias

So in that last section we covered some issues that can arise with science even when everyone has the best intentions, and in this section we are going to cover another one. Procedural bias concerns arose from a thing called the Duhem-Quine hypothesis, that talks about how most scientific research actually rests on a bunch of different assumptions, including that your instruments are actually working correctly. In psychology research, there’s concerns that people could end up only testing their instruments/procedures if their tests show nothing, but could assume any positive result is proof their thesis is correct. This is a pretty human tendency right? If I ask you to show me how far you can hit a baseball and you don’t do well, you’ll probably ask for another try because your finger slipped/there was a loud noise/the wind blew the ball/etc/etc. But if the wind blows the ball in your favor, you would almost certainly accept the extra few feet. We tend to look for what went wrong when we don’t get a particular result we want.

In science? This is not a great tendency. But in the justice system? This is a feature not a bug! When someone is accused of a crime, they get a lawyer whose actual job is to sit and nitpick every single thing that was done to get to the point of indictment. This is good and how the system is supposed to work, a defense attorney who showed up and said “gosh your honor, it actually looks like the police did everything they could, guess we gotta take the L here” would be grossly negligent and probably lose their license. If the police searched a two block radius? Well why didn’t they search a 3 block radius? If the local police handled it? Why didn’t they call in the state police? If they called the state police? Why didn’t they call earlier. If they called early, what were they so worried about? Etc etc etc. Again, this is a good and proper design of the system and anyone who doesn’t get this type of representation should. But I think true crime has taken this tendency a little too far in a few different ways.

First, as trust in police declines, I’m seeing people put a surprising amount of trust in defense attorneys, as though they are not paid to question everything. If the police went right instead of left first, the defense isn’t saying “oh well everyone knows you go right first” because they necessarily believe that to be true, they are saying it because it is absolutely their job. You’d think this would be obvious, but especially with good defense attorneys I see a surprising number of people quote them as though they are authorities on the topic. This works in both directions, btw, with people claiming offense that a defense attorney claimed a rape victim actually consented (not a lotta defenses left if you don’t use that) or people saying the police were obviously wrong because the defense attorney said so.

None of this is to say that defense attorneys can’t cross ethical lines, and indeed, I’ve been discovering there are surprisingly few ways of reigning rogue defense attorneys in. However, the point is just because a defense attorney claims something does not mean it’s true or even what they would be claiming in a non-professional setting. One of the more interesting points I read while looking in to this is that while defense attorneys have done an excellent job branding themselves as defenders of our constitutional rights, it should be noted that defense attorneys at work are only defending their clients constitutional rights. They will absolutely argue that the police could have or should have violated other people’s constitutional rights if it will help their client. In the case I’m familiar with, the lawyers actually argued multiple times for warrantless searches for other people in a way many critics pointed out they’d be infuriated with if it was done to any of their clients.

So this is all fine for defense attorneys, who are doing their job. But I think this tendency has snuck in to true crime, particularly amongst people who fancy themselves civil rights defenders. If your answer to how one persons rights should have been preserved is to suggest violating someone else’s rights, then you’re not a civil rights advocate, you’re a fangirl. While courts attach certain rights to those on trial, it is absolutely insane to act like only those accused of crime have rights. A defense attorney doesn’t get the contents of my phone just because he wants it, he also has to offer evidence just like what had to be offered to get his or her clients phone. The constitution applies to all of us at all times, not just individual people at specific times.

All of this points to a slightly different problem I’ve noticed with a lot of true crime media and fans: they want it both ways using legal standards. Some time ago, I had a heated discussion with someone who felt differently than I did about our local case. She dismissed multiple things I said as “irrelevant to the court case” and declared she wanted to just follow the case like the jury would. Ok, fair enough! But less than 5 minutes later when she wanted to counter a different point, she promptly mentioned several things that had also not been allowed in to court. I’ve known this person for years and truly believe she was not trying to be manipulative, I think she actually didn’t notice what she was doing. I didn’t even notice what she was doing until I thought about it later, but since then I’ve proactively brought it up to people when I see it. We can either talk about everything from a strict legal perspective, or we can talk about it from a colloquial “do we think they’re guilty standard” but we can’t have two different standards depending on which part of the case we’re talking about. Make sure you’re machine’s working when you get results you like and when you get results you don’t like. Having two different standards is human nature, but it’s a recipe for disaster when it comes to truth.

Cultural Evolution

Ok, this is a fun one, based on a paper with a great name “The Natural Selection of Bad Science“. In it, the authors use some fun statistical methods to show that if scientists primarily get promoted based on publications, and there are no particular penalties for your study failing to replicate, the quality of science will, over time, optimize towards a high volume of low quality publications. They explain it this way:

An incentive structure that rewards publication quantity will, in the absence of countervailing forces, select for methods that produce the greatest number of publishable results. This, in turn, will lead to the natural selection of poor methods and increasingly high false discovery rates. Although we have focused on false discoveries, there are additional negative repercussions of this kind of incentive structure. Scrupulous research on difficult problems may require years of intense work before yielding coherent, publishable results. If shallower work generating more publications is favoured, then researchers interested in pursuing complex questions may find themselves without jobs, perhaps to the detriment of the scientific community more broadly.

Yuck.

So how does this apply to true crime? Well, as we covered in the publish or perish section, true crime is a highly competitive space and making sure you have a steady stream of content is more important than having an entirely accurate retelling of the story. Currently, there are almost no ramifications for those who are inaccurate, so one assumes the same dynamics will come in to play.

This is actually one spot where I think true crime may be in a slightly better spot than science, as there are some podcasters who literally make “we are going to do a ton of research and be moderate and careful” their brand. At least some of these have gotten a pretty dedicated following, so it is possible for consumers to demand more of this. With science, sadly, most of us never know people who toiled away in obscurity and then didn’t succeed in their field.

Ok, that’s it for this week! Next up we’ll start getting in to questionable research practices. Some of these are field specific so may not entirely apply, but we’ll see what we can draw out. Thanks for reading, and stay safe out there.

To go straight to part 4, click here.

Welcome back! Last week we covered the proposed historical and sociological causes of the replication crisis and applied it to things we see in the true crime genre, and this week we’ll be doing a similar analysis with the group of causes under “problems with the publication system”. As a reminder, I’m loosely following the order in the replication crisis Wiki page, so if you want more you can go there. There’s about 6 reasons listed under the “problems with the publication system” section, so we’ll be taking those one at a time.

Publication Bias

Publication bias in science is a topic I’ve written a lot about over the years, but perhaps my most succinct post was when I covered it during my review of the Calling Bullshit course, where they did a whole class on it. I think the second paragraph I wrote during that post broke it down nicely:

This week we’re taking a look at publication bias, and all the problems that can cause. And what is publication bias? As one of the readings so succinctly puts it, publication bias  “arises when the probability that a scientific study is published is not independent of its results.” This is a problem because it not only skews our view of what the science actually says, but also is troubling because most of us have no way of gauging how extensive an issue it is.  How do you go about figuring out what you’re not seeing?

In science, some of the findings were skewed by people being more interested in doing novel research rather than trying to replicate others findings, the “file drawer effect” where papers that didn’t find an association between two factors were much less likely to be published, and (outside of science) the fact that the media will mostly focus on unusual findings rather than the full body of scientific literature. My guess is you already see where this is going, but lets think how this applies to true crime.

One of the first thing anyone who looks at true crime as a genre starts to realize is that the crimes covered by traditional true crime are almost never the most common type of crimes. While there’s no “average” homicide, there is certainly a “modal” one! If you were going to describe a typical homicide in the US, most people would pretty quickly come up with something close to this: a young adult man, shot with a handgun, by another young adult man he knows, during an argument or dispute, in a city setting.

Looking at the stats, we see why this would come to mind: about 80% of homicide victims are male, about 90% of perpetrators are male. Age-wise, crime is dominated by the young. The FBI data also tells us that firearms are used in about 73% of homicides, that you are much more likely to be killed by someone you know than someone you don’t know, and just based on population density alone we would assume most homicides happen in crowded cities. I didn’t include race in the “modal” case because it’s actually closer to 50/50 black vs white, but both homicide victims and perpetrators are disproportionately black. Now there’s a lot of holes in this data because we don’t always know who killed someone, but I think most people would agree based on the general news that these data match what we assume.

If you listen to true crime though? You won’t find that type of crime represented almost at all. I think nearly everyone who’s ever glanced at the news knows if you go missing, heaven help you if you’re anything other than a young attractive white woman, and true crime’s racism problem has been remarked upon for years. I asked ChatGPT which 20 true crime cases it thought got the most media attention in the US (post-1980), and it’s pretty clear these cases caught fire in part because they are so unlike the “typical” crime:

1. OJ Simpson Trial (1994-95): famous defendant, white female victim, knife violence
2. JonBenet Ramsey (1996): white female child victim, wealthy family, beauty pageants, strangled
3. Menendez Brothers (1989): Wealthy family, kids murdering parents
4. Jeffrey Dahmer (1991): Male victims, gay sex, cannibalism
5. Casey Anthony (2008): Attractive female defendant, white female child victim, strangulation
6. Scott Peterson (2002): White female pregnant victim, knife victim
7. Central Park Five (1989): white female victim, not murdered
8. Amanda Knox (2007): white female victim, white female defendant, stabbed
9. Michael Jackson child abuse trials (1993, 2005): celebrity defendant
10. OJ Simpson Robbery case (2007): famous defendant, already accused of a crime
11. BTK Killer (2005): Serial killer
12. Richard Ramirez “Night Stalker” (1980s): serial killer
13. Waco Siege (1993): Mass death, cult
14. Columbine high (1999): large school shooting
15. Unabomber (1996): Mass death
16. Tylenol murders (1982): multiple dead, unknown culprit
17. Jodi Arias (2008): white female defendant
18. Gabby Petito (2021): white female victim, social media star victim
19. Michael Skakel/Martha Moxley: Kennedy connection for defendant
20. Pamela Smart (1990): white female defendant

What’s interesting about this list is immediately a few things jump out: gun violence is very underrepresented with only a few cases (Menendez brothers, Pam Smart, Columbine, sort of Jodi Arias) involving a firearm. Outside of the mass deaths/serial killers, almost all of the cases involve someone who was at least middle class or higher. Very few cases involve a solo male victim, it’s mostly solo females or men dying in mixed groups. The exceptions are actually 2 cases where there were accusations of homosexuality (Dahmer, Jackson), and the remaining one is Pam Smart’s husband. There’s also a dearth of black or Hispanic victims by themselves, they only appear in groups. In other words, it’s pretty clear the true crime genre does not get hooked on your “average” crimes, they want the unusual ones. I asked Chat GPT what the modal true crime case was, and it summed it up this way: A White, female (often young) victim, killed in a domestic or intimate context, often by a male partner/family member, in an otherwise “safe” suburban setting. The perpetrator is either wealthy/celebrity or a seemingly ordinary middle-class person hiding darkness. The case includes salacious details, a highly publicized trial, and often an ambiguous or polarizing outcome.

Seems about right.

The problem here is if one pays attention to true crime, you are getting an inaccurate view of how crimes are committed and who the victims are. I think this not only skews people’s perception of their own risk of being a victim, but also people getting weirdly judgmental of police departments. Once I started poking around at older true crime cases, I found that an incredibly common criticism is when police treat a big unusual case as though it was going to be a normal case. Well, yeah. Even large police departments may not be prepared for a celebrity perpetrator, simply because we don’t have too many celebrities running around killing people. The day the call comes in for a surprisingly big case, no one flags for the police “actually better send your best guys down there and double the amount of resources available, this case is going to be on Dateline next year”. They are operating under the assumption they will be responding to the modal crime story, true crimers believe they should have been prepping for the modal true crime story.

I also think it’s very relevant how many of these cases happen in quiet suburbs or small towns. In the case I’ve become familiar with, we’ve had 4 murders here in 40 years, and our county has a murder rate of 1/100,000 a year. That’s on par with the safest countries in the world. The idea that taxpayers were going to pay through the nose to keep our police department in a constant state of readiness for unusual events disregards how most taxpayers actually function. Indeed, there was an audit done on our local police department during all this, and one of their conclusions was “if you want your police trained for unusual events, you’re going to have to increase their training budget so they can go do that” and people FLIPPED OUT. And these were the people who were most viciously critiquing the police! Even after years of unrest they were still unwilling to increase the training budget, believe that (as one person actually publicly put it) “you can just watch CSI to know what you should do”. Sure, and you can skip medical school if you watch old ER reruns.

And finally, I think a lot of people justify the focus on white wealthy attractive people with a sort of “trickle down justice system” type philosophy. If we can only monitor how the police handle the most vaunted in our society, this will somehow trickle down to help the poor and the marginalized. The problem is, I’ve never seen particular evidence that’s true. How did the myriad of resources poured in to JonBenet Ramsey’s case help anyone? I grew up near Pam Smart, and I don’t recall that case making much difference after things settled down. Indeed, I think these cases often give us a false impression of what accused killers and victims “worthy” of sympathy look like. Indeed, attractive people are much more likely to get preferential treatment in every part of the justice system. They are arrested less often, convicted less often, and get shorter sentences when they do. It’s hard to get numbers on what percent have a college degree, but even the most generous estimates suggest it’s around 6% of inmates compared to 37% of the general population. The pre-jail income average for prisoners is about $19k a year. Wealthy educated attractive people have very little trouble getting their stories boosted, the people who need help are those not in those groups.

All of this to say, listening constantly to a non-random assortment of cases is not going to give you a good sense of how our justice system works on a day to day basis, any more than only publishing (or pushing) flashy science results gave us a sense of scientific fields. As a pro-tip, when you hear about a case that’s gaining traction, it’s not a bad idea to try to find a couple similar non-famous cases with victims/perpetrators who aren’t wealthy or attractive to see how those cases were handled. Your concerns may remain, but at least it will give you a baseline to work from that typical true crime reporting lacks.

Mathematical Errors

One interesting issue that has played in to a few replication attempts seems almost too silly to mention, but typos and other errors can and do end up influencing papers and their published conclusions. Within the past few days I’ve actually seen this happen at work when we found out that an abstract had the wrong units for a medication dose we wanted to add to a regimen. The typo was mg vs g, so it would have been a very easy typo to make and a pretty disastrous issue for patients. So at least a few replications might fail due to simple human errors in pulling together the information. For example, an oft quoted study saying that men frequently leave their wives when they are diagnosed with cancer was quickly retracted when it was found the whole result was a coding error. The error was regrettable, but what’s even worse is I still see the original finding quoted any time the topic comes up. It’s not true. It was never true, the finding would definitely not replicate. Even the authors admit this, but the rumor doesn’t die.

So how does this relate to true crime? In almost every major case I’ve peaked at, rumors get going about things that did/didn’t happen, and it is very hard to kill them once they’ve started. One good example is actually the Michael Brown/Ferguson case, where it was initially reported he said something like “Hands Up, Don’t Shoot”, a phrase so popular it now has it’s own Wikipedia page. The problem? It doesn’t exist. When the DOJ looked in to the whole thing, the witness who initially said it happened no longer said it did, none of the evidence matched this account, and it’s considered so debunked even the Washington Post ran an article titled “Hands up Don’t Shoot Did not happen in Ferguson“. For the public though? This is considered gospel. I’ve told a few people in the past few years that this didn’t happen, and they look at me like I kicked a puppy. When I’ve pulled up the WaPo headline, Wiki page or DOJ report, they’re still convinced something is wrong. How is it possible something so repeated just…didn’t happen?

I’m not sure but this is way way way more common in true crime reporting than anyone wants to believe, especially on the internet. Shortly after my local case was resolved I saw a Reddit thread about it on a non-true crime subreddit, and people were naming the evidence that most convinced them of their opinion, 7 out of the first ten things listed didn’t occur. And I’m not talking “are disputed” didn’t occur, I’m talking “both the defense and prosecution would look at you like a crazy person if you made these claims in court” stuff. People were publicly proclaiming they’d based a guilt or innocence opinion on stuff they’d never checked out. Since then I play a game in my head every time I talk to someone about the case, I count how many pieces of evidence they mention before they get to one that’s entirely made up. 90% of people don’t get past their third piece of evidence before they quote something made up. That is…not great.

Interestingly when I’ve corrected people, they generally look at me like I’ve missed the point and I’m dwelling on trivialities. To that I have two responses:

1. If it’s worth your time to lie about it, it’s worth my time to correct you.
2. If I were being accused of a heinous crime I didn’t do, whether in court or just in public opinion, I would want people to correctly quote the evidence against me. So would you. So would everyone. These are real people’s lives, this is not a TV show plot you’re only half remembering.

It’s totally fine in my mind not to be super familiar with any famous public case btw, but if you’re going to speak on it and declare you have a strong opinion, you may want to make sure all of your foundational facts are true. With the internet providing so much of our information now, it is really easy to mistakenly quote something you saw someone tweet about rather than something you actually saw testified to.

Publish or Perish Culture

When you take up a career in science, publication is key to career advancement. One of the issues this leads us to is that papers with large or novel findings are far more likely to be published than those that don’t have those qualities. And what’s less interesting than spending tons of time and resources on a study that someone already did just to say “yeah, seems about right, slightly smaller effect size though”. If there’s no particular reward for trying to replicate studies, people aren’t going to do it. And if people aren’t going to do it, you are not going to spend too much time worrying about if your own study can be replicated or not. One can easily see how this would lead to an issue where studies replicated less often than they would in a system that rewarded replication efforts.

So with true crime, the pressure is all on people to make interesting and bold claims about a story to catch eyes. The remedy for this has basically always been defamation claims, and if you think replications are slow and time consuming, boy have I got news for you about defamation claims. Netflix got incredibly sloppy with it’s documentaries and has a stack of lawsuits waiting to get sorted out in court, but progress is glacial.

This problem has been heightened by the influx of small creators who don’t actually have a lot to lose in court. If you work for the NYTs and report something wrong, your employer takes the hit. If you have a tiktok account you started in your parents basement, you can pretty much say whatever you want knowing no one’s going to spend the cash to go after you. This is starting to change as people realize they need to send messages to these content creators who make reckless accusations, but change is slow. Even true crime podcast redditors have wondered how some of the hosts get away with saying all the stuff they do and why more people don’t sue. Oh, and now the mainstream media can just report on the social media backlash rather than report on things directly. Covington Catholic helped set some better guidelines for this, but the problem remains that none of this has improved the accuracy of reporting.

Even if the content creators confine themselves to facts, they often aren’t their facts. Like all social media, pumping out weekly content is king, and most people simply do not have time to thoroughly research cases themselves. A surprising number of true crime podcasts have been hit with plagiarism accusations, including one where they were just reading other people’s articles on air without attribution. Given that podcasts often end up licensing their content, this drives a lot of possibly sticky legal issues. So what are the consequences for this? As of right now almost nothing. The podcast named in the article above just removed the episode, and as of this writing is the 6th most listened to podcast in the world. Publish or perish, good research be damned.

All right we have a few more publication issues to cover, but I think I’ve gone on long enough and will save that for next week. Stay safe everyone!

To go straight to part 3, click here.