This is a series of posts featuring anecdotes from the book The Signal and the Noise by Nate Silver. Read all the Signal and the Noise Posts here, or go back the Chapter 2 post here.
Baseball talk. The stats guys vs scouts debates are my favorite.
This is a series of posts featuring anecdotes from the book The Signal and the Noise by Nate Silver. Read all the Signal and the Noise Posts here, or go back the Chapter 2 post here.
Baseball talk. The stats guys vs scouts debates are my favorite.
I’ve often repeated on this blog that there are really two ways to be wrong. I bring it up so often because it’s important to remember that being right does not always mean preventing error, but at times requires us to consider how we would prefer to err.
I bring all this up because I had to make a very tough decision this past Saturday, and it all started with Batman.
It was 3 am or so when I heard my 4 year old son crying. This wasn’t terribly unusual…between nightmares or other middle of the night issues this happens just about every other week. I went out in the hall to see what was happening, and I found him crying hysterically. I picked him up and asked him what was wrong, noticing that he seemed particularly upset and very red. “Mama, I swallowed Batman and he’s stuck in my throat and I can’t get him out” he wailed. My heart shot to my throat. He had a small Batman action figure he had taken to bed with him. I had thought it was too big to swallow, and he was a little old for swallowing toys….but in his sleep I had no idea what he could have done. Before I could even look in his mouth he started making a horrible coughing/choking sound l’d never heard before and was gasping for air through the tears. I looked in his mouth and saw nothing, but thought I felt something.
I woke my husband up, and we briefly debated what to do. Our son was still breathing, but he sounded horrible. I was unsure what, if anything was in his throat. I had never called 911 to my own house before, and I ran down the other options. Call the pediatrician? They could take an hour to call back. Drive to the ER? What if something happened in the middle of the highway? Call my mother? She couldn’t do much over the phone. Google? Seriously? Does “Google hypochondriac” have an antonym that means “person googling something that’s way to important for Google”?
Realizing I had no way of getting a better read on the situation and with my son still horrifically coughing and gasping in the background, I took a deep breath and thought about being wrong. Would I rather risk calling 911 unnecessarily, or risk my child starting to fully choke on an object that might be a funny shape and tough get out with the Hemleich manuever? Phrased that way, the answer was immediately clear. I made the call. The whole train of thought plus discussion with my husband took less than two minutes.
The police and EMTs arrive a few minutes later. My son had started to calm down, and they were great with him. They examined his mouth and throat, and were relatively sure there was nothing in the airway. They found the Batman toy still in his bed. Knowing that his breathing was safe, we drove to the ER ourselves to make sure he hadn’t swallowed anything that was now in his stomach, and that his throat hadn’t gotten irritated or reactive. He still had the horrible sounding cough. He brought Batman with him.
In the end, there was nothing in his stomach. He had spasmodic croup (first time he’s had croup at all), and the doctor thinks that his “I swallowed Batman” statement was his way of trying to explain to us that he woke up with either a spasm or painful mucus blockage in his throat. The crying had made it worse, which was why he sounded so bad when I went to him. While we were there he picked up Batman, pointed to the tiny cloth cape and said “see, that’s what was in my throat!”. We got some steroids to calm his throat down, and we were on our way home. We all went back to bed.
In the end, I was wrong. I didn’t really need to call 911, and we could have just driven to the hospital ourselves. We needed the stomach x-ray for reassurance and the steroids so he could get some sleep, but there was no emergency. But I tell this whole story because this is where examining up front the preferred way of being wrong comes in handy: I had already acknowledged that being wrong in this way was something I could live with. My decision making rested in part on being wrong in the right direction. I can live with an unnecessary call. I couldn’t have lived with the alternative way of being wrong.
Written out here, this seems so simplistic. However in a (potential) emergency, the choices that go in to each box can vary the calculation wildly.
That’s a lot to think about in the middle of the night, but I was glad I had the general mental model on hand. I think it helped save some extra panic, and if I had it to do over again I’d make the same decision.
As for my son, the next morning he informed me that from now on “I’m going to keep my coughs in my mouth. They scare mama.” Someone clearly needs his own contingency matrix.
I talk a lot about ways to be wrong on this blog, and most of them are pretty recognizable logical fallacies or statistical issues. For example, I’ve previously talked about the two ways of being wrong when hypothesis testing that are generally accepted by statisticians. If you don’t feel like clicking, here’s the gist: Type I errors are also known as false positives, or the error of believing something to be true when it is not. Type II errors are the opposite, false negatives, or the error of believing an idea to be false when it is not.
Both of those definitions are really useful when testing a scientific hypothesis, which is why they have formal definitions. Today though, I want to bring up the proposal for there to be a recognized Type III error: correctly answering the wrong question.
Here are a couple of examples:
No matter how good the research was in all of these cases, the answer still wouldn’t have helped answer the larger questions at hand. Like a swimmer in open water, the best techniques in the world don’t help if you’re not headed in the right direction. It sounds obvious, but formalizing a definition like this and teaching it while you teach other techniques might help remind scientists/statisticians to look up every once in a while. You know, just to see where you’re going.
I am rather notoriously skeptical of infographics, but I found this one from Wait But Why today and it’s completely fascinating. It’s a comparison of how many people die/have died by various causes, some natural, some not so natural.
The whole thing is huge, but here’s a taste:
I’ve been perusing this for about half an hour now, and I’ve learned about the Masada suicides, the Shensi Earthquake, and the Mao Era in China. It’s not a definitive list, but a really interesting one!
This is a series of posts featuring anecdotes from the book The Signal and the Noise by Nate Silver. Read the Chapter 1 post here.
Chapter 2 of The Signal and the Noise focuses on why political pundits are so often wrong. When TV channels select for those making crazy predictions, it turns out accuracy rates go way down. You can either get bold, or you can be right, but very rarely can you be both.

Basically, networks don’t care about false positives….big predictions that don’t come true. What they do care about is false negatives….possibilities that don’t get raised. They consider the first just understandable bluster, but the second is unforgivable. So next time you wonder why there’s so many stupid opinions on TV, remember that’s a feature not a bug.
Read all The Signal and the Noise posts here, or go back to Chapter 1 here.
This month my book was The Signal and the Noise, which I enjoyed enough that I’m doing a chapter by chapter contingency matrix series on it over at the other blog.
Sampling strategy and research design can sound really boring, until you blow through $1.3 billion dollars and have nothing to show for it. This article on the long slow death of the National Children’s Study should be assigned reading for anyone who ever wanted to know why it was so damn hard to get good research done.
Did you hear the one about all the Brexit voters furiously Googling “What is the EU?” after they voted to leave it? Yeah? That was pretty bogus. It was about 1000 people total, no one knows if their Googling was “furious”, how they voted, or if those people were even eligible to vote.
This article is from a few months ago, but it’s an interesting look at motivations and political bias. It turns out people do better on “political fact” tests when you offer them money for right answers than when they take them with no incentives. The Volokh Conspiracy discusses implications for our understanding of political ignorance.
Also from a few months ago: the Quartz guide to bad data. More properly it might be called “guide to cleaning up your spreadsheet”. If you ever actually get a large data file and don’t know how to find potential problems before you analyze it, this is a good start.
Another good guide is this list of data science books from Stitch Fix. Stitch Fix is an online personal stylist service that I just so happen to use to get most of my work clothes. They also have a REALLY active data science division that helps come up with clothing recommendations. Good stuff.
This is an interesting data visualization of the changing American obesity rates.
I actually listened to this one, but there was an interesting piece on Science Friday about “differential privacy” and response randomization. The transcript is available here, and there’s some interesting discussion about honesty, privacy, and research in the big data era.
Last week I had a lot of fun talking about correlation/causation confusion, and this week I wanted to talk about the flip side: correctly proving causality. While there’s definitely a cost to incorrectly believing that Thing A causes Thing B when it does not, it can also be quite dangerous to NOT believe Thing A causes Thing B when it actually does.
This was the challenge that faced many public health researchers when attempting to establish a link between smoking and lung cancer. With all the doubt around correlation and causation, how do you actually prove your hypothesis? British statistician Austin Bradford Hill was quite concerned with this problem, and he established a set of nine criteria to help prove causal association. While this criteria is primarily used for proving causes for medical conditions, it is a pretty useful framework for assessing correlation/causation claims.
Typically this criteria is explained using smoking (here for example), as that’s what is was developed to assess. I’m actually going to use examples from the book The Ghost Map, which documents the cholera outbreak in London in 1854 and the birth of modern epidemiology. A quick recap: A physician named John Snow witnessed the start of the cholera outbreak in the Soho neighborhood of London, and was desperate to figure out how the disease was spreading. The prevailing wisdom at the time was that cholera and other diseases were transmitted by foul smelling air (miasma theory), but based on his investigation Snow began to believe the problem was actually a contaminated water source. In the era prior to germ theory, the idea of a water-borne illness was a radical one, and Snow had to vigorously document his evidence and defend his case….all while hundreds of people were dying. His investigation and documentation is typically acknowledged as the beginning of the field of formal epidemiology, and it is likely he saved hundreds if not thousands of lives by convincing authorities to remove the handle of the Broad Street pump (the contaminated water source).
With that background, here are the criteria:
Even though Snow didn’t have this checklist available to him, he ended up checking most of the boxes anyway. In particular, he proved his theory using strength of association, coherence, consistency and specificity. He also raised questions about the rival theory by pointing to the lack of dose-response relationship. Ultimately, the experiment of removing the pump handle succeeded in halting the outbreak.
Not bad for a little data visualization:
![]()
While some of these criteria have been modified or improved, this is a great fundamental framework for thinking about causal associations. Also, if you’re looking for a good summer read, I would recommend the book I referenced here: The Ghost Map. At the very least it will help you stop making “You Know Nothing John Snow” jokes.
If you’re looking for a little fun reading on this long holiday weekend, I would like to point you to a series of posts Ann Althouse has put up over the past couple of days. It’s not stats related, but touches on some of my other favorite topics: bias, certainty, and memory.
Act 1: Poetic Justices and Questionable Citations
Linda Greenhouse writes an Op-Ed for the New York Times, in which she complains about the “lack of poetry” in the recent Supreme Court Whole Women’s Health vs Hellerstedt decision. Greenhouse compares it to the decision Planned Parenthood vs Casey, written 24 years earlier.
The next day, Ann Althouse blogs about the article, noting that Greenhouse attributed the line “Liberty finds no refuge in a jurisprudence of doubt” from Planned Parenthood vs Casey to Anthony Kennedy. Althouse points out that the line was taken from a jointly written decision, and that to attribute it to only one justice (Kennedy) is not correct.
Act 2: Challenge Accepted
Ann Althouse posts a follow-up post after Linda Greenhouse emails her to dispute the quote mis-attribution charge. In her email, Greenhouse cites her source for attributing the line to Kennedy: the Jeffrey Toobin book “The Nine” and her own presence in the courtroom the day the justices read the Planned Parenthood vs Casey decision 24 years earlier. She recounts Kennedy leading off with the line in question, and the stir it created in the courtroom. She asserts that the act of reading the line verifies that he was the author. She ends the email with the line “Of course you are completely free to trash my opinions and my writing style. I would caution you against challenging my facts.”
Althouse, choosing to ignore that last part, located the original recording of the reading of the decision. She discovers that not only did Kennedy not lead off, but neither he nor anyone else reads the line that Greenhouse so clearly remembers hearing.
The book in question does attribute the line to him, but has no named source for that information.
Part 3: We’re All a Little Wrong Sometimes, Aren’t We Though?
Confronted with the recording that shows her memory was incorrect, Greenhouse emails Althouse again, conceding that “I guess it’s fair to say that each of us was right and each of us was wrong.”
Althouse posts that email, along with her complete rejection of Greenhouse’s conclusion here. She (Althouse) ends her post with “I didn’t say anything that was wrong. I have a way of blogging that keeps me out of trouble like that. I don’t make assertions about things I don’t know.”
Epilogue: One of my favorite books is Mistakes Were Made (But Not By Me) by Carol Tavris1. There’s a tremendous amount of research in to how and why we rewrite memories, and the book covers a lot of those reasons. The main takeaway here though is that we all need to guard against created memories and overconfidence in our facts….ESPECIALLY if you’re going to be writing for the New York Times and PARTICULARLY if you’re challenged.
The point of who exactly wrote that original line is a minor one to many people. Kennedy was certainly involved in with the decision, so naming him as a solo author isn’t that out there. If Greenhouse had merely cited the book that mentioned the line as his, I would never have thought twice about it. However, when she cited her own vivid recollection that turned out to be completely wrong I have to imagine nearly everyone reading the saga started questioning her more seriously.
To her credit, Greenhouse did fully admit her shock at discovering her memory was incorrect. Hopefully the lesson for all of us here is to be very cautious when we rely on an emotionally charged memory, and EXTRA cautious when we tell someone not to challenge our facts.
1. Conservative readers be warned: in the very first chapter of the book Tavris lets some pretty liberal biased statements through as fact. She cuts this out (I think) after the first chapter, but it’s really bugged at least one person I’ve recommended the book to. I think it’s worthwhile despite that, but YMMV.↩
Welcome to “From the Archives”, where I dig up old posts and see what’s changed in the years since I originally wrote them.
Last week, while researching my post about definitions you should remember while discussing mass shootings, I came across a post from January of 2013 that warranted further investigation. It was my take on a Bloomberg News article that projected that by 2015 automotive deaths would surpass gun deaths. They had showed this chart:

My primary grouse was that they seemed to be extrapolating the 2015 data from the 2008 and 2009 data. I decided to take a look and see how the Bloomberg prediction had turned out.
Interestingly enough, at this point it appears to be a statistical tie. The Violence Policy Center has a chart up through 2014 showing a slight lead for motor vehicle deaths:

The Washington Post OTOH, gave them a tie (rates reported per 100,000 people):

According to this post, the numbers for gun deaths ended up being 33,599 and the car deaths were 33,736. It is interesting to note that Bloomberg underestimated the car deaths by a little less than 2,000 /year, and the gun deaths by about 600/year. So they were wrong in their assumption that motor-vehicle deaths would continue to drop at the same pace they had been, but right in their assumption gun deaths would continue to rise. I’ll give myself half credit on this one. Of course, we do have one more year to go before we get the 2015 data, so I could still entirely eat crow.
It’s worth noting that the rise in firearm death through 2014 was entirely due to an increase in suicide rates. Homicide rates actually decreased during that time:

Someone remind me to check back in next year to see where we went with this!
When I first started blogging about correlation and causation (literally my third and fourth post ever), I asserted that there were three possibilities whenever two variables were correlated. Now that I’m older and wiser, I’ve expanded my list to six:
The obvious conclusion is that years spent blogging about statistics directly correlates to the number of possible ways of confusing correlation and causation you recognize.
Anyway, I’ve talked about this a lot over the years, and this lesson is pretty fundamental in any statistics class…though options #3 and #4 up there aren’t often covered at all. It’s easily forgotten, so I wanted to use this post to pull together an interesting example of each type.

Causal inference, not for the faint of heart.