Chapter 9 has some interesting anecdotes about the quest to create a chess program that could beat Gary Kasparov. It covers some of the limits of humans and machines, and how they are almost better when used in tandem.
Author: bs king
Team USA, Women and the Olympics
With the Olympics officially coming to a close this past Sunday, a reader contacted me and asked about the performance of the female athletes of Team USA. He was curious if the number of medals won by US women in the Olympics had increased as a percentage, absolute count or both since the passing of Title IX. In a year that female athletes got a substantial amount of coverage, this seemed like an interesting question so I ran a few numbers.
Some caveats: Figuring out how many events there are each year is tougher than I thought, especially for the early Olympics. Because some of my data sources disagreed, some of these percentages might be off. Additionally, I may be slightly off on the percent won by women by a few points. In both the winter and summer Olympics, there are some mixed gender events…think paired figure skating. I couldn’t figure out how the data I pulled below was counting that, so it could vary a bit. Since there’s only 3 of those events in the winter Olympics and 9 in the summer, I decided to let it go. Finally, this only counts events, not athletes. Michael Phelps counts as a medal in each of his events, but the relay team also only counts as one. So basically, this reflects the gender breakdown by medal count, not by the number of male or female medalists we have. So Team USA basketball is one medal for each gender, despite making quite a few people “gold medalists”. All data sources at the bottom of the post
Okay, so let’s take a look!
First, how has the percent of Team USA medals won by women changed over time?
Each of the lines is 8 years, if you’re trying to orient yourself. For the youngsters, the dip in 1980 is because we boycotted that year. As you can see though, the percentage has gone steadily up.
But what was the driver of this? The initial asker suggested the driver was Title IX, but I wondered if it might be more closely correlated with the expansion of women’s events. Of course neither of these would be entirely independently causal….we know the social forces that drove one likely drove the other. Anyway, here’s how the percentage of medals available to women varied with the percentage of medals won by women on Team USA for the Summer Olympics:
And winter:
The winter medals variability is almost all because of the low medal counts. The two years they were high were actually not very high medal count years (5 and 9), but basically the men only got 2. I ran a quick regression and the r-squared for the Summer Olympics is around .75, and the Winter about .4.
For juxtaposition, here’s the number of female NCAA Div 1 athletes superimposed on a different scale:
I’m not going to do the overall regression because correcting for the multicollinearity (aka, a regression with two factors that are correlated) can be a bit of a hassle, but I’m guessing it’s the expansion of events driving the medal count more than the number of D1 female athletes. However, it may be the increased number of athletes allowed the US to immediately take advantage of every expansion in medal events. Additionally having more talented female athletes probably incentivized the IOC to add more events.
Confusing correlation, but a great question!
The Team USA Medal Count came from here. The count for female athletes came from here. The number of events came from here for winter and here for summer. The number of events available to women is here. NCAA athlete counts are here.
Accidental Polymath Problems: 10 Subjects You Study Before You Find the One
After my comment last week that I’d sort of friend-zoned physics, I got to thinking about how many different subjects/career choices I stumbled through during my 20s. It’s incredibly interesting to me that even though society has started allowing (and frequently even encouraging) people to wait longer and longer before finding “the one” for marriage, we still put a lot of pressure on people to know exactly what they’re interested in by the age of 18…or 22 if you’re a little behind. Clearly college debt is a huge driver of this, but I do meet a bizarre number of high school students who really think most people figure out “their passion” before they’re even old enough to drink. While clearly there are plenty of people who find what topics they want to study early, I’d like to propose that the whole thing is a little more like dating then we normally think of it.
When I mention “subjects” here and “study”, I am covering a lot of ground. Studying could mean formally studying in school, or getting books out of the library, watching documentaries or talking to a lot of people in the field. While I mention careers, I’m not directly equating intellectual pursuits to careers or work because some people really don’t get to equate those two. It’s an unfortunate reality that many of us have to prioritize paying the bills over feeding our minds, and if you ever find yourself doing both at once you are incredibly lucky. With those caveats, and the knowledge that this is based on nothing but my own experience and that of my friends, here’s the 10 types of subjects you study before you find “the one”:
- The Celebrity Crush Grey’s Anatomy. ER. Bones. CSI. Law and Order. Let’s face it, some professions get all the girls. No seriously, who among us with a television set hasn’t at some point hasn’t developed a crush on an entire profession/field of study? When I was 4 years old I watched a PBS special and spent months telling everyone I wanted to be a paleontologist and begging my parents for dinosaur books. Two decades or so later I binge watched the first 5 seasons of Bones I spent a solid 3 weeks desperately wanting to be a forensic anthropologist and reading every book my library had on the topic. While sometimes these can spark real career choices, most of the time the fantasy is better than the reality. I mean, I still adore dinosaurs but I would NEVER have the patience to catalogue a dig site. Some things just look better from afar.
- The Challenge Subject Similar to the celebrity crush, but you actually encountered it in your real life. This is the subject or path you pursue because you’re not sure you can actually get it. It’s not that you’re not legitimately interested, but if you’re honest with yourself it’s really your competitive streak that’s pushing you through. The truth will hit you when you finally mastered the subject, only to promptly realize that now you really never want to talk about it again. Want an example? Ask me about my biomedical engineering degree.
- The One that Requires Way Too Much Commitment Okay, so you found a subject you really like, and you think “hey, maybe I’d like to consider this as a profession”…and then you realize exactly how much work that would take. You like the subject, but the idea of working hundreds of hours or going to school for a decade to study it further strikes you as waaaaay too much commitment. It’s ready to settle down, and it looks nice, but you just can’t be tied down like that. You have too many other interests, and there’s only so many hours in the day.
- The Summer Fling This is the subject you absolutely love, but only because of the setting you encountered it in. Maybe you got to learn about archeology while studying abroad in Egypt or you had an amazing professor who made an otherwise boring subject unbelievably interesting. When you try to pick this subject back up again, you realize that in a more mundane environment it actually is kind of boring. Ah well, at least you have the memories.
- The Artist This is the subject you love with all your heart, but you realize it will always be a bit of a free spirit. Maybe it’s literally an artistic field mashed up with another topic, or maybe it’s a subject you’re just kind of making up as you go along (like, say teaching people how to read science on the internet) but it doesn’t fit neatly in any sort of traditional box. It’s more exciting to you than almost any other topic, but no one else understands what you see in it and it’s DEFINITELY not a program of study anywhere.
- The Friend With Benefits This is the subject that comes really naturally to you without ever really having to put much effort in. It doesn’t excite you much, but people will pay you to do it and the effort is minimal. For me, this is quality and regulatory. You want someone to memorize obscure regulations, recite them at you when you step out of line, check your work and tell you your faults? I’m your girl. I can do that in my sleep. Ask anyone who’s ever lived with me. Anyway, this one doesn’t require a lot of investment either because it comes so naturally or you’re already qualified for it, but you know you could walk away at any time and never think about it again.
- The Safe One Related to the friend with benefits, but you committed to this one. It doesn’t excite you, but you think you can always find work in this field and it’s not terribly stressful. You sometimes think about leaving, but everything else seems less certain. Tends to work out pretty well unless the field totally collapses on you.
- The Friend Zoned This is the thing you always enjoying hearing about, but simply never want to commit to doing much reading about…..despite a bit of a feeling you should give it a chance. Maybe it’s a field where you could make a lot of money, or something your parents think you should try, but you just can’t bring yourself to try it out.
- The “We’re Better As Friends” A little like the friend zone, but this one is a mutual decision. It’s the subject you like studying and love to be around, but as soon as anything formal or structured was required you did terribly and bailed. Still, having it in your life makes your life richer, as long as it’s on low pressure terms. Interestingly, I try to convince many people that statistics should fall in this category for them. You don’t have to like studying math formally in order to benefit from having a little more statistics in your life.
- The “why did we never work out” This is the subject you always think is pretty great, but really spend very little time studying. You like it, but every time you find a free moment, you forget it exists, or it’s only offered as a class the one semester you’re already overloaded, etc etc. For me this is epidemiology. I’ve taken classes in it, it’s a natural fit, but I never quite seem to follow up. I really should give it a call sometime soon.
Of course the nice thing about intellectual pursuits is that you actually can juggle multiple different subjects at once with a lot less potential for drama than if you tried that while dating. For example, my current job is a mash up of my true love (statistics, analytics and process improvement) my friend with benefits (quality and regulatory) and the safe one (computer systems). My blogging is The Artist, and it gives me a place to research all my thoughts that don’t fit in any other box. I think acknowledging how many different types of intellectual pursuits there are (and how much you can learn from all of them!) could be useful for kids still trying to figure things out. Just like dating can help you hone in on what you want in a spouse, studying a lot of subjects can help you find that sweet spot of “things you want to talk about” and “things people want to pay you to talk about”.
Plus, isn’t the world a little more fun when you consider every new book a low key blind date?
The Signal and the Noise: Chapter 8
The eternal struggle continues.
5 Things I’ve Learned From Reading About Problems in Physics
One of my favorite things about getting an engineering degree was the amount of basic science classes I had to take. It gave me at least a dilettante’s knowledge of quite a few scientific fields, and I’ve always enjoyed using that background to keep at least half an eye on other scientific fields. Of all of those fields, my particular favorite is physics. I always loved physics in that “I’m so glad to see you, but let’s just be friends” kind of way, and I try to make sure I read at least a book or two a year about it.
A few months ago I read Lee Smolin’s book “The Trouble With Physics“, and was intrigued to read a breakdown of some of the current (well, ten years ago now) problems in the field. It got me pretty stressed out about string theory, which is not a problem I had expected to have that week. I digress. Anyway, this physics anxiety got a little worse when James over at I Don’t Know But posted about how physics needed some new ideas, and then he left me this link about the rather embarrassing 750 GeV diphoton excess incident. He compared the whole debacle to priming studies, which seemed fair. Anyway, since blogging is the primary way I deal with my science and statistics related anxiety problems, I thought I’d put together a post on why I actually love reading about issues in physics. Ready? Let’s go!
- Reading outside your field gives you a new perspective on errors Most of my working experience is in healthcare, and one of my degrees is in a psych field. When you’re familiar enough with your field, it can be pretty easy to figure out what all the most common errors are. Since professions tend to attract people who think similarly, it stands to reason that fields will all have certain errors they are particularly susceptible to. Reading outside your normal field is a good way of realizing what problems are actually pretty universal, which ones you may never have thought of, and (ideally!) how other fields have dealt with some problems. Additionally, it’s really easy to see the issues in fields that tend to capture headlines (psych, nutrition, etc), while other fields that are less accessible can seem like they don’t have any problems. Reminding yourself this isn’t true is kind of reassuring.
- Statistical noise is a problem for everyone One of the reasons I went in to statistics in the first place was the allure of how many different fields had to use it. At the time I loved the idea of learning a topic that basically every single discipline had to use. I still do. The link James mentioned originally was about a topic I won’t even pretend I can explain (750 GeV diphoton excess) but focused on a problem I’m REALLY familiar with: over-interpertation of statistical noise. Yeah, basically theoretical physicists published about 500 papers on a phenomena that appeared to be true but then didn’t replicate. Oops. In their defense though, it was a really large anomaly in an area that was theoretically plausible and that they’d had success with before, which is pretty much the perfect storm for confirmation bias.
- So is failing to check basic assumptions. If I had to make a complaint about the way we teach statistics to kids, I would argue that the biggest error we make is not emphasizing to them how important it is to check basic assumptions. Textbooks are always reminding you that you have to make sure assumptions x, y and z hold true before you can use certain equations….then they just let you assume all those things for the rest of the class and send you on your merry way. The real world doesn’t work like this. That was evident back in May when a couple of retraction notices came out from the New Journal of Physics. There was no intentional misconduct, but the authors had assumed the data was symmetrical without checking that assumption. In Smolin’s book, he discusses a few fundamental string theory assumptions (mentioned in the second column on page 2 of this review) that didn’t actually have experimental evidence behind them, despite most people assuming otherwise.
- The goal is to push the limits. In my priming studies post, I mentioned that pushing the limits and studying the fringes of a field is a feature, not a bug. That sentiment is echoed in this interesting article about “The Data That Threatened to Break Physics“. It discusses the struggle of a researcher to cope with completely unexpected results that run contrary to conventional wisdom. In the case of superluminal neutrinos, the results turned out to be the fault of a faulty cable, but the lead researcher quite rightly asks what people thought he should have done differently. Suppressing a potentially controversial result is not really something we want to encourage, and the upshot of that may be that we end up with retractions. To quote the lead researcher: “The worst data are better than the best theory. If you look for reasonable results, you would never make a discovery, or at least you will never make an unexpected discovery”.
- Even when you stop studying people, you can’t get out of dealing with people. At it’s heart, science is as much about bias management as it is about discovery. It is really difficult to do much of the latter if you don’t do the former. In Smolin’s book, two of the most fascinating chapters were “How Do You Fight Sociology?” and “How Science Really Works” (covered a bit in this review). Smolin reviews how tenure, grant related politics and even just plain old ego and groupthink can influence what scientific theories get money and attention. All this occurs without any outside social pressure, since of course it doesn’t matter to most lay people if string theory is true or not. Smolin proposes that to counter this, universities should reserve some money/positions for those who are actually quite polarizing in their work. He proposes that we invest in scientific ideas like many stock market investors work: put most of your money in safe things, but put some of it on ideas that look a little nuts. Nicholas Nassim Taleb famously calls this “the black swan approach”. I’ve heard worse ideas.
So there you have it, and if you have any good physics book recommendations, I’m always looking!
5 Interesting Reasons Priming Studies Go Wrong
Last week, commenter Christopher B left an interesting comment on my post about masculinity threats and voting that made me realize I wanted to do a bigger post on priming studies in general. Priming studies have come under a lot of fire in the past few years, and they have the unfortunate distinction of being called (by some) the “poster child for doubts about the integrity of psychological research“. So what’s going on here? What are these studies and why do they go wrong so often?
Well, as Christopher B pointed out, it’s not because priming isn’t a thing. Priming is typically defined as “an implicit memory effect in which exposure to one stimulus (i.e., perceptual pattern) influences the response to another stimulus“. In other words, something you see or do at one point unconsciously biases you to act differently at a later date. Some of these could be pretty straightforward. If you see a list of words that containing the word “dog” and then someone asks you to name an animal that starts with the letter w, you will probably be more likely to say “wolf” than “walrus”. Lots of marketers attempt to use priming-like effects to get people to buy more or differently than they would have otherwise. There’s even some efforts to see if getting alcoholics to physically (well, in the form of video games) practice pushing away drinks helps lead to lower rates of relapse. I think most of us would accept that your brain does have a bit of an auto-suggest type system, and most people would accept it can probably be manipulated subtly. So where’s the problem? Well, in addition to the p-value and replication issues I’ve raised before, here’s some other reasons things have gone haywire:
- A lot of work gets done at the edges The examples I’ve given above are pretty straightforward, much more straightforward than most of the priming studies that get attention. It’s unsurprising that most researchers aren’t as interested in obvious and straightforward effects, but rather increasingly subtle and indirect effects. For example, in the study I talked about last week, the researchers didn’t ask men to consider a world where women reigned supreme, but rather asked “who makes more money, you or your wife?” The effects they’re interested in are subtle and subconscious, and obviously there’s a limit to how far that can be stretched. Finding that limit is part of the goal. Unfortunately, the edges of any phenomena are going to be those most susceptible to signal and noise problems, and priming researchers got in the habit of casting a broad net at the edges of their conceptual field. Let’s just say that if your field ends up lending itself to parody this pointed, you may want to take a step back.
- Primes themselves are subject to bias There’s a great paper on priming studies out of Stanford called “Why many priming results don’t (and won’t) replicate: A quantitative analysis” that points out a lot of logistical reasons priming studies don’t work. One of the more interesting issues they raise is that it’s really freaking hard to actually establish how strong a prime is, and the choices are made by things that are obvious to the researcher, not necessarily the subjects. For example, the most famous priming study primed undergrads with words associated with the elderly like “Florida” or “sentimental”. The authors of the quantitative analysis paper pointed out that the frequency of those words being associated with “elderly people” has actually been decreasing in the past several decades. So basically things that will be “obvious” associations to a 40 year old researcher may not be as obvious to their 18 year old students. To give a more run of the mill example of this, think of celebrity names. If I ask you to name an actor whose first name is “Alan”, many baby boomers might say “Alda”, whereas younger Harry Potter fans may say “Rickman”. This issue also explains why these studies don’t tend to replicate in other languages.
- Age of subjects matters In addition to the word choice bias, there’s some good evidence that our susceptibility to priming may actually change as we age. When attempts have been made to actually create new word association relationships for people, age is a confounder:
Image from the quantitative analysis paper. The authors in that paper propose that this will translate in to young people being much more susceptible to subtle primes, with older people only responding to more direct ones. This age discrepant behavior is not always accounted for. - Experimenters can prime just as well as their actual primes One of the main blows to priming studies came when a group of researchers attempted to replicate the “hear words about old people/subsequently walk more slowly” study. In a study called “Behavioral Priming: It’s All in the Mind, but Whose Mind?”, researchers found that priming the researcher to believe the subjects had been primed to walk more slowly caused the participants to walk more slowly. In fact the researcher’s belief made a bigger difference than the priming itself:
Turns out subjects aren’t the only ones susceptible to subtle and unconscious biases. You can read the original studies author rather grouchy response to the whole thing here, and Andrew Gelman’s eyeroll back here.
- The field did attract an unfortunate number of frauds. Maybe it was due to the headline grabbing nature of many of these priming studies, but there have been some absolutely audacious fraud cases in priming research. Diederik Staples published over 20 big studies with made up data. Dirk Smeesters also had seven. Lawrence Sanna is up to 8. Is this worse than other fields? Maybe, or maybe it’s just that these studies tended get a lot of attention. It’s not so much the fraud that casts a shadow, but the alarming realization that so many made up studies got through without question. This has led to calls for standards involving immediate replication attempts and other measures to stop bad research before it starts.
Now keep in mind, all of these reasons are over and above the normal file drawer effect and p-hacking that all fields face. Hopefully this gives you a little insight in to a few of the less obvious ways these studies can go wrong, and will trigger you to think about these things when you hear the word “prime”….see what I did there????
The Signal and the Noise: Chapter 7
Chapter 7 was about extrapolation and predictions that influence their own accuracy. One of my favorite examples was predictions about disease spread:
5 Things You Should Know About the Great Flossing Debate of 2016
I got an interesting reader question a few days ago, in the form of a rather perplexed/angry/tentatively excited message asking if he could stop flossing. The asker (who shall remain nameless) was reacting to a story from the Associated Press called “The Medical Benefits of Dental Floss Unproven“. In it, the AP tells their tale of trying to find out why the government was recommending daily flossing, given that it appeared there was no evidence to support the practice. They filed a Freedom of Information Act request, and not only did they never receive any evidence, but they later discovered the Department of Health and Human Services had dropped the recommendation. The reason? The effectiveness had never been studied. Oops.
So what do you need to know about this controversy? Is it okay to stop flossing? Here’s 5 things to help you make up your mind:
- The controversy isn’t new. While the AP story seems to have brought this issue in the public eye, it’s interesting to note that people have tried to call attention to this issue for a few years now. The article I linked to is from 2013, and it cites research from the last decade attempting to figure out if flossing actually works or not. While flossing has been recommended by dentists since about 1902 and by the US government since the 1970s, it has not gone unnoticed that it’s never been studied.
- The current studies are a bit of a mess. Okay, so if everyone kinda knew this was a problem, why hasn’t it been resolved? Well it turns out it’s actually really freaking difficult to resolve something like this. The problem is two-fold: people hate flossing and flossing is hard to do correctly. Some studies have had people get flossed by a hygienist every day, and those folks had fewer cavities. However, when the same study looked at people who had been trained to floss themselves, they found no difference between them and those who didn’t floss. Many other studies found only tiny effects, and a meta-analysis concluded that there was no real evidence it prevented gingivitis or plaque build up. Does this require more time investment? Better technique? Or is it just that conscientious people who brush are pretty much okay either way? We don’t actually know….thus the controversy.
- Absence of evidence isn’t evidence of absence. All that being said, it’s important to note that no one is saying flossing is bad for you. At worst it may be useless, or at least useless the way most of us actually do it. However, most dentists agree that you need to do something to remove bacteria and plaque from between your teeth, and that shouldn’t be taken lightly. It’s absolutely great for people to call out the American Dental Association and the Department of Health and Human Services for recommendations without evidence, but we shouldn’t make the mistake of believing that this proves flossing is useless. That assertion also has no evidence.
- Don’t underestimate the Catch-22 of research ethics. Okay, so now that everyone’s aware of this, we can do a really great rigorous study on this right? Well…maybe not. Clinical trial research ethics dictate that research should have a favorable cost benefit ratio for participants. Since every major dental organization endorses flossing, they’d have to knowingly ask some participants to do something they actually thought was damaging to them. That would be extremely tough to get by an Institutional Review Board for more than a few months. This leaves observational studies, which of course are notorious for being unable to settle correlation/causation issues and probably won’t end the debate. Additionally, some dentists commenting are concerned about how many of the limited research dollars available should be spent on proving something they already believe to be true. None of these are easy questions to answer.
- There may not be a precise answer. As with many health behaviors, it’s important to remember that flossing isn’t limited to a binary yes/no. It may turn out that flossing twice a week is just as effective as flossing every day, or it may turn out they’re dramatically different. There’s some evidence that using mouthwash every day may actually be more effective than flossing, but would some of each be even better or the same? Despite the lack of evidence for the “daily” recommendation, I do think it’s worth listening to your dentist on this one and at least attempting to keep it in your routine. Unlike oh, say, the supplement industry, I’m not really sure “Big Floss” is making a lot of money on the whole thing. On the other hand, it doesn’t appear anyone should feel bad for missing a few days, especially if you use mouthwash regularly.
So after reviewing the controversy, I have to say I will probably keep flossing daily. Or rather, I’ll keep aiming to floss daily because that has literally never translated in to more than 3 times/week. I will probably increase my use of mouthwash based on this study, but that’s something I was meaning to do anyway. Whether it causes a behavior change or not though, we should all be happy with a push for more evidence.
Men, Masculinity Threats and Voting in 2016
Back in February I did a post called Women, Ovulation and Voting in 2016, about various researchers attempts to prove or disprove a link between menstrual cycles and their voting preferences. As part of that critique, I had brought up a point that Andrew Gelman made about the inherently dubious nature of anyone claiming to find a 20+ point swing in voting preference. People just don’t tend to vary their party preference that much over anything, so they claim on it’s face is suspect.
I was thinking of that this week when I saw a link to this HBR article from back in April that sort of gender-flips the ovulation study. In this research (done in March), they asked men whether they would vote for Trump or Clinton if the election were today. For half of the men they first asked them a question about how much their wives made in comparison to them. For the other half, they got that question after they’d stated their political preference. The question was intended to be a “gender prime” to get men thinking about gender and present a threat to their sense of masculinity. Their results showed that men who had to think about gender roles prior to answering political preference showed a 24 point shift in voting patterns. The “unprimed” men (who were asked about income after they were asked about political preference) had preferred Clinton by 16 points, and the “primed” men preferred Trump by 8 points. If the question was changed to Sanders vs Trump, the priming didn’t change the gap at all. For women, being “gender primed” actually increased support for Clinton and decreased support for Trump.
Now given my stated skepticism of 20+ point swing claims, I decided to check out what happened here. The full results of the poll are here, and when I took a look at the data there was one thing that really jumped out at me: a large percent of the increased support for Trump came from people switching from “undecided/refuse to answer/don’t know” to “Trump”. Check it out, and keep in mind the margin of error is +/-3.9:
So basically men who were primed were more likely to give an answer (and that answer was Trump) and women who were primed were less like to answer at all. For the Sanders vs Trump numbers, that held true for men as well:
In both cases there was about a 10% swing in men who wouldn’t answer the question when they were asked candidate preference first, but would answer the question if they were “primed” first. Given the margin of error was +/-3.9 overall, this swing seems to be the critical factor to focus on…..yet it was not mentioned in the original article. One could argue that hearing about gender roles made men get more opinionated, but isn’t it also plausible the order of the questions caused a subtle selection bias? We don’t know how many men hung up on the pollster after being asked about their income with respect to their wives, or if that question incentivized other men to stay on the line. It’s interesting to note that men who were asked about their income first were more likely to say they outearned their wives, and less likely to say they earned “about the same” as them…..which I think at least suggests a bit of selection bias.
As I’ve discussed previously, selection bias can be a big a big deal…and political polls are particularly susceptible to it. I mentioned Andrew Gelman previously, and he had a great article this week about his research on “systemic non-response” in political polling. He took a look at overall polling swings, and used various methods to see if he could differentiate between changes in candidate perception and changes in who picked up the phone. His data suggests that about 66-85% of polling swings are actually due to a change in the number of Republicans and Democrats who are willing to answer pollsters questions as opposed to a real change in perception. This includes widely reported on phenomena such as “post convention bounce” or “post debate effects”. This doesn’t mean the effects studied in these polls (or the studies I covered above) don’t exist at all, but that they may be an order of magnitude more subtle than suggested.
So whether you’re talking about ovulation or threats to male ego, I think it’s important to remember that who answers is just as important as what they answer. In this case 692 people were being used to represent the 5.27 million New Jersey voters, so any the potential for bias is, well, gonna be yuuuuuuuuuuuuuuuuuuge.
The Signal and the Noise: Chapter 6
I’ve been going through the book The Signal and the Noise, and pulling out some of the anecdotes in to contingency matrices. Chapter 6 covers margin of error and communicating uncertainty.
There’s a great anecdote in the opening of this chapter about flood heights and margin of error. If your levee is only built to contain 51 feet of water, then you REALLY need to know that the weather service prediction is 49 feet +/- 9, not just 49 feet.
This is bad enough, but Silver also points out that we almost never get a margin of error or uncertainty for economic predictions. This is probably why they’re all terrible, especially if they come from a politically affiliated group.
The lesson here is knowing what you don’t know is sometimes more important than knowing what you do know.
graph paper diaries 