things I just made up

The Perfect Metric Fallacy

/ bs king / 4 Comments

“The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.” – Daniel Yankelovich

“Andy Grove had the answer: For every metric, there should be another ‘paired’ metric that addresses the adverse consequences of the first metric” -Marc Andreessen

“I didn’t feel the ranking system you created adequately captured my feelings about the vendors we’re looking at, so instead I assigned each of them a member of the Breakfast Club. Here, I made a poster.” -me

I have a confession to make: I don’t always like metrics. There. I said it. Now most people wouldn’t hesitate to make a declaration like that, but for someone who spends a good chunk of her professional and leisure time playing around with numbers it’s kind of a sad thing to have to say. Some metrics are totally fine of course, and super useful. On the other hand, there are times when it seems like the numbers subsume the actual goal, and those become front and center. This is bad. In statistics, numbers are a means to an end, not the end. I need a name for this flip flop, so from here on out I’m calling it  “The Perfect Metric Fallacy”.

The Perfect Metric Fallacy: The belief that if one simply finds the most relevant or accurate set of numbers possible, all bias will be removed, all stress will be negated, and the answer to complicated problems will become simple, clear and completely uncontroversial.

As someone who tends to blog about numbers and such, I see this one a lot.  On the one hand, data and numbers are wonderful because they help us identify reality, improve our ability to compare things, spot trends, and overcome our own biases. On the other hand, picking the wrong metric out of convenience or bias and relying too heavily on it can make everything I just named worse plus piss everyone around you off.

Damn.

While I have a decent number of my own stories about this, what frustrates me is how many I hear from others. When I tell people these days that I’m in to stats and data, almost a third of people respond with some sort of horror story about how data or metrics are making their professional lives miserable. When I talk to teachers, this number goes up to 100%.

This really bums me out.

It seems after years of  disconnected individuals going with their guts and kind of screwing everything up, people decided that now we should put numbers on those grand ideas to prove that they were going to work. When these ideas now fail, people either blame the numbers (if you’re the person who made the decision) or the people who like the numbers (if you’re everybody else).  So why do we let this happen? Almost everyone up front knows that numbers are really just there to guide decision making, so why do we get so obsessed with them?

  1. Math class teaches us that if you play with numbers long enough, there will be a right answer There’s a lot of times in life when your numbers have to be perfect. Math class. Your tax return. You know the drill. Endless calculations, significant figures, etc, etc. In statistics, that’s not true. It’s a phenomena known as “false precision“, where you present data in a way that makes it look more accurate than it really can be. My favorite example of this is a clinic I worked with at one point. They reported weight to two significant figures (as in 130.45 lbs), but didn’t have a standard around whether or not people had to take their coat off before they weighed them. In the beginning of the post, I put a blurb about me converting a ranking system in to a Breakfast Club Poster. This came up after I was presented with a 100 point scale to rank 7 vendor against each other in something like 16 categories. When you have 3 days to read through over 1000 pages of documentation and assign scores, your eyes start to blur a little and you start getting a little existential about the whole thing. Are these 16 categories really the right categories? Do they cover everything I’m getting out of this? Do I really feel 5 points better about this vendor than that other one, and are both of them really 10 points better than that 3rd one? Or did I just start increasing the strictness of my rankings as I went along, or did I get nicer as I had to go faster, or what? It wasn’t a bad ranking system, but the problem was me. If I can’t promise I kept consistent in my rankings over 3 days, how can I attest to my numbers at the end?
  2. We want numbers to take the hit for unpleasant truths A few years ago someone sent me a comic strip that I have promptly sent along to nearly everyone who complains to me about bad metrics in the workplace: This almost always gets a laugh, and most people then admit that it’s not the numbers they have a problem with, it’s the way they’re being used. There’s a lot of unpleasant news to deliver in this world, and people love throwing up numbers to absorb the pain. See, I would totally give you a raise or more time to get things done but the numbers say I can’t. When people know you’re doing exactly what you were going to do to begin with, they don’t trust any number you put up. This gets even worse in political situations. So please, for the love of God, if the numbers you run sincerely match your pre-existing expectations, let people look over your methodology, or show where you really tried to prove yourself wrong. Failing to do this gives all numbers a bad rap.
  3. Good Data is Hard to Find One of the reasons statistician continues to be a profession is because good data is really really really hard to find, and good methods for analysis actually require a lot of leg work. Over the course of trying to find a “perfect metric” many people end up believing that part of being “perfect” is being easily obtainable. As my first quote mentions, this is ridiculous. It’s also called the McNamara Fallacy, and it warns us that the easiest things to quantify are not always the most important.
  4. Our social problems are complicated The power of numbers is strong. Unfortunately, the power of some social problems is even stronger. Most of our worst problems are multi faceted, which of course is why they haven’t been solved yet. When I decided to use metrics to address my personal weight problem, I came up with 10 distinct categories to track for one primary outcome measure. That’s 365,000 data points a year, and that’s just for me. Scaling that up is immensely complicated, and introduces all sorts of issues of variability among individuals that don’t exist when you’re looking at just one person. Even if you do luck out and find a perfect metric, in a constantly shifting system there is a good chance that improving that metric will cause a problem somewhere else. Social structures are like Jenga towers, and knocking on piece out of place can have unforeseen consequences. Proceed with caution, and don’t underestimate the value of small successes.

Now again, I do believe metrics are incredibly valuable and used properly can generate good insights. However, in order to prevent your perfect metric from turning in to a numerical bludgeon, you have to keep an eye on what your goal really is. Are you trying to set kids up for success in life or get them to score well on a test? Are you trying to maximize employee productivity or keep employees over the long term? Are you looking for a number or a fall guy? Can you know what you’re looking to find out with any sort of accuracy? Things to ponder.

 

The Forrest Gump Fallacy

/ bs king / 9 Comments

Back in July, I took my first crack at making up my own logical fallacy. I enjoyed the process, so today I’m going to try it again. With election season hanging over us, I’ve seen a lot of Facebook-status-turned-thinkpieces, and I’ve seen this fallacy pop up more and more frequently. I’m calling it “The Forrest Gump Fallacy”. Yup, like this guy:

For those of you not prone to watching movies or too young to have watched this one, here’s some background: Forrest Gump is a movie from 1994 about a slow-witted but lovable character who manages to get involved in a huge number of political and culturally defining moments over the course of his life from 1944 to 1982. Over the course of the film he meets almost every US president for that time period, causes Watergate, serves in Vietnam and speaks at anti-war rallies, and starts the smiley face craze.  It has heaps of nostalgia and an awesome soundtrack.

So how does this relate to Facebook and politics? Well, as I’ve been watching people attempt to explain their own political leanings recently, I’ve been noticing that many of them seem to assume that the trajectory of their own life and beliefs mirrors the trajectory of the country as a whole. To put it more technically:

Forrest Gump Fallacy: the belief that your own personal cultural and political development and experiences are generalizable to the country as a whole.

There are a lot of subsets of this obviously….particularly things like “this debate around this issue didn’t start until I was old enough to understand it” and “my immediate surroundings are nationally representative”. Fundamentally this is sort of a hasty generalization fallacy, where you draw conclusions from a very limited sample size. Want an example? Okay, let me throw myself under the bus.

If you had asked me a few years ago to describe how conservative vs liberal the US was in various decades that I’d lived through, I probably would have told you the following: the 1980s were pretty conservative, the 1990s also had a strong conservative influence, mostly pushing back against Clinton. Things really liberalized more around the year 2000, when people started pushing back against George W Bush. I was pretty sure this was true, and I was also not particularly right. Here is party affiliation data from that time:

Republican affiliation actually dropped during the 90s and rose again after 2000. Now, I could make some arguments about underdogs and the strength of cultural pushback, but here’s what really happened: I went to a conservative private Baptist school up through 1999, then went to a large secular university for college in the early 2000s. The country didn’t  liberalize in the year 2000, my surroundings did.  This change wasn’t horribly profound, after all engineering profs are not particularly known for their liberalism, but it still shifted the needle. I could come up with all the justifications in the world for my biased knee jerk reaction, but I’d just be self justifying. In superimposing the change in my surroundings and personal development over the US as a whole, I committed the Forrest Gump Fallacy.

So why did I do this? Why do others do this? I think there’s a few reasons:

  1. We really are affected by the events that surround us Most fallacies start with a grain of truth, and this one does too. In many ways, we are affected by watching the events that surround us, and we do really observe the country change around us. For example, most people can quite accurately describe how their own feelings and the feelings of the country changed after September 11th, 2001. I don’t think this fallacy arises around big events, but rather when we’re discussing subtle shifts on more divisive issues.
  2. Good cultural metrics are hard to come by A few paragraphs ago, I used party affiliation as a proxy for “how liberal” or “how conservative” the country was during certain decades. While I don’t think that metric is half bad, it’s not perfect. Specifically, it tells us very little about what’s going on with that “independent” group…and they tend to have the largest numbers. Additionally, it’s totally possible that the meaning of “conservative” or “liberal” will change over time and on certain issues. Positions on social issues don’t always move in lock step with positions on fiscal issues and vice versa. Liberalizing on one social issue doesn’t mean you liberalize on all of them either. In my lifetime, many people have changed their opinion on gay marriage but not on abortion. When it’s complicated to get a good picture of public opinion, we rely on our own perceptions more heavily. This sets us up for bias.
  3. Opinions are not evenly spread around This is perhaps the biggest driver of this fallacy, and it’s no one’s fault really. As divided as things can get, the specifics of the divisions can vary widely in your personal life, your city and your state. While the New Hampshire I grew up in generally leaned conservative, it was still a swing state. My school however was strongly conservative and almost everyone was a Republican, and certainly almost all of the staff. Even with only 25% of people identifying themselves as Republican there are certainly many places where someone could be the only Democrat and vice versa. Ann Althouse (a law professor blogger who voted for Obama in 2008) frequently notes that her law professor colleagues consider her “the conservative faculty member”. She’s not conservative compared to the rest of the country, but compared to her coworkers she very much is. If you don’t keep a good handle on the influence of your environment, you could walk away with a pretty confused perception of “normal”.

So what do we do about something like this? I’m not really sure. The obvious answer is to try to mix with people who don’t think like you, aren’t your age and have a different perspective from you, but that’s easier said than done. There’s some evidence that conservatives and liberals legitimately enjoy living in different types of places and that the polarization of our daily lives is getting worse. Sad news. On the other hand, the internet does make it easier than ever to seek out opinions different from your own and to get feedback on what you might be missing. Will any of it help? Not sure. That’s why I’m sticking with just giving it a name.