You can’t misquote a misquote

/ bs king / 2 Comments

Yesterday I talked about sensational statistics and to always verify that there’s no missing adjectives that would change the statistic.  It was thus a bit serendipitous that today I happened to hear a debate about a misquoted statistic, and whether the quote or the misquote was more accurate.  It was on a podcast I listen to, and it was about a month old (sometimes I don’t keep up well).

It was happening around the time the contraception debate was at it’s most furious (see what I did there?  It was a federally mandated coverage of contraception debate, to give you all the adjectives).  Anyway, at the time the statistic about the prevalence of birth control usage among Catholic women was getting tossed around quite a bit.  The statistic, in it’s most detailed form, is this:  98% of self-identified Catholic women of child bearing age who are sexually active have used a contraceptive method other than natural family planning at some point in their lives.

Now, this stat rarely got quoted in it’s entirety.  First, I always think designating that the religions is self identified is important.  The women answering this survey didn’t have to clarify if they thought they were good Catholics, just Catholic.  Second, the “sexually active” got glossed over as well, despite the fact that it probably cuts down the numbers at least a bit (for young adult Catholics, to approximately 89% of respondents).  Third, “at some point”.  The study’s authors have justified this qualifier by arguing that if a woman is on birth control for years, then decides to start trying to have children and goes off of it, she would have been excluded.  Critics have argued that this strategy was designed to include women who may have tried it, decided it was wrong, and stopped.  Both have a point.

That being said, I most often heard this being quoted as “98% of Catholic women use birth control” or sometimes even “98% of Catholics use birth control”.  

It was that last phrase that got the debate going on the show I was listening to.  Person 1 argued that it annoyed him that people kept dropping the “women” part of the quote.  Person 2 shot back that it actually drove him nuts that people felt the need to add it.  He argued that for every straight female using contraception, there was by definition a straight man using it.  Unless one presumed a statistically significant number of women were misleading their partners, 98% of Catholic men were also using birth control (of course, even if they were being misled, they were actually still using it…just not knowingly).  Since according to Catholic doctrine the contraception mandate is for both genders, both parties are therefore guilty.

I liked the debate, and would be totally fascinated to hear the numbers on men who have used (or had a partner who used) contraception.  I am curious if a significant number don’t know, or would claim not to know.  I still think that clarifying “women” in the quote is fine, as it’s who the study was actually done on.  In my mind extrapolation should always be classified as extrapolation, not an actual finding.

Also of note, this was an in-person survey.  That’s always useful to realize that every answer given in a survey like this had to verbalize their answers to another person….important when the topic is anything highly subject to social pressures.  For a further breakdown of issues with that study, see here.

Beware the Adjective

/ bs king / 4 Comments

My tax return showed up in my bank account this weekend, which is always nice (even if it was my money to begin with).  It brought to mind a few months back when people were big on the “50% of American households don’t pay any federal income tax” statistic.

Now, that was an interesting statistic, and one that no doubt caused a lot of emotion.  I mean, heck, this is my percent breakdown of taxes paid for 2011 (excluding sales-linked taxes…that retrospective would have taken all week):

Edit: My labels got a little hinky, so assume federal tax = federal income tax and state tax = state income tax.  So yes, life would have been a great deal cheaper if I could have avoided federal income tax.

Anyway, I was thinking about this when I stumbled across this chart:

Along with this post explaining that many of the households not paying taxes were actually older workers.  Interesting, but economic data is so easily manipulated it doesn’t normally catch my attention (example: no where on this graph does it indicate how large each population slice is…I’m sure there are far fewer people represented at the end of the graph than at the middle).

Anyway, what this jogged my memory about was how this statistic got quoted by many at the time.  Rick Warren was one of the more notable examples, but many people made the mistake of stating “half of all Americans pay no taxes”.  The “Federal Income” part of that phrase makes a huge difference.

I’m certainly not saying that everyone who misquotes a stat does so intentionally.  Many times it’s innocent, and thus it’s something to keep in mind when you hear a crazy statistic from anything but the source.  Politicians and other public speakers do just flat out miss words sometimes.  There are some pretty horrifying stats out there that become much more reasonable when the correct modifiers are put back in their place.  

Friday links for fun – 4.6.12

/ bs king / 3 Comments

Two fun articles taking on bad data:

This one covers everything I will probably ever say in this blog, but with less pizzazz.

This one is trying to stop bad data before it starts.  Don’t try to make things in to a scientific experiment if you have to fudge around things to do it.  Just call it a model.  I like that.

That’s some bad data, bad to the bone

/ bs king / 5 Comments

Not the most useful data on the planet, but fun never the less….especially if you are a data geek married to a metal head.  Not that I’d know anything about that.

Heavy metal bands per capita for every country except Bhutan:

In case you’re curious, here’s an article explaining more, including the actual numbers used.

Thanks to some research carried out by me and my wonderful husband, we discovered that Bhutan now has 1 metal band that was formed in 2008.  Their name is Metal Visage.  Here’s a review.  Oh, and if you’re super curious, here’s a video.  I have no idea if they’re good or offensive or what, as my dog started barking as soon as I hit play, but my husband assures me they are better than Ugra Karma (one of Nepal’s 12 metal bands)

Anyway, not much to criticize here, as sadly this is probably more accurate than most of the studies I write about.  I did find it amusing that I saw a comment about this where someone was greatly disturbed that the CIA world factbook was cited as a source.  I considered politely explaining to them that that was probably where the population numbers came from, not the metal band numbers, but I decided not to.  Read ALL the sources folks, thank you.

Opinions, everybody’s got one

/ bs king / 3 Comments

I was listening to a management podcast recently where a man named John Blackwell was being interviewed.  He was talking about how he was constantly reading things about how the whole workplace was changing, but he was getting curious as to why he felt like the companies he worked with weren’t reflecting this.  When he tried to investigate, he found out that the ongoing surveys commonly used in British management journals (can’t find a link) were being done on the “up and coming business leaders”.  When he looked in to what that meant, he realized it was people who were second year MBA students.

The problem with this, of course, was that this was asking people not in the workforce what the workforce was going to look like 10 years from now.  They found, not surprisingly, that young people in grad school tend to be very optimistic about things like “working from home” or “flex time” when they’re in school, but when they got in to business, they towed toed the line.  Thus, every survey done was essentially useless.  
This all reminded me of a conversation I got in to several years ago when I was working the overnight shift.  Someone had brought in a magazine (People or Vogue or something like that) and they had a ranking of the 100 most beautiful women in Hollywood.  Drew Barrymore was number one that year, and one of my (young, male) coworkers was actively scoffing at that.  “She’s unattractive,” he stated definitively.  “All the guys I know think so too.”
Now, I was feeling a little feisty feminist that night, so I thought about how to challenge him on that.  Leaving aside that “Hollywood unattractive” would still turn heads in any average crowd (and be more attractive than any girl he’d dated), something about his comment irked my data side.  “So maybe the voting was done by women,” I replied.  
He was floored.
I noted that it was not a men’s magazine that ran the story, so really women’s opinions of other women’s attractiveness would actually be more relevant to this list.  Furthermore, as most of the leading women in Hollywood make their money on romantic comedies, professionally women’s opinions of their attractiveness (which presumably included a certain likeability factor) would actually matter more than men’s.
I was fascinated that this clearly disturbed him.  It had clearly never occurred to him that straight men may not be the target audience for female attractiveness, or even that the relevance of his opinion might get questions.  He wasn’t trying to be a jerk, he was legitimately confused at the whole idea.
A long intro, but the bigger point is important.  In any opinion survey or research, it’s important to figure out whose opinion is most relevant to what you’re trying to get at and why.  When it comes to law and public policy questions, I think every voter is relevant.  When it comes to workplace trends?  You may need to narrow your sample.
Sampling bias is a huge problem in many contexts, but my primary one for today’s post is when the survey was not conducted with the end in mind.  For any sample, you have to figure out how much your subject’s opinions actually matter given what you’re trying to find out.  In social conversation it may be interesting to find out what a particular person thinks of a topic, but for good data, show me why I care.

Stand Back! I’m going to try SCIENCE!

/ bs king / 5 Comments

Today I discovered that my favorite webcomic (xkcd.com) actually has a special comic up if you check it from my employer’s server.  Turns out the artist’s wife is a patient, doing well, and he wanted to show some love.  This post is thus titled for this shirt, which would make an awesome Christmas present for me, even in April.

Anyway, this weekend I saw this story with the headline “Study: Conservatives’ Trust In Science At Record Low”.

My first thought on seeing this was that the word “science” is a loaded word.  I mean, I’m as much a science geek as anyone.  Math’s my favorite, but science will always be a close second.  But do I trust science? I’m not sure.  Something really bothered me about that question, but I couldn’t quite put my finger on it until I read this post on the study from First Things today.  

My love of science makes me a skeptic.  I makes me question relentlessly and then continuously revisit to figure what got left out.  I don’t trust science because not trusting your assumptions is science done right.  If we could all trust our assumptions, what would we need science for?  This is the problem with vague questions and loaded words.  Much like the discussion in the comments section of this post where several commenters weighed in on the word “delegate” in relation to household tasks, it’s clear that people will interpret the phrase “trust science” in many different ways.

Some might say it means the scientific method, scientists, science as a career, science’s role in the world, or something else not springing to mind.  Given the vagueness of the question though, I would have a hard time actually calling anyone’s interpretation wrong.  Mine is based on my own bias, but I would wager everyone’s is.  So isn’t this survey more about how we’re defining a phrase than about anything else?

I thought my annoyance was going to end there, I really did.

Then I looked at the graph with the story, and had no choice but to get annoyed all over again.

That’s what I get for just reading headlines.

So over the course of this survey, moderates have consistently trusted science less than conservatives for all but four data points?  Why didn’t this get mentioned?  I found the original study and took a quick look for the breakdown: 34% self identified as conservative, 39% as moderate, and 27% as liberal.  So 73% of the population has shown a significant drop off in “trust of science” and yet they’re somehow portrayed as the outliers?  Science and technology have changed almost unimaginably since 1974, and yet liberal’s opinions about all that haven’t changed*?  Does that strike anyone else as the more salient feature here?

*Technically this may not be true.  I don’t know what the self identified proportions were in 1974, so it could be a self-identification shift.  Still.  This might be that media bias everyone’s always talking about.

Book Recommendation – How to Lie With Statistics

/ bs king / 2 Comments

If one has free reading time or just really likes lists (and boy do I love a good list!) the Personal MBA reading list is pretty darn cool.  It claims to give you knowledge equivalent to an MBA in 99 books, without any of the crippling debt.  I’m about 10 books in, and there’s some really great stuff on data, statistics, analysis and presentation.

One of the classics of course is How to Lie With Statistics.  It’s a great book, easy reading, though the examples are outdated to the point of near distraction (salaries list at $8000/year, that sort of thing).  Still, clear and concise, and shows you that bad data has been around for quite some time.

One of my favorite moments is when he goes after Joseph Stalin for his bad data….in retrospect that kind of feels like saying Hitler was a bad dresser.  Still, pretty interesting to see where the misinformation starts.  This book should be required reading for everyone.

Arguments and Discussions…learning the rules

/ bs king / 3 Comments

I was struck by something that commenter Erin mentioned in response to my post about data that I hate.   She ended her comments with this:

I teach this stuff to my AP students…I love trying to get them to understand how to break apart political rhetoric and other arguments around them. I figure even if we disagree wildly in politics or social issues, at least I’ll have an intelligent opponent to argue with someday. 

I like that, because I fully endorse that approach to life.  That’s part of why I wanted to do a blog like this.  Quite some time ago, the Assistant Village Idiot put up a post I liked very much (and can’t find now…circa 2007?) about how far too many people treated their political opinions as though they were defense lawyers….never giving an inch, never admitting that anything they had said or cited could be wrong or skewed.  This makes lots of people defend really stupid things.

In my office, this flowchart hangs just to the right of my computer:

I often have fantasies of taking it down during debates and serenely handing it to the other person whilst telling them to try again.  Sadly, I have never done this.  The fantasy keeps me going some days though, doubly so in political debates.
Though I’m probably preaching to the choir hear, I feel the need to state for the record:  Just because something you cited is wrong does not mean you are wrong.  You can keep your belief while also admitting that something that agrees with you is a load of crap.  That actually makes you a better person, not a worse one.  This is not an April fools joke, people actually can operate like this.

Say What?

/ bs king / 5 Comments

I can’t figure out if this is the worst statistic I’ve read this week, or just the most poorly phrased good statistic.  I’m leaning towards that first one:

“….more than half of students earning bachelor’s degrees at public colleges – 56 percent – are graduating with $22,000 of debt, on average.”  –Nancy Zimpher on CNN.com

If I’m reading this correctly, they tossed out everyone at private college, then anyone who didn’t graduate, then (most disturbingly) 44% of who was left?  Why did they get the boot?  I mean, if we’re just tossing out arbitrary numbers of students, can’t we get any average we want?

Nancy Zimpher, what are you up to???