Outcome metrics and the research we do not do

/ bs king / 3 Comments

I’ve spent most of last week at work trying to perfect a grant proposal that pretty much everyone in our program has to sign off on.  On Thursday, Friday and today there was a great deal of discussion about what metrics we could use to measure our outcomes, should we get funding.

It’s actually not an easy question, as the project we’re working on is a general good thing (patient education) designed to address a multitude of issues, as opposed to something more targeted.

Watching half a dozen people go back and forth about all this got me thinking about how often it is taken for granted that somewhere out there is a definition for “success” in various topics.

When I took a child development class in grad school, I remember in one of the first classes someone asked what the best parenting methods were.  Our professor replied that there really couldn’t be a consensus, because no one could agree on what would qualify as a success.  He proceeded to use religion as an example:  for parents of strong religious persuasion, a child who grew up a financially successful atheist would not necessarily be what they were going for.  Conversely, secular atheist parents might be distressed at a strong religious conversion.

There are probably scores of good studies that could have been done on parenting methods if we actually had a definition of success we could all agree on.  Too frequently, I think people overlook this point.  The reason so many strange fads in parenting can get going is because it is really really hard to prove anyone right or wrong.  Even if you try, you might just wind up with the dodo bird verdict.

If you can’t agree on where you’re going, you most certainly can’t tell people how to get there.  The studies you don’t do are often as important as the studies you do.

New Nassim Taleb

/ bs king

Apparently Nassim Taleb has a new book due out in November.

Farnam Street has a bit up from him that I liked quite a bit regarding how we process excessive data, most often to our detriment.

Best quote:

If you want to accelerate someone’s death, give him a personal doctor.

Cutting and pasting OR always check the source data

/ bs king / 2 Comments

I’ve mentioned before that I don’t like infographics.

Normally this is because the infographic itself is misleading, but today I found an equally hideous incarnation of this.

It all started over at feministing.com, where I was greeted with this graph:

This pretty much set off my alarm bells immediately.  I had quite a few questions about all of this, as the graph obviously said very little about the methodology.  Who was included?  How did they account for gaps in years worked?  Most importantly, did they control for profession?

I clicked on the link provided, which took me to this blog post on the New York Times website.  It shows the same picture as above, with an intro of the following two sentences:

We’ve written before about how the gender pay gap grows with age. Generally speaking, the older a woman is, the wider the gap between what she earns and what her male counterpart earns.

I was struck by that phrase “male counterpart”.  Were we really talking about counterparts here?  I was curious again about the profession question.  It struck me that many female dominated professions are actually “terminal” professions….i.e. the job you enter can remain pretty unchanged for years: teachers, nurses, therapists, etc.  On the other hand, many male dominate professions have far more steps on the ladder, which would be a pretty non-sexist explanation for the continued growth seen throughout the decades.

With this in mind, I went to find the methodology for the graph.  I not only found the methodology, but the rest of the infographic.

As it turns out, the profession issue was directly addressed on the original….but it was completely edited out in subsequent reprints.  Profession does have an effect on earnings growth, and the original captured that.  I’m a little concerned about how far this graphic went without all of the important qualifying information they took care to include.

 Interestingly, the NYT columnist did actually write a more comprehensive article on the topic 2 years ago that she linked to in this article, but I’m surprised she didn’t do a recap.  With the ease of transport of info on the web, I don’t think the cut and paste job is an okay thing to do.  It sets up less diligent bloggers to merely reprint, and it undermines the original work.  Someone out there is quoting this right now, having no idea that they’re missing 2/3rds of the information.

Bad data, bad.

Friday Fun Links 6-1-12

/ bs king / 1 Comment

Last time I did a list of fun links, my most curmudgeonly reader informed me they weren’t fun enough.

Fine, I’ll try again.

I don’t even attempt to touch on economic stats on this blog.  Frankly, they make me dizzy.  However, I’m excited to see that George Mason’s Stats.org is launching an Econostats website soon.  Their regular site is pretty darn good for going behind the headlines, and I’m hoping this one will be too.  Here’s a post from the guys who will be running it.

If you’re looking for summer reading for your local math/logic puzzles nerd, this might be a good choice.  Even for those not on the job market it looks fairly interesting. (Fixed the link)

Nate Silver feels about acronyms what I feel about infographics.

I’ve been trying to improve my data visualization skills lately, and I’ve been noticing huge variances in examples on the web.  Thus I liked reading this proposal for creating three different categories: data visualization, data illustration, and data art.

Speaking of data art, I bought David McCandless’s book, which is very pretty, very fun, and answers the burning question “what can facebook teach us about peak break-up times?”

Facebook breakups not of interest to you?  Maybe you’re a tennis fan?  Watching the French Open?

Real World Bad Data: The Airlines

/ bs king / 2 Comments

I hate flying.  I hate nearly everything about the entire experience really….getting to airports, the way they look, the lines, the fees, the TSA, the complete absence of food I’m not allergic to in most terminals, the boarding process, the plane itself, the proximity to other people, the feeling of being totally trapped, trying to get up and maneuver the aisles at all, and baggage claim.

Flying is terrible.

That being said, I was quite interested in reading this article that addressed why airline seats are so darn uncomfortable.  While they address the obvious issues such as increasing obesity and airline companies incentives to cram as many seats in as possible, I was struck by this quote:

In 1962, the U.S. government measured the width of the American backside in the seated position. It averaged 14 inches for men and 14.4 inches for women. Forty years later, an Air Force study directed by Robinette showed male and female butts had blown up on average to more than 15 inches…..But the American rear end isn’t really the important statistic here, Robinette says.  Nor are the male hips, which the industry mistakenly used to determine seat width sometime around the 1960s, she says.

“It’s the wrong dimension. The widest part of your body is your shoulders and arms. And that’s much, much bigger than your hips. Several inches wider.” Furthermore, she says, women actually have larger hip width on average than men.

So even back when the airlines might have made an attempt at having adequate seat size, they picked the wrong metric to play to, and everybody suffers.

I thought this was an interesting example of picking your data points.  Hip width makes intuitive sense to build a seat around, but it turns out it’s wrong.

The article also has some good discussion of perception and how moving rows closer together can give you a sense the seat itself has gotten smaller.  Interesting real world applications of statistics.

Government benefits OR definitions and the census strike again

/ bs king

Last week I got a little fascinated by the census bureau data…..and this weekend I was sent an article from the Wall Street Journal regarding yet another set of Census Bureau Data that was getting passed around.

This one addressed the number of households in the US receiving “government benefits”….apparently it’s up to 49.1%.
Now that’s a scary number, but I am always wary of the phrase “government benefits” when it’s used in a statistical context.  The problem is that it’s an incredibly vague term, and can be used to cover a myriad of programs….not all of which are what initially spring to mind.  
I first learned to be wary of this term when my dear liberal brother mentioned that some group he had been following had claimed that there was some ludicrous number of government handout programs in place today.  The number struck him as high, so he got on their website and found out that they were actually counting both federal assistance programs AND tax breaks (such as home interest deductions, student loan interest deductions, dependent credits, etc) as “entitlements”.  Thus in this case I am extra vigilant about my “find the definition” rule.
I took a look around the census website (we’ve become good friends lately) and found the list they were using as of 2008*:
  • Dept of Veteran’s Affairs – Compensation, Pension, Education Assistance
  • Medicare
  • Social Security
  • Unemployment
  • Workman’s Comp
  • Food Stamps
  • Free/Reduced-Price School Lunch and Breakfast Program
  • Housing Assistance
  • Federal and State Supplemental Security Income (SSI)
  • Medicaid
  • Temporary Assistance for Needy Families (TANF)
  • Supplemental Nutrition Program for Women, Infants, and Children (WIC)
Not a terribly surprising list, though I wouldn’t have realized that Veteran’s benefits were on there.  Even without the economy going down hill or any other expansion of programs, the Veteran’s benefits most certainly would have expanded in the past few years as people continue

Additionally, it would be important to note that only one member of the household needed to receive this in order to be counted.  That struck me because my parents and my grandmother all live in the same house, which means both of my dear hard working parents are lumped in to that 49.1% number.

Whatever your feeling about government benefits, it’s important to know exactly which ones are being counted in any list.  I’d imagine that many people who might dislike Medicaid might not care to eliminate Veteran’s Benefits, and those who don’t like TANF may very well support workman’s comp.  Just something to be aware of, especially in an election year.

*To note: the latest data I could find was from 2008.  I really hate that the WSJ doesn’t link to where the heck it got it’s numbers.  I couldn’t find the stuff they put up anywhere on the census bureau website.  I’m not doubting them, I just wonder if it would have killed them to include a link????

Time to Go Back to Work

/ bs king

But here’s my new superhero alter ego, just for laughs:

H/T to the Assistant Village Idiot, though I think he got it from his son Ben.

It reminded me of my favorite protest sign from the Rally to Restore Sanity:

Happy Tuesday!

Everything old is new again

/ bs king / 1 Comment

One of my favorite things about growing up in the family I did, surrounded by the friends my parents had, was the large amount of historical context I was fed for nearly every topic that interested me.

People like my father (who posts here as Michael) and David (the Assistant Village Idiot) were always quick to fill me in on the history of whatever topic I happened to bring up.  This always gave me a good appreciation for the story behind the story as it were, and made me truly relish a good piece of context.  Growing up in the 80’s, this was like having Wikipedia just sort of follow me around.  Come to think of it, some kids may not have appreciated that as much as I did.

I mention all this because I’m packing up my condo this weekend, and have been toting around my laptop to watch Hans Rosling’s hour long documentary “The Joy of Stats” while I work.  I highly recommend this, if not for any new stats knowledge, than at least for the examples he gives and the history lesson.

One of the more interesting points he made actually related to some of my census data posts from earlier this week, so I thought I’d pass it along.

First, if you haven’t read the comment from Glenn, the former Census Bureau employee, on my post about racial categorizations, you should.  He filled in some details I didn’t know….I would never have guessed that it was the Office of Budget Management that set racial categories for the government….and he concludes his comments with this:

Confusing? Yes. Please keep in mind that the purpose of these categories isn’t always statistical but political. Politics makes for strange statistics at times.

I liked that phrase.  I think that “The politics of statistics” should be an interdisciplinary undergrad class of some sort.

Anyway, according to Rosling’s documentary, it was actually the government of Sweden that helped invent the modern study of statistics, and they began to find it so useful that other governments started using it too.  Apparently, it was not actually referred to as statistics, but instead “political arithmetic”.

It is almost surreal to realize that up until that point, countries often didn’t know how many residents they had, or what their biggest challenges were.  An extra bonus in the film is the map of “Bastardy in England”.  Highly recommended.

Weekend Moment of Zen 5-26-2012

/ bs king / 2 Comments

Hans Rosling’s enthusiasm gets me every time.  Here, he takes on the ideas of unlimited population growth and religions influence on baby making:

http://video.ted.com/assets/player/swf/EmbedPlayer.swf

Apparently he has a one hour documentary on stats.  I’m adding watching it to my list of goals for the long weekend.

Watch the definitions

/ bs king / 2 Comments

A quick one for a Friday:

I’ve blogged before about paying careful attention to the definition of words used in study results.  It is often the case that the definition used in the study/statistic may not actually match what you presume the definition is.

Eugene Volokh posted a good example of this today, when he linked to this op-ed in the Detroit Free Press.  It cites a spokesperson from the Violence Policy Center who states that “Michigan is one of 10 states in which gun deaths now outpace motor vehicle deaths”.

My knee jerk reaction was that seemed high, but my tired Friday brain probably would have kept skimming.  Then I read why Volokh was posting it:

The number of accidental gun deaths in Michigan in 2009 (the most recent year reported in WISQARS) was … 12, compared to 962 accidental motor-vehicle-related deaths. 99% of the gun deaths in Michigan that year consisted of suicides (575) and homicides (495).

To be honest, I had presumed homicides were included, but suicide death didn’t even occur to me.   I’d be interested to see how many of the vehicular deaths were suicides, my guess is the percentage would not be as high as in the gun case.  Either way, I’m sure I’m not the only one who didn’t realize what was being counted.

Watch the definitions, and have a fabulous Memorial Day weekend!