It’s not the question, it’s how you ask it

/ bs king / 1 Comment

Data gathering is a lot harder than most people imagine.  It’s an interesting exercise to take a study and prior to reading it start asking yourself “how would I, if pressed, get the data they claim to have gotten?”.  It’s amazing how many fall apart quickly when you realize how bad the source data is.

I face this all the time at work.  The simplest questions…what is our demand for transplants? can be a never ending labyrinth of opinion, observation, anecdote, and data….all completely enmeshed.  I spend much of my day trying to untangle these strings, and I never underestimate how difficult getting a simple answer can be.

Factcheck.org ran a great piece today illustrating this challenge.  In a post titled “How Many Would Repeal Obamacare?”  they review 4 different surveys that all try to get to the same number: how many people think healthcare reform should be repealed?

It’s a great article that covers sampling practices, question phrasing, date of the poll, and history of the polling organization.

If you looking at the numbers, it shows up pretty quickly that when given dichotomous choices (repeal/keep), people often look like they gave a strong opinion.  In the polls where more moderate answers are given (“it may need small modifications, but we should see how it works), people trend towards that answer.

The phrasing was extremely intriguing though:
“Turning to the health care law passed last year, what is your opinion of the law?”
“If a Republican is elected president in this November’s election, would you strongly favor, favor, oppose, or strongly oppose him repealing the healthcare law when he takes office?”
“Do you think Congress should try to repeal the health care law, or should they let it stand?”

In one, the question focuses on personal opinion, in the next the focus is the presidency, in the third it’s Congress.  All of this for a law that most Americans have yet to feel the effects of in any practical way.

Of course this is not to say that a public opinion poll (or 4) makes one side right or wrong. If constitutionality or effectiveness are your concern, nothing here addresses either.  I am enjoying it immensely for the educational value though, and kind of wishing I was teaching a class so I could use this as an example.  Those of us in Massachusetts do have the luxury of sitting back and just sort of pondering all of this….as this has been our world for 7 years now.

That reminds me….were these samples controlled for that????

Correlation and Causation: the Housework Edition

/ bs king / 5 Comments

After yesterday’s comic, I was hoping to find a good example of a news story where they equated correlation and causation.  In case you’re curious, it took me under 5 minutes.

Headline: Why Being Less of a Control Freak May Make You Happier

To start, let me just mention that correlation implies that two things are moving together….as one goes up, so does the other.  Alternatively, as one goes up, the other goes down, or vice versa.  Either way, their outcomes appear to be tied.

Causation on the other hand, says that one thing is causing another.  What yesterday’s post was referring to is the often made mistake that just because two things are correlated, we can infer that one is causing the other.  This is not always true, and believing so may get you drawn as a stick figure.  

Anyway, the article above illustrates that point nicely.  The author set out to find out if being a control freak mom made people unhappy….and low and behold it appears to.  55% of women who said they delegate to a partner or spouse at least once a week reported themselves as “very satisfied” with their life.  For those who did not delegate that often, the number was 43%.  

Now, I’ll mostly skip the use of the word “delegate” in this article, though it does bother me.  My husband does plenty around the house, but we mostly just consider that “teamwork” not “delegating”.  I don’t start the week handing out tasks to him, and he doesn’t consider the work he does around the house a favor to me.  It’s just what needs to get done.

More importantly however, is the articles conclusion that delegating will make people happier.  While delegating and happiness are perhaps correlated, they are not necessarily causal.  It’s possible that the women who don’t delegate do so because their spouse is lazy, hostile, or generally not involved….all things which would also make them less happy over all.  It’s also possible that women who don’t delegate are controlling, martyr’s, passive aggressive, etc, and that makes them unhappy too.

I had a great stats professor once who opened every class with this:

“If you get one thing out of this class, let it be this:

When x and y or correlated, you have 3 possibilities:

  1. X is causing Y
  2. Y is causing X
  3. Something else is causing both X and Y “
Lack of delegating could cause unhappiness.
Unhappiness could cause people to stop delegating.
Something else entirely could cause people to not delegate and to be unhappy.

When in Doubt, Blame the Journalist

/ bs king / 3 Comments

Within minutes of hitting “publish post” on my mission statement, I found an article that reminded me of one of my worst pet peeves when it comes to data/science/studies of all types.  The headline read  “Keeping Your Name? Midwesterners Are Judging You”.  My ears (eyes?) perked up at this headline, as I am among those women who declined to change her name post-nuptial.  Despite knowing that Jezebel is not often the best place for unbiased reporting, I gave it a read.  

The article linked to a much more well nuanced article here, but the basics are as follows: students at a small midwestern college feel that women who don’t change their last names when they get married are less committed to their relationships than those who do.  This was interesting in part because the number of people who felt negatively about this quadrupled between 1990 and 2006.  
For the personal reasons listed above, I find this interesting.  However, when you look at the numbers (2.7% of 256 and 10.1% of 246 which Jezebel did include) and do a little math, you realize that this “jump” is a difference of 18 people.  
A few things to consider about this:
  1. I couldn’t find that this was published anywhere.  It seemed to be a sort of “FYI for the headlines”.
  2. Apparently there’s no data on whether or not this perception is true.  My bias would be that it’s not, but I couldn’t find data actually saying if the perception was correct.  This happens in many “perception” studies….they quote percentages who believe something with the implication that a certain belief is wrong without ever proving it.
  3. There wasn’t a gender breakdown of who those 18 people were.  If most were female, then isn’t their perception likely to be based on experience?  As in “well if I didn’t do it, it would be because I wasn’t committed”?  That not judgement of others, that’s judgement of self.
  4. Have any of their professors (or TV shows, or other media sources) recently made disparaging remarks about this?  18 people who all very well might know each other (the university surveyed was under 1000 students) could easily be influenced in their answer  by even one strong source.
  5. As college students, presumably very few of those polled were actually married.  From my experience in college, I would conjecture that this is a phase of life during which people are very idealistic regarding their future mates without having many real experiences to back it up.  I put much more stock in what people who are actually married use to feel out level of commitment than what someone who’s never walked down that aisle thinks.
All that being said, it looked like the study authors were careful to address several of these points (especially the “this is not a representative sample” point.  It was only in the translation that conclusions were drawn that were more dubious.  
Scientists have very little incentive to exaggerate the meaning of their findings.  They are in a profession where that could be very damaging.  Reporters for both old and new media have EVERY incentive to spin things in to good headlines.  Remember that.

Mission Statement

/ bs king / 5 Comments

Numbers never lie.  

Unlike people, who are constantly confused by their own biases and perspectives, numbers behave….if you know how to use them.  
This is what I do for work every day:
First, I get what management thinks is the problem.  Second, I talk to the people involved and find out what they think is the problem.  Third, I get to retreat in to the numbers.  I spend time looking at what we’re doing, where we are, and where we’d need to be for everyone to be happy.  It’s the third part that’s my favorite.  No one argues, no political pressure, just puzzles, problems, and unexpected truths.  
I use data every day to help improve health care, and I’ve been pretty successful at it so far.  As I look around though, I realize how few people really understand the importance of good data in our lives.  One needs look no further than election year politics to see bad data, poor interpretations of good data, and blatant misuses that make me cringe.  In the healthcare realm, we don’t have this luxury.  I come from a world where you can’t take chances, where misrepresenting your stats can result in very real human suffering.  
This is why improper uses of data drive me nuts.  Once you know what to look for, it’s hard to stop seeing it. It’s everywhere.  Thus, I am giving myself an ambitious goal.  It’s no longer enough for me to use good data science for my own purposes.  I want to educate others, and hone my own skills along the way.  I want people to know what research is, how to read it, and how to question it.  I want others to be as passionate as I am, and I want a place to vent about the reporting that annoys me.  
Stay tuned.