sampling bias

Rule #6

/ bs king / 3 Comments

Since posting my 5 rules for reading science papers online, I’ve been pondering if I missed anything.  Something  in Erin’s comment on the matter about “experts” tearing in to new mother’s because they didn’t feed their child organic yak’s milk from a glass bottle with a hemp nipple reminded me of a whole genre of ridiculousness I missed.

          6. Be very wary of those whose main argument is “we did this for thousands of years so it must be the best way of doing things”.   Now listen, I actually don’t mind this logic for lots of things.  I confess, I’m one of those people who likes more natural things in general (ESPECIALLY peanut butter.  I HATE peanut butter with anything other than peanuts and salt.)  So I get why the train of though appeals to people.  HOWEVER, in the absence of supporting evidence, it’s not a real argument most of the time.  There’s a couple of subreasons for this:
  • Most people saying this don’t actually know how many people did whatever they’re claiming or for how long.  They’re not anthropologists.  To say that all primitive people ate this or raised their children like that is probably a bit reductionist.  
  • The lifespan in “the good old days” was abysmal.  Yes, they might have fed their infants yak’s milk or eaten saber tooth tiger raw, but many of those children died before the age of 5.  It’s called “survivor bias”, and it means you have to be careful to remember that you only get to see how the people that lived through the whole thing turned out.  Some advances in technology are pretty categorically good, and if you’re going to try to live more “back to the earth” you should remember that you’re probably going to have to be a little selective about this if you want to live past 35 or so.
  • Lots of practices from ancient times are dismissed pretty out of hand these days, we’re selective about the ones we glorify.  I remember the first time I really read the Bible’s retelling of the Moses story and had to ask my mother what a “wet nurse” was.  I was horrified.  Most people today who go with “natural” practices would never consider handing their child off to a stranger to breastfeed, and yet other practices are held up as the ideal.  Our forefathers/foremothers got a lot of things wrong.  On second thought…..I feel the whole wet nurse comeback is due for a comeback….anyone want to try to write a book for rich parents about how this is what you’d do if you really loved your child?
I’m sure there’s more, but now I have a craving for yak’s milk.  I hope my local Whole Foods is open early.

Are law schools liable for misleading statistics?

/ bs king / 5 Comments

An interesting snippet from over at the Volokh Conspiracy, where former students sued their law school for publishing misleading statistics.

The court ruled that the salary statistics published by the school were truly misleading, but in the end caveat emptor prevailed.  Apparently the schools had published average salary data, but only for those students who actually got jobs.  The court ruled that:

….even though Plaintiffs did not know the truth of how many graduates were used to calculate the average salary, at the very least, it is clear that the Employment Report has competing representations of truth. With red flags waiving and cautionary bells ringing, an ordinary prudent person would not have relied on the statistics to decide to spend $100,000 or more.

I do love legal language at times, and I was fairly amused by the phrase “competing representations of truth”. While in this case it was clear cut what information would have been most useful to the consumer, it’s often unclear what statistical breakdown represents “actual reality” and such.  I did think that perhaps the court was giving the public too much credit though, when it cited what an “ordinary prudent” person would do (or is it just that not many prudent people exist?).

I’ve been reading Tom Naughton’s blog quite a bit lately, and he often quotes his college physics professor’s advice to all of his students.  It’s a good quote, one that I think should be taught to all students freshmen year of high school.  In fact, it should have been used in this court decision:  “Learn math.  Math is how you know when they’re lying to you.”

The SCOTUS and perception of statistics

/ bs king / 4 Comments

Finally got internet in the new house.  Can’t complain too much….the guy finished running the wire to our house even though a thunderstorm started.  Clearly that man was getting paid by the job, not the hour.

Anyway, had an interesting chat with my father (a lawyer) after our closing on Thursday about the Supreme Court ruling on health care.  He mentioned that a coworker was griping that the Supreme Court meant nothing any more because they only voted on party lines.  My father, being the good data accuracy man that he is, quickly dissented.

He looked it up, and asserted that nearly half of the decisions last year were unanimous.  For this year, 7-2 votes were the least common (8%), then 8-1 (11%), 6-3 (17%) and then 5-4 (20%).  So overall,  they agree nearly as much as they disagree, and they are only completely divided on about 1 in 5 cases.  Kennedy and Roberts voted with the majority over 90% of the time.  Ginsburg was the least likely to vote in the majority.  Lots of interesting stats to be run on this, another good breakdown of some of the data is here.

It seems the perception that every vote is political is heavily skewed by the very few court cases most of us hear about every year.  I would wager even highly political citizens probably couldn’t rattle off more than a handful.  When you break down the 5-4 decisions exclusively, about 2/3rds of them vote down ideological lines…..which totals to about 10 cases for 2011.

This kind of skewing of perception is common when a few high profile events dramatically overshadow regular operations.  Thanks Dad, for pointing that out.

Sexism and stay at home moms

/ bs king / 10 Comments

I was just thinking I wanted to find a good marriage and family research paper to sink my teeth in to.

This one came across my inbox today, and I didn’t have to get much further than the abstract before I knew it was going to be a doozy.  Read for yourself:

In this article, we examine a heretofore neglected pocket of resistance to the gender revolution in the workplace: married male employees who have stay-at-home wives. We develop and empirically test the theoretical argument suggesting that such organizational members, compared to male employees in modern marriages, are more likely to exhibit attitudes, beliefs, and behaviors that are harmful to women in the workplace.

*Bias Alert*
My mother was a stay at home mom.  Therefore my father would have qualified for this study, and it is hard for me to even read their hypothesis without remembering that.  I happen to credit my father with giving me my passion for statistics and data analysis, and he has never once discouraged me from doing anything I wanted to professionally (with the exception of when I mentioned law school….that he soundly discouraged as a waste of talent….and this was  15 years before anyone was talking about a law school bubble).  I will not go in to all the details of my parents marriage here, but I doubt you could find anyone who would call my parents marriage anything less than an equal partnership focused on doing what was best for the family.

As an extra level of bias, I will be continuing my (full-time) job post baby.
*End Alert*

I’ve noticed a disturbing trend in both the general population and academic research: people seem to get very hung up on conflating “stay at home mom” with “traditional marriage”.  The study authors do this openly….they admit that they classify a marriage as “modern” based solely on whether or not the wife works full time.  The only criteria for “traditional” is that she doesn’t work at all, and part time work is all classified as “neo-traditional”.

To ignore the economic realities that drive families to make decisions about work seems to me to be an immense oversight.  I have met plenty of stay at home mothers who were in very equitable marriages, and I have met quite a few working mothers whose primary source of stress was their husbands continued expectation that they were still responsible for all child care/household duties.  I believe that using only one metric to rank a marriage as “traditional” or “modern” is a horrible over generalization….especially since most women with small children would prefer to work part time.  In fact (from the Pew study):

The public is skeptical about full-time working moms. Just 14% of men and 10% of women say that a full-time job is the “ideal” situation for a woman who has a young child. A plurality of the public (44%) say a part-time job is ideal for such a mother, while a sizable minority (38%) say the ideal situation is for her not to work outside the home at all.

So 90% of women don’t think the “modern” setup is ideal when there are young children involved.  If one of these women than chooses to stay home with her kids, has her husband truly regressed from “modern” to “traditional”?

For both the economic reasons and the “women’s choice” reasons, I reject studies that try to tie stay at home motherhood to anything else.  The sample is just too broad, and the reasons too varied.  It also undermines exactly how expensive child care can be….by my estimate, my mom would have had to bring home at least $4000 a month (in today’s dollars)  to pay for child care for 4 children.  $4000 after tax is a pretty hefty before tax salary.

I don’t argue that personal life can affect professional attitudes, and I would never advocate for sexism in the workplace.  In this study however, I really had to question the motives.  Is it really the best idea to fight gender stereotypes with stereotypes about very broad choices?  Is the point here that the workplace will only be fair when women participate as much as men?  Isn’t it a bit sexist to totally disregard the role women play in the decision to work or not work?  Shouldn’t we all just be able to do what’s best for our families, no questions asked?

Quote of the week and more recall coverage

/ bs king / 2 Comments

Statistics are like bikinis.  What they reveal is suggestive, but what they conceal is vital.  ~Aaron Levenstein


I’ve been reading more of the Scott Walker recall election coverage, and was struck by the frequent references to Walker being “the first governor to survive a recall election”.  Of course this made me curious how many governor’s had been recalled.  I remembered the California governor a few years back, so I had been imagining it would be at least a dozen or so.

Nope.

It’s two.  Lynn Frazier from North Dakota in 1921, and Gray Davis from California in 2003.

I had to laugh at my own sampling bias.  My assumptions were pretty understandable….I’ve been of voting age since 1999, and in that time this has happened twice.  Therefore it was reasonable to assume this happened at least occasionally.   I figured about once every 10 years, which would be 23 or 24 in American history.  I was pretty sure not every state had a recall option, so I halved it.  12 felt good.

This is the problem when data leaves out key points….it relies on our own assumptions to fill in the details.  Engineers are normally trained to get explicit with their assumptions when estimating, as evidenced by the famous Fermi problem.  However, even the most carefully thought through assumptions are still guesses.

That’s why it’s important to remember the quote above: what you’re shown is important, but it’s not half as interesting as what’s hidden.

Opinions, everybody’s got one

/ bs king / 3 Comments

I was listening to a management podcast recently where a man named John Blackwell was being interviewed.  He was talking about how he was constantly reading things about how the whole workplace was changing, but he was getting curious as to why he felt like the companies he worked with weren’t reflecting this.  When he tried to investigate, he found out that the ongoing surveys commonly used in British management journals (can’t find a link) were being done on the “up and coming business leaders”.  When he looked in to what that meant, he realized it was people who were second year MBA students.

The problem with this, of course, was that this was asking people not in the workforce what the workforce was going to look like 10 years from now.  They found, not surprisingly, that young people in grad school tend to be very optimistic about things like “working from home” or “flex time” when they’re in school, but when they got in to business, they towed toed the line.  Thus, every survey done was essentially useless.  
This all reminded me of a conversation I got in to several years ago when I was working the overnight shift.  Someone had brought in a magazine (People or Vogue or something like that) and they had a ranking of the 100 most beautiful women in Hollywood.  Drew Barrymore was number one that year, and one of my (young, male) coworkers was actively scoffing at that.  “She’s unattractive,” he stated definitively.  “All the guys I know think so too.”
Now, I was feeling a little feisty feminist that night, so I thought about how to challenge him on that.  Leaving aside that “Hollywood unattractive” would still turn heads in any average crowd (and be more attractive than any girl he’d dated), something about his comment irked my data side.  “So maybe the voting was done by women,” I replied.  
He was floored.
I noted that it was not a men’s magazine that ran the story, so really women’s opinions of other women’s attractiveness would actually be more relevant to this list.  Furthermore, as most of the leading women in Hollywood make their money on romantic comedies, professionally women’s opinions of their attractiveness (which presumably included a certain likeability factor) would actually matter more than men’s.
I was fascinated that this clearly disturbed him.  It had clearly never occurred to him that straight men may not be the target audience for female attractiveness, or even that the relevance of his opinion might get questions.  He wasn’t trying to be a jerk, he was legitimately confused at the whole idea.
A long intro, but the bigger point is important.  In any opinion survey or research, it’s important to figure out whose opinion is most relevant to what you’re trying to get at and why.  When it comes to law and public policy questions, I think every voter is relevant.  When it comes to workplace trends?  You may need to narrow your sample.
Sampling bias is a huge problem in many contexts, but my primary one for today’s post is when the survey was not conducted with the end in mind.  For any sample, you have to figure out how much your subject’s opinions actually matter given what you’re trying to find out.  In social conversation it may be interesting to find out what a particular person thinks of a topic, but for good data, show me why I care.