Correlation and Causation: Real World Problems

/ bs king / 2 Comments

In yesterday’s post, I got a bit worked up over sloppy reporting on a study on dietary interventions in pregnancy. 

This led to an interesting comment from the Assistant Village Idiot regarding weight gain recommendations for pregnant women.  The current weight gain recommendation is 25 – 35 pounds for a normal BMI woman, but AVI commented that it used to be much lower, and that women were hospitalized to stop them from eating too much.
I didn’t actually know that, so I immediately decided to look it up.  
I stumbled on to a fascinating presentation put together by an OB at UCSF on the history of maternal weight gain recommendations (link goes to the PowerPoint slides).  It not only confirmed what AVI had mentioned, but also gave some of the reasoning….which turned out to be a very interesting example of people erroneously conflating correlation with causation.  
Apparently part of the reason why they (they being doctor’s circa 1930) were so nervous about weight gain in pregnancy was that they were trying to prevent preeclampsia.  Now preeclampsia is a life threatening condition if left untreated, and one of the warning signs is rapid weight gain.  Apparently some doctors actually thought that the symptom was the cause, and believed that all excessive weight gain was a sign the patient was about to become preeclamptic.  Thus, the theory went, limiting weight gain would prevent preeclampsia and aid in “figure preservation” to boot*.  
Sadly, this also led to higher infant mortality, disability, and mental retardation….which seems a pretty steep price to pay for what was really a data analysis error.  As I’ve said before, this is why statistics are so relevant in medicine….the cost for getting things wrong is too steep to not be careful.
*To note, it is actually true that preeclampsia is linked to higher weight/glucose/insulin production….but the way they went about addressing it did as much harm to the fetus as good.  Current weight gain recommendations are set to optimize outcomes for the babies, not the mothers.  

When in doubt, blame the journalist: prenatal dieting edition

/ bs king / 4 Comments

Sometimes bad science reporting makes me laugh, and sometimes it actually kind of stresses me out.  This is one of the “this stresses me out” times.

The headline reads: Diet during pregnancy is safe and reduces risk for complications, study finds

Now aside from being a bit on the garbled side, it’s a pretty provocative headline.  As someone who has been in and out of obstetrician’s offices for the past 7 months or so, it also runs counter to everything I’ve been told.  According to this write-up however, here’s a few things this study found:

 Is it safe for a pregnant woman to go on a diet? According to a new study, not only is it safe, but it can even be beneficial and reduce the risk of dangerous complications.

That would seem to contradict what my doctor has told me….but let’s read on (to what they found about dieting methods):

The researchers found that all three methods reduced a mother’s weight, but diet showed the greatest effect with an average reduction of almost 9 pounds. Pregnant moms who only exercised lost about 1.5 pounds, and moms who did a combination of diet and exercise lost an average of 2.2 pounds.

So they had mothers to be lose weight during pregnancy?  That seems….extra wrong….but go on:

Women who went on a calorie-restricted diet were 33 percent less likely to develop pre-eclampsia, a spike in blood pressure caused by significant amounts of protein in the urine.

Wait, now I know he’s just phoning it in.  Pre-eclampsia is not high blood pressure caused by protein in the urine, it’s high blood pressure AND high protein in the urine….in fact the Mayo Clinic article he links to says so.  

At this point, I took a look at the original study, and found other “oops” moments in the reporting.  First, the study never looked at “diets”.  What they actually looked at was “dietary interventions”…which they describe as follows:

Typical dietary interventions included a balanced diet consisting of carbohydrates, proteins, and fat and maintenance of a food diary. 

Since this was a meta-analysis, I took a look at the references, and in fact only one study cited directly looked at caloric restriction….the sort of thing most of us think of when we hear the word “diet”.

Furthermore, that part about the women’s weight being reduced?  It wasn’t.  Their weight gain was reduced.   …something the study authors are clear about, but the subsequent write up completely leaves out.

I actually got a little angry about this.  You can feel free to blame pregnancy hormones, but I find this sort of thing is just irresponsible.  CBS is a major news network, and people are going to take what they say seriously.  As the Assistant Village Idiot likes to point out, people believing faulty science on small things can be funny and doesn’t matter much….but when you realize bad studies could actually affect the way people live, it gets scary.  Someone following this story could do some real damage.  In fact, the article does get clearer towards the end (when it quotes the original study author), but that’s 6 paragraphs in.  It drives me nuts that a good a carefully thought through study can get reported so sloppily and potentially dangerously.  There is a world of difference between what most of us think of when we say “diet” and what the researchers here described, which was essentially just formalized pre-natal nutritional counseling.

Overall, real dieting during pregnancy is still dangerous….and can backfire in a big way.  Mother’s who are forced to restrict calories during pregnancy (famine victims, etc) actually wind up having children who are more likely to be obese and develop diabetes.  As a side note, one of the most fascinating studies on this is the Dutch Famine Study where mother’s who had temporary famine conditions during pregnancy could be studied for the long term effects on the children.

This is why it matters that the media report things correctly.  People should not walk away from reading about good science with bad ideas.  Words like “diet” or “weight reduction” do not mean the same thing as “dietary interventions” or “weight gain reduction”. No one should have to read to paragraph 7 to get accurate information.  That’s just bad form.

The only thing that could have made this story worse would have been an infographic.  I’m going to have nightmares about that tonight.

Friday Fun Links 5-18-12

/ bs king / 4 Comments

When someone who writes about bad science for a living calls something “The worst government statistic ever created“, you know it’s going to be good.

Okay, that report was from the UK….now do you US folks want to know what’s wrong with your state?  Massachusetts has blisters, apparently.

If there’s something wrong with this data, I don’t want to know about it.  There is no such thing as strong coffee, only weak people.

I kid actually, the above study has all the normal problems of nutritional research.  The Time write up did give me the quote of the week however:

Since the study was observational only, the authors couldn’t conclude that coffee drinking actually reduces death risk.

Gee, with a headline like “Coffee: Drink More, Live Longer?” I can’t see why anyone would jump to that conclusion.  Also, I kinda hate the phrase “death risk”.  Unless we’re about to get in to an eschatology debate, I’m pretty sure my death risk is 100%, no matter how much coffee I drink.

Moving on, the Pew Research Group started meta-analyzing their own analysis…with sad results.  
On a perkier note, if you want to win your weekend geek-off, here’s a (NSFW…sorta) guide to why Tesla > Edison…even with that whole pigeon thing.




 

…and now for something completely different.

/ bs king / 6 Comments

I don’t normally get that involved with sports statistics, if only because it’s the one place in the stats world where you could study them for an hour every day and still be barely a rookie.  However, something awfully strange is happening in my house recently, and I feel it’s worth mentioning:  the Orioles are leading the AL East (in fact the whole American League), and the Red Sox are last.

Now, this is particularly interesting to my household, as my husband happens to be a lifelong Orioles fan.  I on the other hand, have always been a Red Sox fan.  Since we met almost 6 years ago, this has pretty much meant that I have had exclusive bragging rights when it came to baseball.  I know it’s not even a quarter of the way in to the season, but this is the longest we’ve gone so far, and it’s surreal.

Yesterday, Grantland put up an article on the Orioles under .500 curse.  Apparently they have not finished over .500 since 1997….more than enough seasons for the baseball stats guys to go nuts with.  I was curious exactly how bad it was, so I looked around until I found this graph generator*.

For those of you who don’t know much about the Orioles, here’s what they’ve looked like since 1998

Yowza.  Even if this season doesn’t hang in there, it’s still the most encouraging thing to happen in 7 years or so.
Now, here’s the Red Sox in the same time period:
Yikes.  If they don’t pick it up soon this will be the worst they’ve started off in 15 years.  
Sweetly enough, if the Sox win tonight against the Rays, that will both increase the Oriole’s lead in the AL East, and look good for the Red Sox.  
Honey, the data proves it, tonight we’re both Red Sox fans.
*If it shows you how crazy sports stats people are, I found that graph generator in exactly one try on google.  Conversely, when I tried to find historic gas prices for this post, I searched for almost half an hour trying to find an official source for anything pre-1978.  Didn’t happen.  

Correlation and Causation: the Teen Pregnancy Edition

/ bs king / 4 Comments

One of the first posts I ever did was on correlation and causation.  In it, I spelled out the three rules to consider whenever two variables (x and y) are linked:

  1. X is causing Y
  2. Y is causing X
  3. Something else is causing both X and Y
While most people jump to the conclusion that it’s number 1, Matthew Yglesias wrote a piece for Slate.com this week where he rather awkwardly jumps to conclusion number 2.  
He starts off well with the second paragraph, but then goes to very strange place in the third: 

Delivering the commencement address last weekend at the evangelical Liberty University, Mitt Romney naturally stuck primarily to “family values” and religious themes. He did, however, make one economic observation that intersects with some fascinating new research. “For those who graduate from high school, get a full-time job, and marry before they have their first child,” he said, “the probability that they will be poor is 2 percent. But if [all] those things are absent, 76 percent will be poor.”
These are striking numbers, but they raise the age-old question of correlation and causation. Does this mean that the representative high-school dropout would be doing much better had he stuck it out in school for a few more years? Or is it instead the case that the population of high-school dropouts is disproportionately composed of people who have attributes that lead to low earnings?
When it comes to early pregnancy, surprising new evidence indicates that Romney and most everyone else have it backward: Having a baby early does not hamper a young woman’s economic prospects, as Romney implies. Rather, young women choose to become mothers because their economic outlook is so objectively bleak.

Say what?

As a former teenage girl myself, this is a strange conclusion….I certainly never met a teen mom who would have put it that way.  But surely there was some wonderful evidence to support this scathing conclusion?

Well, not really.  Here’s the original paper….and  here’s how the authors conveyed their thoughts:

We describe some recent analysis indicating that the combination of being poor and living in a more unequal (and less mobile) location, like the United States, leads young women to choose early, non-marital childbearing at elevated rates, potentially because of their lower expectations of future economic success. …These findings lead us to conclude that the high rate of teen childbearing in the United States matters mostly because it is a marker of larger, underlying social problems.

The emphasis was mine….but notice how much more careful they are in their language.  If you take my list above, you see that they are challenging possibility number 1, seeing if #2 is a feasible conclusion, but ultimately pointing the finger at #3….i.e. “larger, underlying social problems”.

For example, the cite low maternal education as a risk factor for teen pregnancy…which one could presume could be either the result of or the cause of low income.

Teen pregnancy is complicated, and honestly I would be very surprised if you could ever figure out a way to pin it on just one factor.  Additionally, so much information is unavailable that it can be hard to parse through what you have left.  A key factor in all of this would be to determine if higher income girls weren’t having babies because they weren’t getting pregnant or because they were having abortions….data which could lead to very different conclusions.

I fully support this study, by the way, questioning the prevailing wisdom is always a good thing. What I resent is when people think just by flipping the order of a normal conclusion that they’re being clever.

X could cause Y, Y could cause X, something else could be causing both.

Then again, it could also just be a coincidence.  

The price of bad data

/ bs king / 2 Comments

Yesterday Instapundit linked to a story on “the perfect data storm”.

Thinking that sounded up my alley, I went and read the article.  It’s from a professor named David Clemens at Monterey Peninsula College, complaining about the use of data in higher education:

While knowing full well data’s vulnerability, education managers cannot resist the temptation to be data driven because data absolves them of responsibility; to be data driven lets them say “the data made me do it” (hat tip to Flip Wilson).

That made me sad.  
He cited a few numbers floating around his campus that he knew were bad…transfer rates that only counted transfers to state schools for example….and yet they were still being included in policy decisions as though they were comprehensive.
That made me really sad.
While I enjoy mocking bad data, it’s important to remember that there is a real price to it.  That’s why I think it’s important to empower people to question the data they’re hearing and to know what weaknesses to look for when you hear numbers that sound implausible.  
Clemens continues:

….we discover that information does not touch any of the important problems of life. If there are children starving in Somalia, or any other place, it has nothing to do with inadequate information. If our oceans are polluted and the rain forests depleted, it has nothing to do with inadequate information. 

I am going to make a radical suggestion about data and higher education:  colleges and universities will be better served if they avoid kneeling at the altar of data and instead fill key positions with people driven by intuition, experience, values, conviction, and principle.  A good place to start would be looking for leadership guided by a transcendent educational narrative.

I both agree and take issue with this statement.  Data doesn’t solve problems, but in a world of limited resources, data can guide us on where to put our efforts.  It’s not that most of us don’t agree children shouldn’t starve in Somalia, it’s that the “act first figure out if it works later” approach has the potential to cause as much harm as good.  That’s why health care is data driven by necessity…..courts are notoriously unsympathetic to the excuse “I treated the patient this way because my transcendent narrative said it was a good idea”.  Data is a good idea when you have an outcome you can’t afford to take a chance with.

In the end, I don’t think data is to blame for this backlash.  I am relatively sure that the same people who “kneel at the altar of data” to justify their own behavior are the same people who would, absent data, pursue their own gut feelings to the exclusion of rationality.  Intuition is very easily confused with emotion, experience can lead to falsely limiting possibilities, values can be misguided, conviction is dangerous in the wrong hands, and principle is easily warped.  No amount of data can change the way people are, but the more people who can spot the flaws in data and call BS, the better.

*Steps off soap box*

Trudge on friends, and don’t let the weasels get you down.

Why most marriage statistics are completely skewed

/ bs king / 3 Comments

Apparently Slate.com is now doing a “map of the week”.  This week, it was a map of states by marriage rate.  Can’t get it to format well….click on the map and drag to see other states.

http://a.tiles.mapbox.com/v3/slate.marriage.html#4.00/40.65/-95.45

It shows Nevada as the overwhelming winner, with Hawaii second.  This reminded me about my annoyance at most marriage data.

Marriage data is often quoted, but fairly poorly understood.  The top two states in the map above should tip you off as to the major problem with marriage data derived from the CDC in particular….it’s based on the state that issued the marriage license, not the state where the couple resides.  Since all (heterosexual) marriages affirmed by one state are currently recognized by every other state, state of residence information is not reported to the CDC.  This means that states with destination wedding type locations (Las Vegas anyone?) skew high, and all others are presumably a bit lower than they should be.  Anecdotally, it’s also conceivable that states with large meccas for young people (New York City, Boston, DC) may be artificially low because many young people return to their childhood home states to marry.  This

The other problem with marriage data is the resulting divorce data is even more skewed.  Quite a few states don’t report divorce statistics at all (California, Georgia, Hawaii, Indiana, Louisiana, Minnesota) and the statistics from the remaining states are often misinterpreted.  One of the most commonly quoted statistics is that “50% of marriages end in divorce”.  This isn’t true.

In any given year, there are about twice as many marriages as there are divorces….but thanks to changing population, changing marriage rates, people with multiple divorces, and the pool of the already married, this does not mean that half of all marriages end in divorce.  In fact, if you change the stat to “percent of people who have been married and divorced”, you wind up at only about 33%.  More explanation here.

Ultimately, when considering any marriage data, it is important to remember that there are no national databases for this stuff.  All data has to come from somewhere, and if the source is spotty, the conclusions drawn from the data will likely be wrong.  This all applies to quite a few types of data….but marriage data is used with such confidence that it’s tough to remember how terrible the sources are.  A few people have let me know that I’ve ruined infographics for them forever, and I’m hoping to do the same with all marriage data.

You’re welcome.

Compensation Data for Mother’s Day

/ bs king

This year for Mother’s Day, use data to figure out how much you owe your mother for her pregnancy and labor.

It turns out I owe mine $99.28*.  I got some good discounts for my low birth weight and my early arrival.  I also got a decent “good offspring” discount for calling her this morning to wish her a happy Mother’s Day, so that was positive.  
Of course, one could quibble that perhaps a mother should not be charging her child for a pregnancy that the child did not have a say in….though the idea of issuing a bill to my own child in 12 weeks or so when he shows up is tempting.  For now though, I think I’ll pass the bill off to my Dad and see if he’d like to chip in.  I’m pretty sure the Edible Arrangement I sent her should cover my half. 
Good luck with the rest Dad.

Love you Mom!

*I am not even going to try to criticize this number.  There is absolutely no explanation for any of the numbers or why they vary the way they do.  This is actually somewhat refreshing to me.  Normally you have overly precise numbers being justified by vague guesses.  Here they don’t even pretend to have reasons.  I like the tacit admission of complete BS.  

Historical accuracy, ngram style

/ bs king

I’ve used google ngram’s a few times on this blog already, mostly for silly things, but this website has the best use of it I’ve seen so far.

He takes the scripts of Downton Abbey (WWI) and Mad Men (1960’s) and feeds them through the ngram to find out which phrases are the most anachronistic.

I find the whole project pretty cool, because apparently he took the whole project on as a response to a few magazine articles about phrases that wouldn’t have been said at the time.  It struck him that those phrases were just the ones that people could hear and think “hey, that sounds modern!”, but no one was thinking through what phrases we might have gotten so used to we weren’t even recognizing as out of place.

I’ve never seen Downton Abbey, and only seen an episode or two of Mad Men, but I still found it interesting what they got wrong.  The last episode of Mad Men apparently had an aspiring actress use the phrase “got a callback”, which apparently was barely used in a theater context at the time (he cross references the OED).  He also makes pretty charts, which I loved (this one is for Downton Abbey):

Overall, a very fun use of data.