I sat in a fascinating talk yesterday about some pretty interesting team failures. One in particular stuck out to me: two teams, working on the east and west coast, funded by a huge grant from the NSF. One team was tasked with building a database, the other was going to populate it with all of the data. A year’s worth of work later, it was discovered that the two teams had never clarified what they meant by several words (including the word data) and that the whole thing was completely useless.
Now, there are several lessons in that story, but one of them is the importance of knowing what certain words mean to the people who are saying them. This can be a big issue in reading research and interpreting data, especially around popular public health type issues. There are many issues….”rape” “excessive drinking” “binge eating” and “substance abuse” to name a few….that people tend to believe there is one hard and fast definition for. When reading studies on these things, always verify that the authors definition matches your own.
In looking for good examples of this, I found this report
on some drinking statistics that were being floated around a few years ago.
A new study from Columbia University’s National Center on Addiction and Substance Abuse (CASA) claims that adults who drink excessively and youths who drink illegally account for over half of the alcohol consumed in the United States, and that the alcoholic beverage industry makes too much money from these groups to ever voluntarily address the problem.
The article goes on to point out that if you look at the data, “excessive drinking” was defined as more than two servings of alcohol in one day, with no respect for height, weight, or frequency. I somehow doubt this is the picture most people got when they read “adults who drink excessively”.
This comes up a lot in studies that have psychiatric diagnoses attached as well. I have a friend who works with eating disorders who gets annoyed to no end that you can’t technically call someone anorexic until they’re 15% under a healthy body weight or have had their period stop, even if they stop eating for weeks. Not many people know that up until this year, the FBI defined rape as something that could only happen to women.
Things to watch out for.