5 Interesting Examples of Self Reporting Bias

/ bs king

News flash! People lie. Some more than others. Now there are all sorts of reasons why we get upset when people don’t tell the truth, but I’m not here to talk about those today. No, today I’m here to give a few interesting examples of where self-reporting bias can really kinda screw up research and how we perceive the world.

Now, self reporting bias can happen for all sorts of reasons, and not all of them are terrible. Some bias happens because people want to make themselves look better, some happens because people really think they do things differently than they do, some happens because people just don’t remember things well and try to fill in gaps. Regardless of the reason, here’s 5 places bias may pop up:

  1. Nutrition/Food Intake Self reported nutrition data may be the worst example of research skewed by self reporting. For most nutrition/intake surveys, about 67% of respondents give implausibly low answers….an effect that actually shows up cross culturally. Interestingly there are some methods known to improve this (doubly labeled water for example), but they tend to be more expensive and thus are used less often. Unfortunately this effect isn’t random, so it’s hard to know exactly how bad they effect is across the board.
  2. Height While it’s pretty ubiquitous that people lie about their weight, lying about height is a less recognized but still interesting problem. It’s pervasive in online dating for both men AND women, both of whom exaggerate by about 2 inches. On medical/research surveys we all get slightly more honest, with men overestimating their height by about .5 inches, and women by .33 inches.
  3. Work hours Know anyone who says they work a 70 hour week? Do they do this regularly? Yeah, they’re probably not remembering that correctly.  Edit: My snark got ahead of me here, and I got called out in the comments, so I’m taking it back. I also added some text in bold to clarify what the problem is. When people are asked how much they work per week, they tend to give much higher answers than when they are asked to list out the hours they worked during the week. The more they say they work, the more likely to have inflated the number. People who say they work 75+ hours work an average of 50 hours/week, and  those who say they work 40 hours/week tend to work about 37. Added: While some professions do actually require crazy hours (especially early in your career….looking at you medical residencies, and first year teachers are notorious for never going home), very few keep this up forever. Additionally, what people work most weeks almost never equals what they work when averaged over the course of a year. That 40 hour a week office worker almost certainly gets some vacation time, and even 2 weeks of vacation and a few paid holiday take that yearly average down to 37 hours per week…and that’s before you add in sick time.  Some of this probably gets confusing because of business travel or other “grey areas” like professional development time, but it also speaks to our tendency to remember our worst weeks better than our good ones.
  4. Childhood memories It is not uncommon in psychological/developmental research that adults will be asked various questions about the state of their life currently while also being queried about their upbringing. This typically leads to conclusions about parenting type x leading to outcome y in children. I was recently reading a paper about various discipline methods and long term outcomes in kids, when I ran across a possible confounder I hadn’t considered: sex differences in the recollection of childhood memories. Apparently overall men are not as good at identifying family dynamics from their childhoods, and the authors wondered if that led to some false findings. They didn’t have direct evidence, but it’s an interesting thing to keep in mind.
  5. Base 10 madness You wouldn’t think our fingers would cause a reporting bias, but they probably do. Our obsession with doing things in multiples of 5 or 10 probably comes from our use of our hands for counting. When it comes to surveys and self reports, this leads to a phenomena called “heaping”, where people tend to round their reports to multiples of 5 and 10.  There’s some interesting math you can use to try to correct for this, but given that rounding tends to be non-constant (ie we round smaller numbers to 5 and larger numbers to 10) this can actually affect some research results.

Base 10 aside: one of the more interesting math/pop-culture videos I’ve seen is this one, where they explore why the Simpson’s (who have 4 fingers on each hand) still use base 10 counting (7:45 mark):

 

8 thoughts on “5 Interesting Examples of Self Reporting Bias”

  1. It’s a mystery to me why we don’t use base 12. Like 10, it’s small enough to handle the numbers mentally, but 12 is divisible by 2, 3, 4, and 6—and we use quarters and thirds of things all the time, yet can’t split 10 by those numbers. Ten fingers and two feet. It’s a natural!

    Like

  2. Cultures that revere age have heaping upward to the next 0. In societies that do not write full dates often this may not be intentionally dishonest. The Puritans, especially in the Bay Colony, were notorious for this.

    Like

  3. Work hours Know anyone who says they work a 70 hour week? Do they do this regularly? Yeah, they’re probably not remembering that correctly.

    I did in the two years I taught math. Yes, I AM remembering that correctly. I also recall being too exhausted on the weekends to do much beside rest and recuperate. Granted, not 70 hours EVERY week- definitely down to 60 the last month of the school year- but pretty much 70 otherwise. If one wonders why about half of beginning teachers are out of the profession within five years of starting teaching, the time demands on beginning teachers are something to consider- though there are many other factors.

    Could I do it with fewer hours today? Probably. In one school, the problem sets of the textbook were not aligned at all with the abilities of the students- too much too soon. That required a lot of writing out of problems. Nor was the textbook well aligned with the requirements of standardized testing. More problem writing. There should be a better math textbook today. In the last month of teaching 9th grade math, I learned that a requirement of my 9th grade Biology teacher- handing in class notes to be graded- worked for better behavior and better achievement. That saved me time, with not as much time spent calling parents etc.

    Like

    • Upon further reflection, you’re right I don’t like the way I phrased this initially. That was too snarky and dismissive of certain situations. I work in a hospital and our residents/fellows are definitely NOT lying about their work weeks. 80 hour weeks are a requirement for them, and that’s come down in recent years. I have edited the post to reflect some of this.

      Other thoughts: Teachers have it rough because their work weeks are simply more variable than other professions, which can take it’s toll but doesn’t always show up in the numbers. My brother started his career in a charter school on the South Side of Chicago where he worked absolutely crazy hours. We tracked and compared time for a while, and it turned out he was working pretty close to the 50 hours/week I was working…but it looked very different. He could work two 70 hour weeks in a row, then hit a school vacation and work 10 hours that week. I’d just work right around 50 for all 3 weeks. The psychological toll for that is very different, and as you point at some point “recovery time” becomes a factor. (Side note: I worked night shift for a few years, and that happens there too. 5 days in a row of overnights means you barely get out of bed on the sixth day. That’s just not the case when you work regular daytime hours).

      Teachers seem to have a lot more grey areas too….things like attending big sporting events at their schools can be expected, but probably don’t count towards the totals of “work time”. A good friend of ours recently quit his HS principal job for this reason…the amount of “face time” he was expected to have was almost another whole job.

      I poked around to see if there was an overall “average” for teachers, and the National Education Association says it’s about 50 hours/week on “instructional duties”:
      http://www.nea.org/home/12661.htm

      They also clarify that 50% of new teachers leave the profession in the first 5 years 😦

      Like

  4. I suspect that car reliability ratings are another area that is subject to self-reporting bias. I remember a couple of conversations with folks who owned expensive cars, telling me about all of the problems with them. But when I would ask, “So, it wasn’t very reliable?” they would respond, “Oh yes,it was very reliable.” It was as if they knew the cars were supposed to be reliable, so they couldn’t bring themselves to say that they weren’t, even though some of them had issues that would have driven me nuts. I think most car reliability ratings come from surveys, so they are essentially self-reporting.

    On the subject of a 70-hour week: I once worked a couple of one-month periods of 12.5 hour days (12 hours of nominal work, plus a half-hour handover meeting), without a day off. By my calculations, that is an 87.5 hour week. Yes, it is possible. But what I found was that after about a week or so of this, you get stupid. You are so consistently tired that it is impossible to concentrate or think clearly. I tried to make the point to management that this was unsafe, and got nowhere. Then one guy was killed and another seriously injured due to some careless actions. The investigation concluded that fatigue was a factor. Another fellow fell asleep while driving home, and ran off the road. After that, it was forbidden to work more than 10 days, without a day off.

    Like

    • The car rating thing is interesting….I bet that applies to quite a few luxury items.

      For work weeks, yes, that’s been my experience. Last year I end up going in to the hospital for 35 days in a row, and it was not pretty. I know they’ve focused on the safety piece a lot at hospitals in the last few decades, limiting shifts and capping hours for medical residents.

      Like

  5. Pingback: 3 More Examples of Self Reporting Bias | graph paper diaries

Comments are closed.