The Real Dunning-Kruger Graph

I’m off camping this weekend, so you’re getting a short but important PSA.

If you’ve hung out on the internet for any length of time or in circles that talk about psych/cognitive biases a lot, you’ve likely heard of the Dunning-Kruger effect. Defined by Wiki as “a cognitive bias wherein persons of low ability suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is.”, it’s often cited to explain why people who know very little about something get so confident in their ignorance.

Recently, I’ve seen a few references to it accompanied by a graph that looks like this (one example here):

While that chart is rather funny, you should keep in mind it doesn’t really reflect the graphs Dunning and Kruger actually obtained in their study. There were 4 graphs in that study (each one from a slightly different version of the study) and they looked like this:


Logic and reasoning (first of two):


And one more logic and reasoning (performed under different conditions):

So based on the actual graphs, Dunning and Kruger did not find that the lowest quartile thought they did better than the highest quartile, they found that they just thought they were more average than they actually were. Additionally it appears the 3rd quartile (above average but not quite the top), is the group most likely to be clearsighted about their own performance.

Also, in terms of generalizability, it should be noted that the participants in this study were all Cornell undergrads being ranked against each other. Those bottom quartile kids for the grammar graph are almost certainly not bottom quartile in comparison to the general population, so their overconfidence likely has at least some basis.  It’s a little like if I asked readers of this blog to rank their math skills against other readers of this blog….even the bottom of the pack is probably above average. When you’re in a self selected group like that,  your ranking mistakes may be more due to a misjudging of those around you as opposed to just an overconfidence in yourself.

I don’t mean to suggest the phenomena isn’t real (follow up studies suggest it is), but it’s worth keeping in mind that the effect is more “subpar people thinking they’re middle of the pack” than “ignorant people thinking they’re experts”. For more interesting analysis, see here, and remember that graphs drawn in MS Paint rarely reflect actual published work.


5 thoughts on “The Real Dunning-Kruger Graph

  1. It was all going along swimmingly until you got to the part about who the subject group was. You are right that it’s not representative, and I wonder if this WEIRD set of subjects tells us much.

    OTOH, I have never believed that all people of low ability in something don’t know that and think they are just fine, just some of them. Bullies believe they are more popular and have higher class rank than is real, but there are people who know they are unpopular or that they are at the bottom of the class, and are correct.


    • Yeah, I waffled on calling that out more, but I do think the underlying premise is still really likely. I would also guess that you are correct that it’s an average dragged down by some folks more than others, and the rest of it may just be a natural tendency to avoid ranking ourselves as the best or worst at anything.


  2. As the instigator of many home dis-improvements (and a veteran in IT) I think the overall premise is pretty accurate even if the infographic doesn’t match the study. I don’t think people reference Dunning-Kruger so much as a claim that the lowest quartile thinks they are better than an expert but, more in line with your restatement, that people with little training and experience often vastly underestimate the gap between the performance/knowledge of a novice and a journeyman fully familiar with subject but still not a master craftsman/expert. Lots of people seem to think Google can substitute for actual experience or education.


    • Yeah, I think some of what I was seeing is just people getting sloppy. I agree the milder version seems completely plausible and hard to argue with.

      At the hospital, our nurses complain all the time that people will argue things with them but then immediately acquiesce when the doctor steps in the room and says the same thing. The inevitable “see, you just have to know how to handle people” comment from the doctor probably doesn’t help.


  3. Pingback: Grade Prediction by Subject | graph paper diaries

Comments are closed.