One of the more important (but often overlooked) parts of research is the choice of control group, i.e. what we are comparing the group of interest to. While this seems like a small thing, it can actually have some big implications for interpreting research.  I’ve seen a few interesting examples recently, so I figured I’d do a quick list here:

First up, a new-to-me article about personality assessments in a traditional hunter-gatherer tribe. I’ve mentioned the problem of psychological research focusing too much on WEIRD (Westernized, Educated, Industrialized, Rich and Democratic) countries before, and this study sought to correct that error. Basically, they used the “Big 5” personality testing model and then tried to assess members of a traditional South American tribe according to this “universal” personality measurement. It failed. While it seemed like extraversion and conscientiousness could actually translate somewhat, agreeableness and openness were mixed, and neuroticism didn’t translate all that well. They ended up with a “Big Two”, which were basically an agreeableness/extraversion mix (pro-sociality) and something like conscientiousness (industriousness). They talk a lot about the challenges (translation issues, non-literate populations, etc), but the point is that what we call “universal” relies on a very narrow set of circumstances. Western college kids don’t make a good baseline.

Second, a new dietary study shows that nutritional education can be an effective treatment for depression.  It’s a good study, and I was interested to see the control group was given increased social support/time with a trained listener/companion type person. At 12 weeks, almost a third of the diet group were no longer depressed, whereas only 8% of the control group were feeling better. Interesting to note though: this was advertised as a dietary study, so those who didn’t get the diet intervention knew they were the control group. There was a higher dropout rate in the control group (25% vs 6%), and interestingly it was the most educated people who dropped out. Gotta admit, part of me wonders if it was the introverts driving this result. Just wondering how many people really enjoyed the whole “hang out with a stranger who’s not a therapist” thing. I would be interested to see how this works when paired with some sort of “hour of general relaxation” type thing.

Finally, after putting up my pre-cognition post on Sunday, I realized there was a Slate Star Codex post a few years back about the Bem paper that I wanted to reread. It was called “The Control Group is out of Control” and took the stance that parapsychology was actually a great control group for all of science. Given that you have a whole group of people attempting to follow the scientific method to prove something that most people believe doesn’t exist, they end up serving as a sort of “placebo science”, or an indicator of what science looks like when it’s chasing after nothing.

He has some really interesting anecdotes here about the amount of evidence we have that researchers are influencing their own results in ways that seem nearly impossible to control for. For example, he talks about a case in which rival researchers who supported different hypotheses and had gotten different results teamed up to use the same protocol and watched each other execute the experiments to see if they could figure out where the other one was going wrong. They still both ended up proving their preferred hypothesis, and in the discussion section brought up the (mutual) possibility that one or the other of them had hacked the computer records. That’s an odd thing to ponder, but it’s even odder when you wonder what this means for every other study ever done.