An Anecdote About Paranoia and Baseline Assumptions

The Assistant Village Idiot has re-posted one of my favorite anecdotes of his. For those not familiar with him, he has 40+ years experience in a state mental hospital. It’s short, so I’ll repost it in its entirety here (source):

A paranoid patient of ours had taken the book 1984 out of the patient library.  His particular paranoia is very much concerned with thought reading and thought broadcasting. He is not a person one might expect to have good general knowledge of literature and political culture, and he did not have much preconceived notion what it might be about.  He had heard somewhere it was an important book.  We were a little concerned what he might take away from the book, but we don’t get much involved in people’s selections.

He found it sad.  This guy had a girlfriend, but he lost her.

He didn’t really notice the paranoia-inducing parts of the book.  Those were just normal background to him

I think about that a lot, most often when I see a poll question asking people how they feel about current events or to compare previous years to this one. Getting people’s impressions without knowing their baseline can be highly misleading.

5 Disorders With Suprising Sex Differences in Diagnoses

There was a great article in the Atlantic this past week called “What Joe Biden Can’t Bring Himself to Say“.  The article focused on his (and the authors) struggle with stuttering, and contained a lot of fascinating information about stuttering that I never knew. Regardless of your political orientation and/or feelings about Joe Biden, it’s a very worthwhile read.

One of the interesting stats it contained was that stuttering was twice in common in boys than in girls, and that girls have a higher recovery rate. I was interested in this, because aside from a vague “girls have better verbal skills earlier, so I guess that makes sense” train of thought, there doesn’t seem to be a clear reason for this. I Googled a bit and found that no one is really clear on the reason for the discrepancy, though there is a thought that girls may tend to get earlier help because people expect them to be more verbal. This discussion got me interested in other similar disorders. We’re not surprised to hear that issues like prostate cancer or breast cancer are more common in one sex than the other, but some things feel like they should be more gender neutral.

I decided to look up a few other examples, though I excluded mental health type disorders since some of the sex differences there can be a bit controversial, and excluded diseases or disorders that seem to be linked to differences in behavior (such as lung cancer):

Student Debt: A Few Facts and Figures

I wasn’t intending to write about student debt this week, but oddly enough I had two different people ask me about it on the same day. The first was a younger coworker, who had heard Elizabeth Warren say that student loan debt disproportionately affected African Americans and was curious if that was true. He also wanted to where “average student loan debt” numbers came from. The second was the AVIs wife, who sent me a new report looking at the return on investment from different types of colleges, and wanted to know if family income was taken in to account.

Okay, so let’s take this one thing at a time. First, Warren’s comments came from a Tweet where she also shared this article. For clarity, I want to note that this article ISN’T by Warren, but her Tweet would seem to indicate some agreement. The article started with the stat that the average student loan debt was $37,102. My colleague (a fairly recent grad) thought that sounded low.

The average student loan debt number comes from this Chamber of Commerce report. Now this report was interesting because it is looking only at those people who graduated in 2017. When my colleague had first mentioned this to me, I had wondered if the “average” number was including those further from graduation, but it doesn’t. It did however, point out that student loan debt varies wildly based on the region of the country you live in:

New England is one of the highest average levels, so those of us living here will tend to see higher loan totals among our peers. Additionally, borrowers owing 6 figures are the fastest growing group of borrowers,  with about 2 million people owing over $100,000. While much of that is due to graduate school debt, one would suspect those folks would be concentrated in the same areas as the higher levels of debt.

So what about the disproportionate impact claim? Well, that also came from the Chamber of Commerce report. More black students take out loans to pay for their education (77% vs a national average of 60%), they take out higher amounts ($29,000 vs $25,000) and are more likely to default on their loans within 12 years of graduation (50% vs 36% of Hispanic students and 21% of white students). However, it’s important to note that this is comparing graduating students to other graduating students….it excludes those who didn’t go to college or didn’t graduate. Those groups are also disproportionately comprised of minorities. Inside Higher Ed has a good graph of the outcomes by race 6 years after people matriculate:

I think this is striking because it’s something I always wonder about when we talk about student loan forgiveness. Some people choose not to go to college or start in community colleges because of the expense of college. Forgiveness of debt may really help some people, but in many cases big choices have already been made. If there is inequality in those initial choices, then loan forgiveness will not solve those inequalities. We know that white 18 to 24 year olds are more likely to enroll in college than black or Hispanic students, so while the loans taken out by black students may be higher, the proportion of people taking them out is lower. It may still be true, but I think it should be clearer that we’re only talking about students here.

I think this is an important point because if we’re talking fairness, then we have to consider the poorest among us may be among the least likely to take out student loans to begin with. Indeed, an analysis of Warren’s plan showed only 10% of the benefits of this plan would go to the bottom 20% of households. By contrast, the top 20% of qualifying households would get 18% of the benefits. This may even out as the other parts of her plan were implemented (reduction in college cost going forward), but it’s something to consider.  (Note: I will fully admit I haven’t spent much time studying Warren’s proposal, so I may be missing something. Let me know in the comments if I’ve misstated something and I’ll update. I’m using her plan as an example to discuss the broader point about who currently carries student loan debt, not to knock her proposal over others. I really appreciate that she was willing to publicly release her plan for discussion like this.)

Alright so now to the last point….what’s the return on investment for college students? Well according to this calculation in the short run (10 years) it’s better to have gone to a public school than a private one, but by the 40 year mark it’s better to have gone to a private school. For example, my alma mater is Boston University. At the 10 year mark, it’s the 3,318th best ROI in the country. By year 20 post grad, it jumps to 464. By year 30, it’s at 142, and by 40 years it’s almost one of the top 100 best values at 116. The calculator is fun to play around with because you note some interesting patterns. Small schools in the Boston area do better than small schools in New Hampshire, which I will guarantee is a function of the graduates staying near cities. There’s no cost of living adjustment in alumni salary calculations. Some of the Protestant colleges my friends and family went to don’t fare well, but I’d suspect an inordinate number of graduates go in to things like social work, teaching or other ministry positions. In fact a good number of the “worst” ROI schools are actually Rabbincal colleges.

So are these institutions cherry picking rich students and then taking credit for their earnings? Possibly. Nothing in the calculations takes your family’s wealth in to consideration, so a kid who inherits the family business gets counted the same way as a kid who comes from nothing. Additionally, it’s interesting to note that some specialty schools do really well (pharmacy) and some really poorly (art). Schools who don’t have a lot of different types of graduates are very tied to how the professions associated with them are doing.

Additionally, families with money probably tend to send their kids to private schools to begin with. For example, Stephanie and Shane McMahon (children of Vince and Linda McMahon, owners of the WWE) both went to Boston University and then promptly went to work for the family business. I don’t know for sure, but I suspect they never looked at UCONN when they were applying. Now this wasn’t every kid at BU, but having even a handful of the already wealthy can be enough to boost your lifetime earnings scores. In other words, we don’t know if BU is a better deal for a kid from a middle class household who wants to be a high school teacher, or if UMASS would be equal in those circumstances. We only know that overall, BU grads do better 40 years out.

So overall we don’t really know what we don’t know here, but we do know that many college stats leave out some confounders (who didn’t go to college, who was going to have money handed to them regardless of college status). Overall I think they are good for getting a general sense of things, but up close they have some issues. Like a Monet painting or something.

Weather in Minneapolis vs Boston

I’m just getting back from a conference in Minneapolis, which is an interesting city to go to in November. I’m from Boston so cold doesn’t bother me, but it did strike me as interesting how much colder it seemed to be this time of year.

I did a quick Google search and found the climate data for Minneapolis and Boston and decided to do a quick comparison.

The average high temps in both states are nearly identical (+/-4 degrees) from March to October. In November the average high drops 10 degrees lower in Minneapolis, then the gap widens to 12-14 degrees for Dec-Jan, then back to a 10 degree gap for February, then back to similar climates for the rest of the year. My guess is that’s some sort of ocean moderating effect.

The precipitation levels were even more interesting:

Note: the temperature axes are different on these graphs, with the Boston one starting at 10 degrees and going to 90, and Minneapolis going 0-90. Still, you see that Boston doesn’t get the same level of “dry winter air” that Minneapolis does. I felt that when I got my first nosebleed in years on day 3 there.

Always interesting to see the side by side.

What I’m Reading: November 2019

My migraines have been in full swing this week, so we’ve got a few lighter ones here. Like the Audobon “What Kind of Owl Are You?” quiz. I’m apparently a spotted owl, but that may just be because a dark woods sounds good right about now.

For those who Tweet, if you ever want to see how many words you’ve racked up over time, this link hooks you up.  It tells you what your Twitter feed would be if it were a book. I’m slacking at “Where the Wild Things Are”. The goal apparently is to beat Proust at 1.5 million words.

The above led me down a rabbit hole of “longest novel” Googling, which got me here. Turns out a lot of long novels end up with controversy over whether they are one or many books. Regardless, I’d only even heard of 3 of these. Interesting.

For a slightly longer read, I thought SSCs post on the fall of New Atheism was pretty interesting. As someone who blogged for one of the websites involved in all this for a few years, I’d say Scott hits on a lot of interesting things, and he’s right that more people should be asking “what happened here?”. My two cents: I think people involved in the movement were there for two different reasons. One group rejected religion primarily because they believed religion opposed science and reason, the other because they believed religion promoted oppression. When the second group started to accuse the first group of being oppressive, they were upset to find the first group didn’t care as much as they’d assumed. When the first group started hearing about oppression, they got upset because they believed it to be a secondary concern. I think this was a case of finding out the hard way that the enemy of your enemy isn’t always your friend, but if any readers who were involved have other thoughts I’d like to hear them.

On a related note, the AVI wrote me this week to tell me he wants to lend me The Genesis of Science, which chronicles the history of science and the church in the middle ages. It looks interesting.

 

From the Archives: Blinded Orchestra Auditions Update

Welcome to “From the Archives”, where I dig up old posts and see what’s changed in the years since I originally wrote them.

A few years ago after the now infamous James D’Amore/Google memo incident, I decided to write a post about one of the most famous “unconscious sexism” studies of all time. Known as the “blinded orchestra auditions” study, it is frequently used to claim that when orchestras started hiding the appearance of the applicant by using a screen, they increased the number of women getting a job.  When I started reading the paper however, I realized the situation was a bit more complicated. Sometimes women were helped by the blinding, sometimes they weren’t. It certainly wasn’t as clear cut as often got reported, and I thought there was some interesting details that got left out of popular retellings. Read my original post if interested.

This post was decently well received when I put it up in 2017, but I was surprised back in May to see it suddenly getting traffic again. Turns out a data scientist from Denmark, Jonatan Pallesen, had written a very thorough post criticizing this study. That post got flipped to Andrew Gelman, who agreed the conclusions of the paper were much murkier than the press seemed to think they were.  He also pointed out that these observations weren’t new, and as proof pointed to….my post. That felt good.

After all this, I was interested to see my post spike again this week, and I wondered what happened. A quick jaunt to Twitter showed me that Christina Hoff Sommers had done a YouTube video explainer about this study, raising some of the same objections. She also wrote a Wall Street Journal op-ed on the same topic.

Now obviously I was pretty happy to see that my original concerns concerns had some merit. I had felt a little crazy when I originally wrote my post because I couldn’t figure out how a paper with so many caveats had been portrayed as such definitive proof for the effectiveness of blinding. However, I started to get some concerns that the pushback was overstepping a bit too.

For example, Jesse Singal (who I follow and whose work I generally like) said this:

I questioned this on Twitter, as typically when we say a study “fell” we mean failed to replicate or that the authors had evidence of fraud. In this case there was neither. All the evidence we have that these conclusions were not as strong as often repeated comes from the paper itself. I questioned Singal’s wording on Twitter, and got a reply from Sommers herself:

I think this statement needs to be kept in mind. While the replication crisis has rocked a lot of our understanding of social science studies, it’s a little incredible that so many people cited this study without noticing the very clear limitations that were presented within the paper itself. As Gelman said in his post “Pallesen’s objections are strongly stated but they’re not new. Indeed, the authors of the original paper were pretty clear about its limitations. The evidence was all in plain sight.

Additionally, while the author’s 50% claim in the concluding paragraph seems unwise, it should be noted that this is the paper abstract (bold mine):

A change in the audition procedures of symphony orchestras adoption of “blind” auditions with a “screen” to conceal the candidate’s identity from the jury provides a test for sex-biased hiring. Using data from actual auditions, in an individual fixed-effects framework, we find that the screen increases the probability a woman will be advanced and hired. Although some of our estimates have large standard errors and there is one persistent effect in the opposite direction, the weight of the evidence suggests that the blind audition procedure fostered impartiality in hiring and increased the proportion women in symphony orchestras.

Journalists and others quoting this study weren’t being limited by a paywall and relying on the abstract, because that stat wasn’t in the abstract. Those stats appear to have been in the press release, and that seems to be what everyone copied them from.

While I totally agree that the study authors could have been more careful, I do think they deserve credit for putting the caveats and limitations in the abstract itself. They didn’t know when that press release was put together that this study would still be quoted as gospel 2 decades later, and it’s not clear how much control they had over it. They deserve credit for not putting those stats in their abstract, and for making sure some of the limitations were mentioned there instead.

I’m hammering on this because I think it’s worth examining what really went wrong here. I suspect at some point people stopped reading this study entirely, and started just copying and pasting conclusions they saw printed elsewhere. This is a phenomena I noted back in 2017 and have dubbed The Bullshit Two-Step:  A dance in which a story or research with nuanced points and specific parameters is shared via social media. With each share some of the nuance or specificity is eroded, finally resulting in a story that is almost total bullshit but that no one individually feels responsible for.

While I do think the researchers bear some responsibility, it’s worth noting that there’s no clear set of ethics for how researchers should handle seeing their studies misquoted. Misquotes or unnuanced recitations of studies can happen at any time, and researchers may not see them, or might be busy with an illness or something. I do think it would be interesting for someone to pose a set of standards for this….if anyone knows of such a thing, let me know.

For the rest of us, I think the moral of this story is that no matter how often you hear a study quoted, it’s always worth taking a look at the original information. You never know what you could find.

Short Takes: Perception vs Reality vs Others

I was going to do a normal “what I’m reading” column this week, but I thought so much about the first two links I just decided to turn it in to a short takes. I’m seeing a lot of interesting parallels between these two articles, so I wanted to highlight a few things.

The first link was a Medium post called “How to Change a Mind“, an excerpt from an upcoming book called “Stop Being Reasonable: How We Really Change Our Minds“. It tells the story of a woman named Missy, and how she got her husband Dylan to leave a cult. The whole story is worth reading, but it the ultimate conclusion is worth pondering. Dylan didn’t leave because she was able to point out some of the ridiculousness in what the cult believed (though she tried), but because one of the leaders ended up offering a large and objectively unfair critique of Missy ending with an encouragement to leave her.

This was the proverbial straw that broke the camel’s back. Dylan knew his wife had been nothing but kind and supportive, and the attempt to cast her in a different light caused him to doubt the leaders in a way he never had. As the article says “Dylan did not need to lose his faith in what his elders were saying; he needed to lose his faith in them.” And lose it he did. He spent two days straight Googling every critique of the group that was out there, then severed his ties. He describes his faith in them like a faucet that just got suddenly shut off.

The article does a good job of contextualizing this, and pointing out the lessons here for all of us. While most of us have never joined a cult, many of us take the word of others for granted on many topics. We have faith in certain sources, and barring any challenges will continue to believe those things. Maybe the topic is history, chemistry, math or some other topic we are aware of but didn’t study much personally. Even something as simple as another person’s name is mostly taken on faith. The point is, we can’t check every single thing that comes across our path, so we all have short cuts and rubrics to decide what information we believe and what we don’t. The point of this story is that the “who” part of that rubric can at times be more important than the “what”.

Given that, it was interesting that this next link landed in my inbox this morning “The Dangers of Fluent Lectures“. The article is based on a study that compared Harvard freshmen who took a physics class with lots of well polished lectures (passive learning) and those who took a class that made students work through problems on their own before explaining the answers to them (active learning). The results were interesting. Those who sat through the nicely polished lecture believed they learned more, but those who sat through the active lecture actually learned more:

There’s a couple theories about why this happens, but I think at least some of it has to do with the first article. Feeling that you are in the presence of someone hyper-competent could end up giving you the impression that you are more competent than you are. The active learning forces students to focus on their own deficiencies, while the passive learning lets them ignore that and focus on the professor. As the study authors say “novice students are poor at judging their actual learning and thus rely on inaccurate metacognitive cues such as fluency of instruction when they attempt to assess their own learning.” Again, it’s not always what you believe, it’s who.

Now there’s a couple caveats with this study: it’s not clear what would have happened if they had tried this study on 4th year students who were doing more advanced work, or if they had tried this at a state school rather than Harvard. They also mentioned that the kids in the study weren’t given any warning about teaching methods up front. In a later version of the study, they spent a few minutes in the first lecture teaching kids about active learning methods and the proof that they help students learn more. The students subsequently rated those classes as more effective, and said they felt better about the learning methods.

As always, we continue to be poor judges of our own objectivity.