I Got a Problem, Don’t Know What to do About It

Help and feedback request! This past weekend I encountered an interesting situation where I discovered that a study I had used to help make a point in several posts over the years has come under some scrutiny (full story at the bottom of the post). I have often blogged about meta-science, but this whole incident got me thinking about meta-blogging, and what the responsibility of someone like me is when you find out a study you’ve leaned on may not be as good as you thought it was. I’ve been poking around the internet for a few days, and I really can’t find much guidance on this.

I decided to put together a couple quick poll questions to gauge people’s feelings on this. Given that I tend to have some incredibly savvy readers, I would also love to hear more lengthy opinions either in the comments or sent to me directly.  The polls will stay open for a month, and I plan on doing a write up of the results. The goal of these poll questions is to assess a starting point for error correction, as I completely acknowledge the specifics of a situation may change people’s views. If you have strong feelings about what would make you take error correction more or less seriously, please leave it in the comments!

Why I’m asking (aka the full story)

This past weekend I encountered a rather interesting situation that I’m looking for some feedback on. I was writing my post for week 6 of the Calling BS read-along, and remembered an interesting study that found that  people were more likely to find stories with “science pictures” or graphs credible than those that were just text. It’s a study I had talked about in one of my Intro to Internet Science posts  and I have used it in presentations to back up my point that graphs are something you should watch closely. Since the topic of the post was data visualization and the study seemed relevant, I included it in the intro to my write up.

The post had only been up for a few hours when I got a message from someone tipping me off that the lab the study was from was under some scrutiny for some questionable data/research practices. They thought I might want to review the evidence and consider removing the reference to the study from my post. While the study I used doesn’t appear to be one of the ones being reviewed at the moment, I did find the allegations against the lab concerning. Since the post didn’t really change without the citation, I edited the post to remove the citation and replaced it with a note alerting people the paragraph had been modified. I put a full explanation at the bottom of the post that included the links to a summary of the issue and the research lab’s response.

I didn’t stop thinking about it though. There’s not much I could have done about using the study originally….I started citing it almost a full year before concerns were raised, and the “visuals influence perception” point seemed reasonable. I’ll admit I missed the story about the concerns with the research group, but even if I’d seen it I don’t know if I would have remembered that they were the ones who had done that study. Now that I know though, I’ve been mulling over what the best course of action is in situations like this. As someone who at least aspires to blog about truth and accuracy, I’ve always felt that I should watch my own blogging habits pretty carefully. I didn’t really question removing the reference, as I’ve always tried to update/modify things when people raise concerns. I also don’t modify posts after they’ve been published without noting that I’ve done so, other than fixing small typos. I feel good about what I did with that part.

What troubled me more was the question of “how far back to I go?” As I mentioned, I know I’ve cited that study previously. I know of at least one post where I used it, and there may be more. Given that my Intro to Internet Science series is occasionally assigned by high school teachers, I feel I have some obligation to go a little retro on this.

 

Current hypothesis (aka my gut reaction)

My gut reaction here is that I should probably start keeping an updates/corrections/times I was wrong page just to discuss these issues. While I think notations should be made in the posts themselves, some of them warrant their own discussion. If I’m going to blog about where others go wrong, having a dedicated place to discuss where I go wrong seems pretty fair.  I also would likely put some links to my “from the archives” columns to have a repository for posts that have more updates versions. Not only would this give people somewhere easy to look for updates, give some transparency to my own process and weaknesses, but it would also probably give me a better overview of where I tend to get tripped up and help me improve. If I get really crazy I might even start doing root cause analysis investigations in to my own missteps. Thoughts on this or examples of others doing this would be appreciated.

 

11 thoughts on “I Got a Problem, Don’t Know What to do About It

  1. I’m of two minds on the first question, depending on whether the ethical concerns potentially affect the conclusion.

    If not (if it’s a matter of problems with informed consent or something like that), I’d blog about it if moved to do so. I might or might not be, depending on the nature and severity of the allegations. If it’s something where actual harm was done, far more likely to do so than if it’s some complaint that an IRB t was not crossed at some point. If I blogged about it, I’d add a link to other posts where I’d mentioned the study.

    If so, it falls into the same box as the second and third (failure to replicate, and retraction)—situations where the accuracy of the study itself is in question. In this case, I’d definitely add an update to posts where I’d used it. I might or might not substantially re-write the post in the update (while leaving the original text), depending on how much of my argument depended on that particular study. If the new information looked like it completely invalidated the study, I’d stop using it, but I might continue if the new information was less dire.

    Failure to replicate might or might not mean that, depending on the quality of the attempted replication. The issue of replication is interesting to me, because it seems to me that some people have swung from “All studies are credible! Peer-reviewed!” to “No study is credible if it failed to replicate! We can just throw it out!”, without stopping at a reasonable middle ground such as “Failure to replicate is an interesting mismatch which means more investigation needs to be done, and maybe look more closely at methods and populations for the two studies, but we can’t assume one was right just because it was done later.”

    A credible finding (or admission) that the PI had simply made up the raw data, OTOH….

    I’m sort of meh about the idea of a Mistakes page. I don’t think it replaces noting in the original post that there was a problem with a study.

    On the other hand, if it’s helpful to you, then there’s certainly no reason not to do it. Maybe write a blog post for each issue, with a “my mistakes” tag?

    Like

    • All good points. I should clarify I wasn’t thinking of a mistakes page as a replacement for updating posts, actually more an “in addition to” where people could see more discussion and other errors. I also realized I made the same type of error twice recently (copying an old statement from myself without checking to see if it was currently correct) and that’s when I started thinking I should keep a better eye on myself.

      I like the idea of using tags and will almost definitely do that in addition to whatever else I decide to do. Thanks!

      Like

  2. Part of this is a resources question. The fact that you are working on this blog as a passion project and are not making a living doing it is an important consideration. Are you doing more good through writing another blog post or through making corrections? The answer is going to be different in a lot of situations.

    For you in particular, the bar is higher since par of the purpose of your big is helping sort through the b.s. That makes retractions, failures to replicate and ethical questions moments of opportunity to further your objectives.

    Like

    • I do think there’s use in me making corrections, both because it’s a good idea personally/non-hypocritically and because I wish more people would admit they got things wrong. Be the change you wish to see and all that. I think one of the reasons we stray so far in to “post-fact” sentiments sometimes is because we’ve all become convinced our own personal errors don’t matter.

      I agree the time thing is an issue (though it makes it easy to offer a full refund to dissatisfied customers), which is why I wanted to think strategically about how to do this. Anyway, thanks for your thoughts 🙂

      Like

  3. I don’t think updating older posts has to be a time-sink. If you have a mistakes page on your sidebar, it should be there for every post, no matter how remote. You cans simply add a quick disclaimer in your old post: “Update: There is new information which suggests this organisation/study is not as solid as I originally thought. Please see mistakes page in sidebar for full discussion.” You might announce that you are looking for a replacement or similar example – someone might have one. For a full retraction, you should point out the blank space in the old posts.

    Greg Cochran at West Hunter announces when new information contradicts an earlier theory and is quite blunt about it. I don’t think he goes back to the previous posts to fix them, though. He seems to regard it as an ongoing discussion.

    Like

    • Yeah, I think the “disclaimer in the post” combined with a spot for longer discussions is a good idea that appeals to me. It seems to hit a nice balance between alerting people who are reading it while also allowing for a more meta-discussion. Also an excellent idea about calling for replacement papers.

      I was thinking if I did a retraction type page I might also do a post quarterly or twice a year updating people on what may have changed. Still kicking ideas around.

      Like

  4. Yep, what they said — except the possibility that Google will pick up one of your old posts that cited the problem study. I’d hate to see you get “used” if your corrections weren’t obviously available on the old post. However, I’m not as concerned about ethics (and that makes me feel bad just typing that!) as I am about failure to replicate.

    Like

    • Yeah, that Google part had me worried too. My 3 most popular posts are all from last year, and they still pull pretty good traffic. I definitely want to make sure I don’t get used as a reference somewhere without an addendum.

      Interestingly the ethics that concerned me in this case are heavily intertwined with failure to replicate. The concerns came to light when the researcher did a blog post praising a grad student who had taken a null data set and run the numbers through enough different analyses that she ended up with 5 papers. Given that approach, the likelihood that any of those papers will replicate later goes way down. There’s a lot to think through here.

      Like

  5. The google effect is a big problem. If somebody wants a post from the WayBackMachine, they’re on their own, but I’d like anything current to be reasonably accurate.
    I tend to run my blog on the Invisible Maniac principle (Out of sight, out of mind–line stolen from Gamow), but if I remember that something is wrong it makes me itchy not to put some kind of update in it. BOLDed.
    I don’t know if a sidebar “Corrections” page is the way to go, but I think I’ll start adding a “Corrected” tag to the posts that need correction.

    Like

  6. I say “other” on each question. What you write is true to you at the time it is written. If you find out differently at a later date, you can point that out if the study in question is related to your current works. You can not go through life apologizing for all past mistakes. I will contradict myself by saying timeliness
    matters. If you find the error within a few days or weeks of publication, then a retraction or correction is called for.

    Like

Comments are closed.