“Show me the peer-reviewed publication.”

This is a common refrain used, particularly by the scientifically-minded, when talking to someone about the latest trendy diagnosis, treatment or other medical "discovery." It used to be relatively easy to use this method to assess the reliability of new information. If something was published in a peer-reviewed journal, it was probably at least reasonably sound (not always, but it did provide a greater level of assurance).

Now, it’s getting tougher.

One reason for this is the proliferation of online journals in particular. Some are legitimate journals that have been created by people that perceived a need in a specific area. Yes, with more research, more journals can be supported. Some good journals reject 50-90% of submissions, and many rejected papers are still valid and indeed important. Having a wide range of journal options is important because everything can’t be published in Science, Nature or the New England Journal of Medicine.

But, there’s a limit.

I review a lot of papers for journals every year. Sometimes, I end up reviewing the same study a few times for different journals, after I’ve recommended rejection previously. Some of these get accepted if they are a better fit for the journal (e.g. scope of the study, priority of the type of research for the journal) or they end up making substantial changes to improve the manuscript. However, I’ve also rejected some papers a few times and eventually seen them published in pretty crappy journals. The sad reality is that marginal research can usually be published somewhere if the authors are persistent. Most people in academia know the strengths of journals in their field and can approach a paper in a "bottom-feeder" journal with care, but the general public doesn’t know that, may not have access to the full paper, and probably wouldn’t be able to assess the quality of the study anyway. So, just knowing that something’s been published in a peer reviewed journal these days only tells you part of the story.

Even worse, thousands of new journals have been created by for-profit groups. That doesn’t inherently mean they are bad, but it’s getting clear that for some, the size of the cheque is probably more important than the quality of research. I get emails from them pretty much every day from one group advertising a new journal, asking for journal submissions or requesting that I serve on their editorial board (yet curiously, I’ve never been asked to peer-review a paper for one). Unsuspecting academics have signed onto editorial boards thinking they were legitimate and have had a hard time getting their names removed once they realize the problems.

Some of these journals use impressive sounding names or ones that are very close to highly respected journals, further confusing the reading public.

All these things make it hard for the average person who wants to explore something a bit further to know what information can be trusted. Not only do you need to think about whether a study is published, you need to figure out if the study is valid and published in a reputable journal where the research was actually scrutinized, not published just because the authors paid a few thousand dollars for the publication fee.

Some of these for-profit groups sponsor scientific conferences that have the same issues. Normally, conference presenters are invited by scientific committees made up of experts in the area and/or are chosen through submission of research abstracts that undergo review. The more dodgy conferences solicit abstracts (I probably get an email a day from these) and get people to attend conferences, but then send them a bill at the end. So, they sucker paying attendees by making the conference look real (sometimes by putting names of high profile people as organizers or presenters without their knowledge) and get more money by charging presenters (thereby getting money from people who will pay to get their research presented or others with good research that are unsuspecting). In the end, you might be left with good research or invalid research, and it’s hard for the average person to sort this out.

There’s a good article in Nature about this subject, as well as a related article recently published in the New York Times.

How do we get this get under control? I have no idea.