When we’re faced with making decisions about treatments, “show me the evidence” is a common refrain. There are a lot of products being sold as “cures” but with little or no evidence to back up their claims. So, looking for any scientific backing is important. One of the highest levels of evidence would be a controlled clinical trial that’s published in a scientific journal.
Sometimes, it’s not quite that clear.
One problem is predatory publishers. There’s a booming market in these. Journals spring up out of nowhere, often with official sounding names or names mimicking established journals; however, the main criteria for publication is the ability of the author to pay a couple of thousand dollars in publication fees.
Another issue is journal overload. Somewhat related to the above, there are just so many journals now (predatory and otherwise) that it’s possible to eventually find a home for even pretty poor research. People in the scientific community may recognize a bottom feeder journal and be wary of a paper there, but the broader public doesn’t know that.
Yet another problem is when bad research gets published in a reputable scientific journal. I had to apply my 24-hour rule (don’t post anything that gets me worked up for 24 hours after I’ve written it so I can tone it down when I’m less cranky) to the latest example of this.
I found out about this paper via a press release by a company claiming their cranberry-based supplement was as effective as antibiotics for preventing urinary tract infections.
- My first thought was… doubtful. Cranberry studies haven’t been overly promising in animals and research involving commercial veterinary products is often pretty weak.
Then they referenced an independent study that was just published in the American Journal of Veterinary Research (Chou et al, 2015).
- My next thought… interesting. It would be nice to see some good data.
Then I read the paper… and my blood pressure rose significantly.
There are lots of issues and, long story short, from my perspective, this paper is an embarrassment to veterinary research, the journal and the authors (wouldn’t you have liked to see what I wanted to write 24 hours ago?).
- The study compared the supplement to antibiotics for prevention of UTIs. First off, we don’t use antibiotics to prevent recurrent UTIs so the comparison makes no biological sense. Why did they do it? Presumably to say (as they do) that antibiotics were no more effective than their product. With an underpowered study, that’s easy to prove. What they needed to do was compare their product to a placebo. With an underpowered study like this, their data would have shown no difference between their product and the placebo either. Not as good for marketing.
- There are standard criteria for the design and reporting of clinical trials. This study fulfills pretty much none of them. They don’t even do basic things like report the sex of the dogs that were involved (important for UTI risk) and whether they randomized them between groups.
- The study involves two groups of dogs. Each group had 6 dogs. From a statistical standpoint, that’s useless. They market this study by saying the two approaches are equally effective. That’s waht’s known as a “non-inferiority” trial. For such a trial, you need to have enough subjects to make sure you can actually detect a difference. If we say that 4/6 dogs in each group would be expected to have developed a UTI if untreated, and if the cranberry product is no worse that 30% worse than antibiotics (not exactly a high threshold), 30 dogs per group would be required. If the expected rate of infection was lower or we wanted to set the threshold for the maximum difference to a smaller value, even more dogs would be required. This small number of dogs tells us basically nothing.
Does this cranberry supplement work?
- Who knows? This study tells us absolutely nothing more than we would have known without it. But now a company has a “published scientific study” to cite, all because of a flawed peer review process.
Is there a problem with using the supplement?
- No. I’m not saying there’s something wrong with it, and this post isn’t anti-supplement (either this one in particular or cranberry in general). It may very well work. I just know absolutely no more about the potential that it works after reading this study than I did before.
We need good evidence. We need researchers to design proper studies. We need journals to do their jobs with regard to proper scrutiny of research before allowing poor studies to be published, which ultimately only muddy the waters when it comes to finding reliable information.