Posted on 10 Jan 2012 16:22
Raise your hand if you have recently read a news or magazine article, on the web or elsewhere, explaining the results of one study and making concrete conclusions based on that one study. Yep, all of you. I figured as much.
A study in Australia revealed that young women fight off colds better than young men. Case closed. Not. First of all, "a study in Australia" is not an appropriate reference. No reference, no credibility. Second of all, there is no way that ONE study could possibly "reveal" conclusively that young women have colds that go away quicker than young men.
I know that you probably don't care about a study that says young women have shorter colds. But what if the study reveals something you'd like to believe? What then? I know it is tough to not cling to any little tidbit that confirms our favored reality. Trust me, I do. For instance, you know those toning shoes? Sketchers Shape-Ups, Reebox EasyTone, MBT…you've seen the ads. There was one study from The American Council on Exercise which suggested that, low and behold, those shoes don't do any of the things they claim to do, such as activate the calves and hamstrings better and burn more calories. The study 1 by Porcori, et al. was divided into two trials, with 12 females each, one group for muscle activation and one for caloric expenditure. I have very little doubt that these shoes are complete and utter crap. But one study of 12 just ain't enough for me to go around saying we have conclusive proof. I'd say we have a good indication, but you won't see a front page banner on GUS proclaiming this NEW STUDY!
And let's be fair here. Concerning the referenced study, the author of the related article 2 on the Ace Website complains that the studies performed by the manufacturers of these shoes are not peer-reviewed, failing to see the irony in the fact that neither is the Ace study, having not been published in a peer-reviewed journal! Something is amiss, is it not? In fact, according to an article by Denise Mann 3 the Skechers company president actually called out the study based on this fact. Man, you've got a problem when the company guilty of BS in-house "studies" proving their products effectiveness actually call you out on your study not being peer-reviewed. Could Ace be biased as an organization? Sure, it's possible. Should we take Ace funded studies as seriously as other studies funded by independent sources? Maybe not. After all, Ace is about producing fitness professionals who make a living helping people get fit. Still, this does not mean that the study is completely bogus. It's just not enough and questionable, as all studies are. And before you ask, Mann is pretty good at reporting on these things and yes, she actually references her articles.
None of this means that I won't think you are silly for spending over 100 bucks on any of these shoes. I don't need studies, most of the time, to tell me when a fitness product is not worth the material it's made from. But I'll need to wait for some confirming evidence before I start referencing research to support my opinion. Right now, it's just an opinion backed up by one small study, which may have its flaws, including its lack of scholarly publication. If you want to know, you go read it; I actually referenced it! Boy was that tough! I want to get back on track.
From my perspective there are two problems. One is that news and magazine articles routinely fail to properly reference the research they mention. Studies absolutely must be properly listed, with a link, complete journal citation, or both. If you cannot easily go find the study and read it yourself, the article's validity is suspect. Many times you will find, even once you search out the study in question, that the conclusions of the study's authors are not anything like the conclusions of the magazine article. This happens constantly, in fact. Not just in big publications but all over the web. Usually, people look for articles that seem at a glance to support what they already believe, and have no regard for what the study's authors actually conclude, if, that is, they actually conclude anything at all!
The other problem, from my perspective, is that it is nothing more than content baiting. The public is being led to believe that huge discoveries are being made in the realm of fitness and health almost on a daily basis. Single small studies are used for no other purpose than to support a provocative headline to draw you, the reader, in. These news and magazine outfits do not care if what they report is accurate and they surely do not care if it makes a difference in your fitness pursuit. If they did, they would research the articles much more widely. The typical formula for these articles goes like this:
1. Vague reference to some study, usually only with a place named, like Australia, or the last name of one of the researchers
2. Vaguely related quote by an expert in the same field, usually someone who has a vested interest in confirming the conclusions of the said study.
I would love to be able to have a subject for an article every time someone does a study related, in some way, to fitness. Then, all I'd have to do is check Pubmed every day looking for something juicy and write some half-ass post on it. I'd have a website filled with content and provocative headlines. I wouldn't even need to stick strictly to fitness. What about a "recent study" that shows that pregnant women who eat chocolate have healthier babies with better temperaments. What do you think the response to that would be? Yippee, I can eat chocolate! Like a pregnant woman needs an excuse, anyway. 4
Even when it comes to reading and analyzing single studies, though, there is a particular skill and expertize required, which the average journalist does NOT posses. There are, however, many good scientific blogs related to fitness that do a much better job. So, all you need to do is skip the big headline at the top of the Google pile and look for some blog entries by people like, say, Bryan Chung, who, it just so happens, weighed in on the barefoot shoe thing recently. You know, Vibram FiveFinger shoes and all of that. Just so happens there is a study from Ace on the barefoot shoes as well, according to this article. 5
Self Plagiarism, Salami Slicing, and Publish or Perish
It is good to be aware of emerging research but don't let the horse get ahead of the cart. It takes a while for enough data to be gathered to make meaningful conclusions. Before I leave you I want to give you one more thing to watch out for, in regards to scientific research published in peer reviewed journals. It's a "dirty little secret" if you will.
The unethical and damaging practice I am talking about is sometimes referred to as "self-plagiarism". Let's say I write what is essentially the same article, with the same thesis and conclusion, and publish with different wording in several different places. Is this a big deal? Nah. These are just informal articles and if I were to follow this practice it would just be for the purpose of fitting the article to a particular audience I was aiming towards. Plus, if I had a "parent" article I would most likely acknowledge this article in some way.
But what if a scientific researcher does this? Well then it is a very damaging and, to me, quite unethical practice. The major problem is using some data, or other part of an earlier study or paper in a new paper without cross-referencing it, to make it appear as the data, study sample, etc. is completely new and original to the study or paper in question. This absolutely puts a wrench in the whole kit-n-caboodle we call science. And it is done constantly. Most studies I find, in fact, are published in several different journals with no reference made to ever being published before. The question is, is it done deceptively? Well, if no mention is made of how parts of one paper overlap with another, it is always deceptive! If we think something is new, when it is not, it completely skews our evidence base, making it hard to make informed decisions.
Usually, once an article has been published in a peer-reviewed journal, it is not acceptable to re-publish it, and this could be considered copyright infringement.
Many times, you will see the same name pop up again and again in regards to a specific scientific claim and it will seem like said person is just publishing reams of research in regards to this claim. When you see this happen, consider that this person may be doing another form of self-plagiarism called salami slicing. This is when a researcher takes what is essentially one large study, that could be reported in one paper and slices it up into smaller chunks which he scatters all over the place, mostly to increase the author's number of publications. This makes the importance of the work seem greater than it actually is..call it a shotgun effect.
Don't blame it entirely on the researchers, though. Not many want to increase their publication count just for the sake of it! It's publish or perish and, according to Nature Materials: "Much of the problem arises not from an inherent desire among researchers to maximize their publication count, but from the conditions that are set by funding and appointment bodies, which determine what gets funded and who gets tenure. In the 'publish or perish' climate that has evolved over recent decades, overemphasis on the size of an individual's (and, increasingly, entire research group's) publication record as a means of quantifying their research output inevitably rewards quantity over quality. Moreover, this has the effect of abdicating responsibility for such assessment to the journals in which they publish — a responsibility that is neither appropriate nor desired." 6, 7
The explosion of researchers and journals is huge enough without the above practices adding to the pile. I mentioned the term "peer-reviewed" a couple of times in this post but it has started to become a joke since there are just too many papers appearing on a daily basis for most of them to be peer-reviewed.
You can bet that your average fitness article citing some research study (while not actually citing it) is completely oblivious to these problems.
This page created 10 Jan 2012 16:22
Last updated 21 Oct 2015 00:13