Posted on 26 Dec 2010 21:32
By Eric Troy
Scenario: Guy comes on to a bodybuilding board and posts a thread about how such and such a supplement or program put 20 pounds on him in a month and grew two inches on his arms. Ignoring the obvious conclusion that the guy is bullshitting, the board members hit him with one of the following answers: "Anecdotal Evidence. Doesn't prove Anything." Or, "Case study of one. Doesn't prove anything."
Supposing it is 'evidence' of any kind which is it? Anecdotal evidence or a case study? It's anecdotal evidence. Guys coming on to bodybuilding boards talking about their new guns don't do case studies.
Let's not blame the forum people again. That cop out is getting old. They get this stuff from so called experts who throw around both those terms as if they are interchangeable. I've complained before about strength professionals and the pretense to science by people who not only don't understand how science is done, but don't even have a clear notion of valid scientific reasoning. Lip service again.
When someone tells you case studies don't prove anything they are belaboring the obvious. Case studies are not meant to prove anything. The conclusions reached from a case study, whether the study is of one person or a small group, are never meant to be generalized and most case studies are not setting out to "prove" any cause and effect relationship. Case studies are instead qualitative, exploratory, and descriptive. Generalized theories are not developed from a case study. In fact, the main complaint about Sigmund Freud in the psychology world is that all his theories were developed from individual case studies…and his theories therefore are and should be bunk.
It is frequently reported that case studies are a form of anecdotal evidence. Although it is fair to call a case report an anecdote, the idea that they are used for evidence is the problem. Case studies are not meant to be evidence. Anecdotal evidence is. A case study, at it's heart, is much like any other research paper. Anecdotal evidence, on the other hand, has nothing to do with research whatsoever. The desire to lump case reports in with anecdotal evidence is borne of a need to categorize different "types" of scientific evidence as strong or weak.
For instance, in Physical Therapy Professional Foundations: Keys to Success in School and Career, Kathleen A. Curtis lists three types of evidence, empirical, analogical, and anecdotal. Empirical evidence gets the gold letter score, analogical evidence comes second, and case reports, lumped in with anecdotal evidence, comes last, being the weakest. Curtis points out that not many studies of complex human phenomena, such as rehabilitation and physical fitness, meet a set of rigid criteria.5
Clearly there is a gross misunderstanding about how scientific wheels turn. The idea that empirical evidence is the high tech modern engine that drives science and everything else is a slow, lumbering beast is prevalent. However if all knowledge had to be derived solely from empirical evidence then the acquisition of knowledge would be slow indeed! To reject all but the best "empirical evidence" would be to engage in bad thinking and decision making. It's like looking for gold in a copper mine.
Say you want to understand how and why a certain social situation exists in certain work settings. You can't set up controlled experiments. You can't control the behaviors or the outcome of those behaviors. The situation is highly complex with a great many variables. To describe it, explore it, or explain it you use a case study.
Now, case studies of more than one person or instance will tend to be a bit more rigorous and valid than a case study of one. But it is still not generalizable. It is, however, a scientific inquiry. Case studies are used for many different reasons and the frequent assertion that case studies are not scientific is incorrect. I am by no means a scientist nor do I pretend to know all the different uses of case studies. But one thing case studies can do is lead us to hypothesize.
Instead of giving us answers they can help tell us what questions to ask. If you don't think that this is useful…you are kidding yourself. There is hardly a day that goes by that I don't have someone say to me, I need some answers but I'm not even sure what questions to ask. Well, the questions aren't always clear in science either.
Indeed, in many instances, case studies may be the first inquiry into a certain phenomenon. In medicine, this is certainly the case. One only needs to think of a patient presenting with an unknown and never before documented condition and imagine that his or her doctor prepares a careful case report of this patient's unique case. This may constitute the first time this condition has ever been documented in any way and such a case report could be a basis to recognize further cases and to drive further inquiry into this new disease or condition. Milos Jenicek in the book Clinical Case Reporting in Evidence-Based Medicine said, in fact, that case reports are the "first line of evidence, where everything begins." 1.
If science is the accumulation of knowledge, then in order order to gain knowledge of a particular thing we must first recognize it's existence and be able to describe it based on our prior knowledge! The great variety of human cancers is an example of how case reports have often been the first recognition of a particular condition and many particular cancers were first described in a physician's case report; as was, for example, Hodgkin's disease. 2
Case studies are used most often in classroom settings. They are a way to bring real-world situations into a teaching situation and make them accessible and understandable for the student. These are teaching cases. Not all case studies are done on unique things. Many case studies are done on examples of well-known phenomenon. Psychology students are presented with these types of case studies and often required to write their own. This type of study could be about a particular problem and the idea is to propose a solution to that problem. Or, the study might be an example of problem solving in itself. Often, case studies are meant to show examples of how NOT to do something! This is a very useful function indeed.
There is disagreement about the use of case studies as research tools. According to David P. Stevens of the Dartmouth Institute for Health Policy and Clinical Care, "The clinical case report is viewed with circumspection in an era of clinical evidence that is often measured by the rigor of the randomized controlled trial.3 Nevertheless, case reports have enjoyed resurgence, particularly given their accessibility provided by electronic publication. 2, 4 At a minimum, they broaden clinical experience from the comfort of ones study, and at their most profound, they provide fresh insights into pathogenesis and treatment." 3
I'll leave that to the professionals to grapple with. The purpose of this post is to differentiate between case studies and anecdotal evidence. The value of case reports were summed up by Vandenbroucke. 4 Case studies:
- permit discovery of new diseases and unexpected effects (adverse or beneficial)
- permit the study of mechanisms
- play an important role in medical education
- have a high sensitivity for detecting novelty and so are part of the foundation of medical progress
- provide many new ideas
However, while anecdotal reports are simply individual accounts of individual experience, they are not studies. Studies have an underlying organization and plan. Anecdotal reports do not. Case studies themselves could be called a form of anecdotal evidence but the idea of a carefully written study of a particular case is certainly not what we mean by the term. And case studies must be clear, concise, and focused. They are not just about telling a story but about communicating how it fits in to our existing knowledge…why is it important?
Many strength and conditioning ideas are put forth by training authorities based on their observations of their own trainees. These should be considered anecdotal unless the claimant can provide an organized study of the situation. However, a case study is NOT just a careful narrative of some training situation. Case studies ARE research. So first the question (or questions) must be defined. And the questions must be kept to a minimum. Then the cases must be selected based on whatever criteria the researcher has determined. Then data must be collected and analyzed, the report written.
Since case studies ask specific questions they are FOCUSED. Anecdotal reports have no focus since they were never meant to be reports in the first place. They are simply stories told after the fact. In fact, the very act of composition clearly separates the case report from anecdote. Writing begins early in the analytical process and continues throughout the investigation. A case study is more a "first-hand" report "from the ground", therefore. Anecdotal evidence is rarely first-hand. Remember, there are degrees of "first handedness". An account told ten years after the fact is not as "first hand" as an account recorded while the actual events were going on.
The most frequent misunderstanding about anecdotal evidence is that if something is anecdotal it is simply "untrue". While such a report may indeed be untrue that is not always what we mean by the term "anecdotal evidence".
According to Wikipedia "the expression anecdotal evidence has two distinct meanings. Evidence in the form of an anecdote or hearsay is called anecdotal if there is doubt about its veracity; the evidence itself is considered untrustworthy. Evidence, which may itself be true and verifiable, used to deduce a conclusion which does not follow from it, usually by generalizing from an insufficient amount of evidence. For example "my grandfather smoked like a chimney and died healthy in a car crash at the age of 99" does not disprove the proposition that "smoking markedly increases the probability of cancer and heart disease at a relatively early age." In this case, the evidence may itself be true, but does not warrant the conclusion." 6
So, an anecdote is a story. A story may be true or untrue. It need not have anything to do with presenting "evidence". However, if a story is used as evidence, we may doubt it's truth and call it "anecdotal" when we simply mean "unreliable" or "untrue".
Most often, however, when we refer to anecdotal evidence in the fitness world we mean that the conclusions given do not follow from the story. Examples of this are easy to come by in the bodybuilding and strength training world and this requirement, that the conclusion must follow from the evidence is not at work.
A very typical example of anecdotal evidence in bodybuilding was given by Jamie Hale in "The Fitness Skeptic""
"I supplemented with BCAAs (branch chained amino acids) when I was preparing for my bodybuilding show. I didn’t lose any muscle. Therefore, supplementary BCAAs prevent muscle loss." - The bodybuilder also weight trained, supplemented with protein shakes, ate a boatload of whole protein foods (means high levels of BCAAs) and so on. The point is there is no way that it could be accurately determined that the BCAA supplement was the reason the bodybuilder did not lose muscle. This is also an example of confusing correlation with causation. - Jamie Hale
What we see here is that anecdotal reports do not take into account the many variables at work. Probably the most frequent use of anecdotal evidence in regards to training concerns training injuries or lack thereof:
- "I've always trained this way and I've never been injured"
- "I've always worn a belt when deadlifting and I've never injured my back"
Here is one of my favorites, concerning the use of a false grip on the bench press:
- "I've always used a false grip and never had an accident, plus I can bench press over 300 lbs".
Now we see just how silly anecdotal evidence can be. A false grip, during the bench press, is when the thumb is allowed to remain behind the bar, in line with the hands and fingers, instead of adducting and wrapping around the bar. Arnold is "demonstrating" it in the image below. Many trainees find this kind of grip more comfortable. But without the thumb in place to provide a buttress against the bar it can easily slide off the palms of the hands. The results can be quite horrible.
Arnold did it; it must be safe, right?
In order for the bar to rest firmly in the hands the wrists must always by allowed to bend back, or extend, so that the bar sits on the palms. Without the thumb as a buttress even the slightest flexing of the wrists can cause the bar to slip out of the hand to come crashing down onto your chest, or even worse, your neck.
Having never had a mishap in no way proves the safety of this grip! And of course accidents have happened. Saying "it's never happened to me" is like blithely stepping out into the street and playing chicken with a moving car. Stupid.
This brings us to another feature of anecdotal reports that are at work in the strength and bodybuilding world, and I discuss this in more detail in "Why Programs Work". Failures and accidents are rarely reported compared to "successes". Not many people will tell you about the stupid thing they did using a false grip and having the bar drop onto their chest. You can be sure they will no longer use a false grip but they will not be eager to share this story.
Back in September of 2009, Stafon Johnson, a University of Southern California running back couldn't help but have his bench press accident reported to the world. Stafon was bench pressing when the bar slipped out of his right hand and fell onto his neck. He was said to have been coughing up blood and needed a tracheotomy to breathe. Severely injuring his larynx and neck, according to news stories at the time, Savon's larynx had to be surgically realigned.
Savon was most assuredly using a false grip. The bar does not slip from one's hands using a buttressed grip. Unfortunately, many people were quick to blame his spotter or to just say "these things happen during the bench press", further clouding the issue of proper grip and technique. With a proper grip, it is quite possible for catastrophic failure to occur, without warning, during the bench press, resulting in the bar coming down on the chest but without leaving the hands. This can happen to quickly for a spotter, or spotters, to correct. It is absolutely impossible for a spotter to catch a barbell in freefall, with no warning, and arrest it in time. Spotters are not catchers! Use a proper grip!
Regardless, those who have been using this grip for years will stubbornly cling to it and will be eager to "report" on their success. This is why we do not only hear about the lack of an accident but are also regaled with the fascinating but mythical correlation between the false grip and a huge bench press!
The "facts" may be true. The trainee may never have suffered an accident while using the false grip and the trainee may have quite an impressive bench press. But the conclusions, that the false grip is safe and helps one achieve a bigger bench press, does not follow from these "facts". Likewise, as with the example given above your grandfather may well have smoked like a chimney only to perish in a car crash at the healthy age of 99. But smoking still kills many people at a relatively early age.
This page contains affiliate links to Amazon.com. We have not been compelled in any way to place links to particular products and have received no compensation for doing so. We receive a very small commission only if you buy a product after clicking on one of these affiliate links.
This page created 26 Dec 2010 21:32
Last updated 06 Aug 2016 19:28