How Do We Know What Treatment Works?

How Do We Know What Treatment Works?

By Cindy Perlin.

How Do We Know What Treatment Works?

We’re often told that alternative medicine is unproven and pharmacological treatments are well validated. Is this true?

Cindy Perlin

It’s important for patients to understand the many factors that go into creating the body of medical evidence available, the degree of reliability of that information, and its applicability to their specific situation. I’ve included a brief discussion of research issues here to meet that need.

Evidence-Based Medicine

Since the early 1990s, there has been a push from many quarters for “evidence-based medicine” (EBM). According to the British Medical Journal, “The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.”[i]

According to EBM, all research is not created equal. The best evidence is considered to be meta-analyses of randomized clinical trials. A meta-analysis statistically pools results from many similar studies to draw conclusions about treatment effectiveness.

Second best, according to EBM, is evidence obtained from at least one randomized controlled clinical trial. A randomized controlled trial is a study in which participants are assigned, by chance alone, to receive one of several interventions. One of these interventions is the comparison, or control. The control may be a standard treatment, a placebo (a sugar pill or fake treatment), or no intervention at all.

Double-blind randomized controlled studies are held in even higher regard. In a double-blind study, no one—patient, researcher, or any other results evaluator—knows which participants received which treatment. This ensures that no biases or expectations will influence results. Of course, double-blind studies are not possible when the treatment is anything more complicated than a pill, as practitioners administering the treatment must know what they’re doing.

Nonrandomized studies and observational studies are considered less conclusive. Expert opinion is considered the lowest level of evidence for making treatment decisions.[ii]

In the EBM model, the evidence that comes from thousands of years of use and observation by healers across many cultures, such as in the use of acupuncture or herbs, is not considered valid, whereas results from a four- to eight-week study of a novel drug is given one of the highest rankings.

Flaws in the Research

EBM sounds good in theory, but its application to medical decision-making has left much to be desired. A 2014 article in the Journal of Evaluation in Clinical Practice reported that EBM has failed to achieve its main objectives: to improve health care outcomes and reduce health care costs. Summarizing critics’ concerns about EBM, authors Susanna Every-Palmer and Jeremy Howick wrote that one issue of major concern is that most studies available in medical literature are funded by parties that have a financial interest in the outcome, such as pharmaceutical companies. These entities manipulate the outcome of the research by their choice of issues studied and by manipulating the study design to favor their desired results. When the study still does not turn out as they had hoped, they fail to publish it and bury the results. Howick and Every-Palmer concluded that relying indiscriminately on industry-funded studies to make clinical decisions is like trusting politicians to count their own votes.[iii]

Howick and Every-Palmer cited the evidence for selective serotonin re-uptake inhibitor (SSRI) antidepressants as an example. More than 1,000 randomized trials of SSRI antidepressants have been conducted, resulting in seemingly overwhelming evidence that they provide clinically significant benefits.[iv] Doctors and patients were convinced that SSRI antidepressants (including Paxil and Prozac) worked. These blockbuster drugs posted global sales in 2011 of $11.9 billion.[v] SSRIs replaced older antidepressants and psychotherapy as the treatment of choice for depression. However, two independent analyses, published in 2008 and 2010, which included all published and unpublished studies on SSRI antidepressants, found that only favorable studies, and those that could be doctored to look favorable, were published. The pooled data from all studies showed that SSRIs are no more effective than placebo for the treatment of mild to moderate depression.[vi],[vii]

A review of pharmaceutical company-funded studies that compared newer, atypical antipsychotics—touted as safer and more effective than the older generation—revealed that the drug produced by the company funding the study was found to be superior 90% of the time.[viii] Studies were designed to make the study sponsor’s drug appear superior by such means as setting the dose of the competing drug too low to be effective or so high that the side effects would be intolerable.[ix]

Low-dose studies of natural treatments have also been funded by parties with an interest in discrediting competing treatments. This was the case with biofeedback, a treatment in which sensitive electronic instruments are used to measure the patient’s physiology. That information is used to teach the patient to control her physiology and resolve her symptoms without medication. Biofeedback is used to treat chronic pain, headaches, high blood pressure, anxiety, insomnia, and many other conditions. The 1986 book From the Ghost in the Box to Successful Biofeedback Training documented key “errors” researchers made when conducting these studies.[x] One research methodology flaw involved an insufficient number of training sessions. Successful biofeedback treatment requires the mastery of complex skills, which is highly unlikely in the limited number of training sessions allowed by many studies.[xi] Of 167 studies that took place before 1982 that used biofeedback to teach voluntary control of heart rate, 75% used only one to three sessions of biofeedback training.[xii] Other research flaws included insufficient length of training sessions; lack of homework exercises (standard in most types of biofeedback treatment); failure to instruct subjects that the goal was to learn to control their responses; failure to provide adequate rationales, instructions, and coaching; use of a relaxation control group for comparison to biofeedback training (biofeedback is relaxation training); and failure to train to mastery.[xiii] While initially biofeedback generated a great deal of excitement, after these flawed studies were published, the interest substantially waned.

Failure to disclose the risks of pharmaceuticals is another issue with published research conducted by the drug’s manufacturer. Pharmaceutical giants GlaxoSmithKline, Johnson & Johnson, Eli Lilly, and AstraZeneca have all been fined by the Food and Drug Administration (FDA) for hiding adverse effects of their products to make them more marketable.[xiv],[xv],[xvi],[xvii],[xviii] More specific to pain research, Purdue Pharmaceuticals was charged by the federal government with a criminal count of misbranding the narcotic OxyContin (oxycodone), with intent to defraud, and falsely promoting it as nearly addiction-proof.[xix]

In 2009, the world of pain medicine was rocked by the revelation that well-known anesthesiologist and researcher Scott Reuben had fabricated data in at least 21 published studies. Having received five research grants from Pfizer to study its drug Celebrex (celecoxib), Reuben went so far in some of the “studies” as to not bother enrolling patients and to make up all of the data. His findings naturally favored Pfizer. These falsified findings influenced surgeons and the way they treated postoperative pain all over the world, affecting the care of millions of patients.[xx] Reuben pled guilty to engaging in health care fraud. He was sentenced to six months in prison followed by three years of probation, a $5,000 fine, restitution of $361,932, and forfeiture of $50,000.[xxi] It’s unclear how much harm was done to patients by Reuben’s fabrications, but experts believe that implementation of his recommendations may have slowed surgical recovery.[xxii]

Another problem with the available body of evidence is that after research has been discredited, it is not retracted or labeled as fraudulent. The research studies remain available for the unwary to draw false conclusions regarding treatment safety and effectiveness.[xxiii]

The prevailing research models are not appropriate for many alternative therapies, such as acupuncture or homeopathy, in part because treatments are not the same for every patient who presents for treatment of a specific condition. The choice of intervention is based on an assessment of the whole person, not just the presenting symptoms.[xxiv] As a result, studies that give every patient with back pain, for instance, the same intervention are not valid measures of the effectiveness of these therapies.

Another related issue is the averaging of results in a study. What does or does not work for most people with a condition is not necessarily applicable to every individual in a study or in the overall population. Patients are individuals with different characteristics and combinations of symptoms, which might have different causes. Treatment decisions that don’t take individual differences into account are less likely to be effective and can cause harm.[xxv]

Bias in Funding

Another significant problem in evaluating evidence is that there is an uneven playing field when it comes to funding for research. Pharmaceutical companies, which can earn tens of billions of dollars in profit from patenting a single successful drug, can easily afford the hundreds of thousands of dollars it takes to conduct a drug study. Companies that sell foods, vitamins, or herbs that cannot be patented don’t have that kind of money to invest in research that will benefit all companies selling a similar product. Health care providers who provide hands-on treatment—such as chiropractic, massage, or psychotherapy—are even more financially constrained, as they are limited to charging for their services by the hour. They cannot possibly generate enough extra income from their clinical work to fund a large research study.

Adding to the problem is the limited government funding available for medical research, and the fact that this funding rarely goes to those who challenge either the prevailing paradigm or those with power and influence.

What Can Patients Do?

Patients need to recognize that claims of pharmaceutical safety and effectiveness are often overblown and the benefits of alternative therapies are often minimized. Alternative therapies are much safer than pharmaceuticals and, if accessible and affordable, are worth trying first.

My book, The Truth About Chronic Pain Treatments: The Best and Worst Strategies for Becoming Pain Free evaluates the evidence for a broad range of conventional and alternative treatments for chronic pain. My new online Alternative Pain Treatment Directory also provides information and resources for pain patients.

Cindy Perlin is a Licensed Clinical Social Worker, certified biofeedback provider and chronic pain survivor who lives and works in the Albany, NY area.


[i] Sackett D. L., Rosenberg W. M. C., Gray J. A. M., Haynes R. B., & Richardson, W. S. (1996) Evidence based medicine: What it is and what it isn’t. The BMJ, 312, 71-2.

[ii] U.S. Agency for Health Care Policy and Research guidelines

[iii] Every-Palmer, S. & Howick, J. (2014) How evidence-based medicine is failing due to biased trials and selective publication. Journal of Evaluation in Clinical Practice, 20, 908-14.

[iv] Ioannidis, J. P. (2008) Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philosophy, Ethics and Humanities in Medicine, 3, 14.

[v] CBI Research. (2012) Antidepressants market to 2018–Despite safety concerns, selective serotonin re-uptake inhibitors (SSRIs) continue to dominate in the absence of effective therapeutic alternatives. http://www.gbiresearch.com/Report.aspx?ID=Antidepressants-Market-to-2018-Despite-Safety-Concerns

-Selective-Serotonin-Re-uptake-Inhibitors-(SSRIs)-Continue-to-Dominate-in-the-Absence-of-Effective-Therapeutic-Alternatives.

[vi] Kirsch, I., Deacon, B. J., Huedo-Medina, T. B., Scoboria, A., Moore, T. J. & Johnson, B. T. (2008) Initial severity and antidepressant benefits: a meta-analysis of data submitted to the food and drug administration. PLoS Medicine, 5, e45.

[vii] Fournier, J. C., DeRubeis, R. J., Hollon, S. D., Dimidjian, S., Amsterdam, J. D., Shelton, R. C. & Fawcett, J. (2010). Antidepressant drug effects and depression severity: A patient level meta-analysis. Journal of the American Medical Association, 303, 47–53.

[viii] Heres, S., Davis, J., Maino, K., Jetzinger, E., Kissling,W. & Leucht, S. (2006). Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. American Journal of Psychiatry, 163,185–94.

[ix] Smith, R. (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Medicine, 2 (5), e138.

[x] Shellenberger, R. and Green, J. A. (1986). From the Ghost in the Box to Successful Biofeedback Training. Greely, CO: Health Psychology Publications.

[xi] Shellenberger & Green. (1986). From the Ghost in the Box to Successful Biofeedback Training.

[xii] Banderia, M., Bouchard, M., & Granger L. Voluntary control of autonomic responses: A case for a dialogue between individual and group experimental methodolgies. Biofeedback and Self-Regulation, 7, 317-329.

[xiii] Shellenberger & Green. (1986). From the Ghost in the Box, 15-64.

[xiv] Roehr, B. (2012). GlaxoSmithKline is fined record $3 billion in US. British Medical Journal, 345, e4568.

[xv] Lenzer, J. (2006). Manufacturer admits increase in suicidal behaviour in patients taking paroxetine. British Medical Journal, 332(7551), 1175.

[xvi] Department of Justice. (2012) GlaxoSmithKline to plead guilty and pay $3 billion to resolve fraud allegations and failure to report safety data. http://www.justice.gov/opa/pr/2012/July/12-civ-842.html.

[xvii] Kmietowicz, Z. (2012). Johnson & Johnson to pay $2.2 bn to settle charges of false marketing on three drugs. British Medical Journal, 347, f6696.

[xviii] Tanne, J. H. (2012). US judge fines Johnson & Johnson $1.1 bn for misleading marketing of risperidone. British Medical Journal, 344, e2772.

[xix] Tanne. (2012). US judge fines Johnson & Johnson.

[xx] Borrell, B. (2009, March 10). A medical Madoff: Anesthesiologist faked data in 21 studies. Scientific American,

[xxi] Press release, FDA Office of Criminal Investigations/US Department of Justice. Anesthesiologist sentenced on health care fraud charge, June 24, 2010. http://www.fda.gov/ICECI/CriminalInvestigations/ucm217302.htm on 8/10/15.

[xxii] Borrell. (2009). A medical Madoff.

[xxiii] Every-Palmer & Howick. (2014). How evidence-based medicine is failing.

[xxiv] Researching Integrative Medicine: Challenges and Innovations available at http://exploreim.ucla.edu/research/researching-integrative-medicine-challenges-and-innovations/#funding

[xxv] Hartzband, P. & Groopman, J. (2014, November 18). How medical care is being corrupted. New York Times

Subscribe to our blog via email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

 

Authored by: Cindy Perlin

newest oldest
Notify of
D.I.S.May

I am one of the most intelligent, well trained medical professionals I know. I had to research my own symptoms to learn I got PTSD during surgery, consult with three nationally well regarded experts to locate an expert practitioner in my region of the country. All my training did was enable me to learn what happened to me myself. Is Philadelphia a backwater? Apparently. Most of my chronic pain problems are due to iatrogenic causes. Then also, I am a redhead; red hair results from a mutancy affecting about 1% of the population. We do not respond to pain medication like blonds and brunettes do. We need on average twice as much as normal human beings do. Is that in your book, Ms. Expert? Does medicine know this? Do you cover our special needs? I cannot take NSAIDS, I almost died from tylenol, only one pain medication works for my pain. I did 6 months of acupuncture. It did not one thing. I think VERY little of experts. A genuine expert knows that human beings are unique, and must be treated individually. Your double-blind controlled prospective and/or retrospective studies are worthless to a redhead with chronic pain and ulcerative colitis. Worse than useless, they are harmful. They have harmed me. AND I studied drug policy in the United States doing a masters in social policy. Opiates and cocaine were criminalized through acts of treasury. Harrison Act 1914, State of California. No problems with addiction caused this. Rather, Laudanum, a little household remedy with opiates in it was like aspirin today; no more, no less trouble. Treasury was concerned with Asians entering the workforce in California. Smoking opium was associated with Chinese immigrants. Bingo: slam-dunk. Criminalize opioids, get the Chinese out of the workplace. Florida was so impressed they criminalized Cocaine (then in Coca Cola, surely you’ve heard of all the coca cola rehabs shooting up all across the country? Nope?) Cocaine was associated, God knows how, with people of color immigrating to Florida from the Caribbean. So, treasury criminalized cocaine. Another slam-dunk for treasury. And who is the target now? Women; older women and lower socio-economic groups. Only now it is not treasury calling the shots. It’s the corrupt medical system that I have been privy to for 35 years. I know how ineffective 30,000USD in addiction treatment is! Idiots spouting nonsense. I learned at Harvard medical school that breast cancer research was initially conducted only on male study subjects; all those hormone fluctuations in women , well, it was too messy. Made the research too difficult to conduct. So they just studied males. Please don’t think I am impressed with your puffed up self-assessment. You are nobody to me. You are someone who thinks you know my body better than I do, part of a minute fraction of the population being subjected to your generalizable N of 1000. Not an N=1000 redheads! I wish you the same as you’ve given to me. The arrogance is simply unspeakable. Whose payroll… Read more »

Rachel

I have just paid £215 for a small bottle of 30 per cent CBD tincture, way more than the street price of hash, yet I get my opioids free. Something is very wrong here. The UK Government was advised to legalise medical marijuana for pain last year and they were presented with a great deal of what was called sound research. I know marijuana works as all my friends with pain and most doctors do too, but although all my pain doctors advise using it they are telling me to break the law and buy it on the street. Who can afford £215 a month to get it legally when they can’t work anymore? CBD is not illegal it is THC (the high) which is illegal here.

Rachel

In the UK it is reported by the Royal College of General Practitioners that EBM has achieved nothing and saved no money. It was universally introduced in the National Health Service to save money but it has saved no money at all. Pain Doctors here advise their patients to do “whatever works for them”in terms of “alternative” therapies like chiropractic, osteopathy, massage, acupuncture, cranio sacral therapy etc. The Pain Doctors look after the medication and maybe, if you are lucky you will be offered certain proceedures like Lidocaine Infusions with ketamine, or Botox. Lidocaine infusion worked for me the first time but not the second. In both cases I had to go private and pay at about £1400 a time. Most inpatient pain rehab courses include postural education, meditation, relaxation, visualisation, pacing and locus of control management (not doing lots of things at once and prioritising carefully). Some have hydrotherapy and physio as well which is useful if you have had pain for a while and are physically de conditioned. Lifestyle management does help and these courses are worthwhile but there are very few and they take a long time to get onto. I don’t know why doctors and researchers are not taught The History and Philosophy of Science which would show them that the effect of most therapies cannot be measured because every body is different, I read a piece of research on Medscape on low back pain, it they did the trial on the population they always use for trials: fit young men. Well chronic pain patients are mostly female and middle aged or older. At one time they thought that Caucasians with red hair were most prone to chronic pain, I have no idea if that is still the case. I would say be wary of the very fashionable “mindfulness”. According to some research, I have no idea how sound it is, it has a 15 per cent incidence of serious psychiatric side effects yet it is sold by popular culture as harmless.

Notasheep

The evidence using what thousands of doctors and millions of patients say shows narcotics work, period. The same level of evidence you sites for alt therapies applies to narcotics.

Notasheep

Cindy,

I appreciate the difficulties all method face as far as real scientific evidence based on unbiased studies and your effort to educate on those failures. You mentioned using practical reports of effect, some with thousands of years of history. The mind is a wondrous organ, is it not? There in lies the problem with most alternative therapies in a population steeped in hard science. I myself used self taught biotherapy when I was first injured 30 years ago. As a nurse, I have used the healing hands effect on patients and seen it work. There came a point when none of the froofroo stuff worked. My mind could no longer give into and keep the pain to a tolerable level. Thus, 15 years ago I began narcotic therapy.

Most people with intolerable pain have all ready tried all the alts praying they will work. Our conditioning from birth that the alt therapies were froofroo means as a population medicine is the only answer. It is why placebos can work on subset of us. As I said, the mind is a wondrous organ! The problem is, once all the other therapies fail, narcotic relief is all we have left. Now we do not have that.

Another point I would like to make is the abhorrent practice of killing the disabled that still exists in the countries who depend on alt therapy. A small matter that most fail to mention, or do not know about. My own ancestors ‘took the walk’ when they became useless to the tribe. The way I see it, the USA is now forcing 60 million people to ‘take the walk’.

Cindy Perlin has reviewed a real problem which is sometimes called in medical literature “the replication crisis.” For a variety of reasons, it is common practice for “studies” to be biased by their investigators in order to create support for therapies or drugs which don’t work nearly as well in general practice as they are made to appear in the studies.

One of the recent classics in this field of investigation was an attempt by University-based investigators to duplicate the outcomes of 100 widely discussed and referenced studies of social and clinical psychology. The investigators found that the reported outcomes could not be duplicated in more than 60% of the repeated experiments. Among the 40% where results appeared to be similar to those earlier reported, the magnitude of observed effects was significantly lower and less significant than originally reported, for the great majority.

The replication study of psychology does not invalidate that field, despite the negative reactions of many laymen, and a general feeling of “we’ve been had by charlatans.” But it does and should lead many people to exercise a healthy level of skepticism toward claims of universal truth in advertising.

I would suggest, however, that Cindy Perlin jumps to an unsupportable conclusion in her extrapolation of such results to what is generally called “alternative medicine”. Simply because a technique or therapy is represented as “traditional” doesn’t make it useful under modern conditions of urban life. Ayurvedic medicine, for instance, is both traditional and widely practiced in India. However, it has killed quite a number of people due to heavy metal poisoning.

Speaking personally, I would advise any client of medicine to check to see that their pocket hasn’t been picked, when they hear a medical practitioner use the term “holistic”. This terminology has become a variety of hype and misrepresentation in the hands of many “alternative” practitioners.

However, there may be a way through this particular thicket, and it doesn’t involve swearing off evidence based medicine or embracing quackery in hopes that placebo effect will help us. Any study funded by the NIH grant (and that’s a LOT every year) should be required to publish both positive and negative outcomes. Likewise, any NIH study should be subjected to validation in a second replication study by a different and independent team of investigators. Finally, every published study team should be willing to preserve and later make available to independent reviewers, their entire data set, case notes and protocols. If these approaches were used widely, we might likely see at least a reduction in the number of published papers which amount to wishful thinking or outright scientific fraud.

Bill

If something dosent work tell your Doctor and if they won’t changed it move on .U have to try different meds to see what works for u.Plus getting pain meds is a thing of the past.Alot of older people are going to Methadone clinics.But soon that will be a problem.

How do we know what treatment works? As for pain management, I have been under pain management supervision for over 20 years. Surely, the physicians medical journal, reviewed documented,narratives among physicians,and physicians’ experience with an underlying pain issue would be excellent information in regard to how….we know a treatment works. Patient feedback, documentation of the effect of a treatment, which seemingly is being toatlly ignored, would be evidence based upon fact in regard to how……an intractable pain issue should be treated. As far as acute pain, if there is not documented, positive effective treatment by now, approximately how long medicarion “should” be prescribed and the strength,there never will be Individual factors can and should be accounted for each patient. Simply one, effective treatment for acute and chronic pain for every patient without distinguishing personal health and reaction factors among each patient is rediculous. The authority and ability to use it, must be returned to the attending doctor or doctors.

Virginia

All that sounds absolutely like what most of my pain management, physical therapist, chiropractor’s, and physical medicine doctor’s have said just before they got down to business of their way of treatments.
We who are of the opioid-use for pain patients are all aware of all these things you’ve said here. I FOR ONE HATE THEM!!!
I WISH YOU COULD PROVE ME WRONG, THAT THERE IS SOMETHING ELSE WORKS!!!!
All my life, since I was ten yrs old I’ve been searching for a way to be rid of this pain. I’ve tried everything you’ve mentioned, plus many, many others which are not even in Vogue anymore.
I’ve tried every pain medicine too. It’s always come down to morphine. I pray I go up to Jesus every day of my life. I know this pain will all be gone then, and I’ll be home free. 🙏

I can only state what has worked for me after having my meds cut in half by a LYING damn doctor who promised me he would not touch my dosage . I had been on 300mg of morphine/a day for a very long time and doing fine. The reduction in my dose caused my pain level to rise and become uncomfortable. I then started taking CBD OIL and a product called Stem Release. I also started on a low dose regiment of Suboxone to help with the withdrawals. 6 months ago I was on 300mg of morphine a day, now I take none and my pain level is less than it was while on morphine. When I am in pain which is usually very late in the day I smoke weed and that takes care of what pain I have. I’m not trying to sell anything.Maybe this will help someone who has been screwed by [edit]Andrew Kolondy!

Rachel

Cindy Perrin, a good article and an important subject. In my experience UK Pain Consultants will advise a patient to do what works for them in terms of “alternative therapies’. Personally I would be dead now if it weren’t for my Cranio Sacral and Trauma Resolution Counsellor. He diagnosed me accurately with vertebral compression at 3 levels and a bruised dural membrane as well as chronic pain states two years before any doctor did. His knowledge of neurology is phenomenal. He was then invited by the Doctor running my inpatient Pain rehab to take part. Massage is good too if not too firm it keeps one fitter than one would otherwise be if unable to exercise thoroughly. A few years ago osteopaths did a lot for me. I have a degree in the History and Philosophy of Science which taught us to assess the validity of the methodology in any piece of research and it is the best part of my education. I think all doctors should be made to study it: it would improve their judgement of whether a piece of research is valid and help them design better studies.

Sorry but EBM is found to be the best clinical research that drs & researchers hav to offer. Alternative ths r very random in effectiveness. The body is unique in that one size does nor fit all!