A Simple Method for Evaluating the Clinical Literature

The “PP-ICONS” approach will help you separate the clinical wheat from the chaff in mere minutes.

Robert J. Flaherty, MD

Keeping up with the latest advances in diagnosis and treatment is a challenge we all face as phycians. We need information that is both valid (that is, accurate and correct) and relevant to our patients and practices. While we have many sources of clinical information, such as CME lectures, textbooks, pharmaceutical advertising, pharmaceutical representatives and colleagues, we often turn to journal articles for the most current clinical information.

Unfortunately, a great deal of research reported in journal articles is poorly done, poorly analyzed or both, and thus is not valid. A great deal of research is also irrelevant to our patients and practices. Separating the clinical wheat from the chaff can take skills that many of us never were taught.

KEY POINTS

• Reading the abstract is often sufficient when evaluating an article using the PP-ICONS approach.
• The most relevant studies will involve outcomes that matter to patients (e.g., morbidity, mortality and cost) versus outcomes that matter to physiologists (e.g., blood pressure, blood sugar or cholesterol levels).
• Ignore the relative risk reduction, as it overstates research findings and will mislead you.

The article “Making Evidence-Based Medicine Doable in Everyday Practice” in the February 2004 issue of FPM describes several organizations that can help us. These organizations, such as the Cochrane Library, Bandolier and Clinical Evidence, develop clinical questions and then review one or more journal articles to identify the best available evidence that answers the question, with a focus on the quality of the study, the validity of the results and the relevance of the findings to everyday practice. These organizations provide a very valuable service, and the number of important clinical questions that they have studied has grown steadily over the past five years. (See “Four steps to an evidence-based answer.”)

FOUR STEPS TO AN EVIDENCE-BASED ANSWER

When faced with a clinical question, follow these steps to find an evidence-based answer:

1. Search the Web site of one of the evidence review organizations, such as Cochrane (http://www.cochrane.org/cochrane/revabstr/mainindex.htm), Bandolier (http://www.jr2.ox.ac.uk/bandolier) or Clinical Evidence (http://www.clinicalevidence.com), described in “Making Evidence-Based Medicine Doable in Everyday Practice,” FPM, February 2004, page 51. You can also search the TRIP+ Web site (http://www.tripdatabase.com), which simultaneously searches the databases of many of the review organizations. If you find a systematic review or meta-analysis by one of these organizations, you can be confident that you’ve found the best evidence available.
2. If you don’t find the information you need through step 1, search for meta-analyses and systematic reviews using the PubMed Web site (see the tutorial athttp://www.nlm.nih.gov/bsd/pubmed_tutorial/m1001.html). Most of the recent abstracts found on PubMed provide enough information for you to determine the validity and relevance of the findings. If needed, you can get a copy of the full article through your hospital library or the journal’s Web site.
3. If you cannot find a systematic review or meta-analysis on PubMed, look for a randomized controlled trial (RCT). The RCT is the “gold standard” in medical research. Case reports, cohort studies and other research methods simply are not good enough to use for making patient care decisions.
4. Once you find the article you need, use the PP-ICONS approach to evaluate its usefulness to your patient.

If you find a systematic review or meta-analysis done by one of these organizations, you can feel confident that you have found the current best evidence. However, these organizations have not asked all of the common clinical questions yet, and you will frequently be faced with finding the pertinent articles and determining for yourself whether they are valuable. This is where the PP-ICONS approach can help.

What is PP-ICONS?

When you find a systematic review, meta-analysis or randomized controlled trial while reading your clinical journals or searching PubMed (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi), you need to determine whether it is valid and relevant. There are many different ways to analyze an abstract or journal article, some more rigorous than others.1,2 I have found a simple but effective way to identify a valid or relevant article within a couple of minutes, ensuring that I can use or discard the conclusions with confidence. This approach works well on articles regarding treatment and prevention, and can also be used with articles on diagnosis and screening.

The most important information to look for when reviewing an article can be summarized by the acronym “PP-ICONS,” which stands for the following:

• Problem,
• Patient or population,
• Intervention,
• Comparison,
• Outcome,
• Number of subjects,
• Statistics.

For example, imagine that you just saw a nine-year-old patient in the office with common warts on her hands, an ideal candidate for your usual cryotherapy. Her mother had heard about treating warts with duct tape and wondered if you would recommend this treatment. You promised to call Mom back after you had a chance to investigate this rather odd treatment.

When you get a free moment, you write down your clinical question: “Is duct tape an effective treatment for warts in children?” Writing down your clinical question is useful, as it can help you clarify exactly what you are looking for. Use the PPICO parts of the acronym to help you write your clinical question; this is actually how many researchers develop their research questions.

You search Cochrane and Bandolier without success, so now you search PubMed, which returns an abstract for the following article: “Focht DR 3rd, Spicer C, Fairchok MP. The efficacy of duct tape vs cryotherapy in the treatment of verruca vulgaris (the common wart). Arch Pediatr Adolesc Med. 2002 Oct;156(10):971-974.”

You decide to apply PP-ICONS to this abstract (see “Abstract from PubMed”) to determine if the information is both valid and relevant.

ABSTRACT FROM PUBMED

Using the PP-ICONS approach, physicians can evaluate the validity and relevance of clinical articles in minutes using only the abstract, such as this one, obtained free online from PubMed,http://www.ncbi. nlm.nih.gov/entrez/query.fcgi. The author uses this abstract to evaluate the use of duct tape to treat common warts.

Problem. The first P in PP-ICONS is for “problem,” which refers to the clinical condition that was studied. From the abstract, it is clear that the researchers studied the same problem you are interested in, which is important since flat warts or genital warts may have responded differently. Obviously, if the problem studied were not sufficiently similar to your clinical problem, the results would not be relevant.

Patient or population. Next, consider the patient or population. Is the study group similar to your patient or practice? Are they primary care patients, for example, or are they patients who have been referred to a tertiary care center? Are they of a similar age and gender? In this case, the researchers studied children and young adults in outpatient clinics, which is similar to your patient population. If the patients in the study are not similar to your patient, for example if they are sicker, older, a different gender or more clinically complicated, the results might not be relevant.

Intervention. The intervention could be a diagnostic test or a treatment. Make sure the intervention is the same as what you are looking for. The patient’s mother was asking about duct tape for warts, so this is a relevant study.

Comparison. The comparison is what the intervention is tested against. It could be a different diagnostic test or another therapy, such as cryotherapy in this wart study. It could even be placebo or no treatment. Make sure the comparison fits your question. You usually use cryotherapy for common warts, so this is a relevant comparison.

Outcome. The outcome is particularly important. Many outcomes are “disease-oriented outcomes,” which are based on “disease-oriented evidence” (DOEs). DOEs usually reflect changes in physiologic parameters, such as blood pressure, blood sugar, cholesterol, etc. We have long assumed that improving the physiologic parameters of a disease will result in a better disease outcome, but that is not necessarily true. For instance, finasteride can improve urinary flow rate in prostatic hypertrophy, but it does not significantly change symptom scores.3

DOEs look at the kinds of outcomes that physiologists care about. More relevant are outcomes that patients care about, often called “patient-oriented outcomes.” These are based on “patient-oriented evidence that matters” (POEMs) and look at outcomes such as morbidity, mortality and cost. Thus, when looking at a journal article, DOEs are interesting but of questionable relevance, whereas POEMs are very interesting and very relevant. In the study on the previous page, the outcome is complete resolution of the wart, which is something your patient is interested in.

Number. The number of subjects is crucial to whether accurate statistics can be generated from the data. Too few patients in a research study may not be enough to show that a difference actually exists between the intervention and comparison groups (known as the “power” of a study). Many studies are published with less than 100 subjects, which is usually inadequate to provide reliable statistics. A good rule of thumb is 400 subjects.4 Fifty-one patients completed the wart study, which is a pretty small number to generate good statistics.

Statistics. The statistics you are interested in are few in number and easy to understand. Since statistics are frequently misused in journal articles, it is worth a few minutes to learn which to believe and which to ignore.

Relative risk reduction. It is not unusual to find a summary statement in a journal article similar to this one from an article titled “Long-Term Effects of Mammography Screening: Updated Overview of the Swedish Randomised Trials”

“There were 511 breast cancer deaths in 1,864,770 women-years in the invited groups and 584 breast cancer deaths in 1,688,440 women-years in the control groups, a significant 21 percent reduction in breast cancer mortality.”

This 21-percent statistic is the relative risk reduction (RRR), which is the percent reduction in the measured outcome between the experimental and control groups. (See “Some important statistics” for more information on calculating the RRR and other statistics.) The RRR is not a good way to compare outcomes. It amplifies small differences and makes insignificant findings appear significant, and it doesn’t reflect the baseline risk of the outcome event. Nevertheless, the RRR is very popular and will be reported in nearly every journal article, perhaps because it makes weak results look good. Think of the RRR as the “reputation reviving ratio” or the “reporter’s reason for ‘riting.” Ignore the RRR. It will mislead you. In our wart treatment example, the RRR would be (85 percent – 60 percent)/60 percent x 100 = 42 percent. The RRR could thus be interpreted as showing that duct tape is 42 percent more effective than cryotherapy in treating warts.

SOME IMPORTANT STATISTICS

Absolute risk reduction (ARR): The difference between the control group’s event rate (CER) and the experimental group’s event rate (EER).

Control event rate (CER): The proportion of patients responding to placebo or other control treatment. For example, if 25 patients are in a control group and the event being studied is observed in 15 of those patients, the control event rate would be 15/25 = 0.60.

Experimental event rate (EER): The proportion of patients responding to the experimental treatment or intervention. For example, if 26 patients are in an experimental group and the event being studied is observed in 22 of those patients, the experimental event rate would be 22/26 = 0.85.

Number needed to treat (NNT): The number of patients that must be treated to prevent one adverse outcome or for one patient to benefit. The NNT is the inverse of the ARR; NNT = 1/ARR.

Relative risk reduction (RRR): The percent reduction in events in the treated group compared to the control group event rate.

View/Print Table

	When the experimental treatment reduces the risk of a bad event:	Example: Beta-blockers to prevent deaths in high-risk patients with recent myocardial infarction:	When the experimental treatment increases the probability of a good event:	Example: Duct tape to eliminate common warts.
Relative risk reduction (RRR)	CER-EER/CER	(.66 -. 50)/.66 = .24 or 24 percent	EER-CER/CER	(.85-.60)/.60 = .42 or 42 percent
Absolute risk reduction (ARR):	CER-EER	(.66 – .50) = .16 or 16 percent	EER-CER	.85-.60 = .25 or 25 percent
Number needed to treat (NNT)	1/ARR	1/.16 = 6	1/ARR	1/.25 = 4

Absolute risk reduction. A better statistic is the absolute risk reduction (ARR), which is the difference in the outcome event rate between the control group and the experimental treated group. Thus, in our wart treatment example, the ARR is the outcome event rate (complete resolution of warts) for duct tape (85 percent) minus the outcome event rate for cryotherapy (60 percent) = 25 percent. Unlike the RRR, the ARR does not amplify small differences but shows the true difference between the experimental and control interventions. Using the ARR, it would be accurate to say that duct tape is 25-percent more effective than cryotherapy in treating warts.

Number needed to treat. The single most clinically useful statistic is the number needed to treat (NNT). The NNT is the number of patients who must be treated to prevent one adverse outcome. To think about it another way, the NNT is the number of patients who must be treated for one patient to benefit. (The rest who were treated obtained no benefit, although they still suffered the risks and costs of treatment.) In our wart therapy article, the NNT would tell us how many patients must be treated with the experimental treatment for one to benefit more than if he or she had been treated with the standard treatment.

Now this is a statistic that physicians and their patients can really appreciate! Furthermore, the NNT is easy to calculate, as it is simply the inverse of the ARR. For our wart treatment study, the NNT is 1/25 percent =1/0.25 = 4, meaning that 4 patients need to be treated with duct tape for one to benefit more than if treated by cryotherapy.

Wrapped up in this simple little statistic are some very important concepts. The NNT provides you with the likelihood that the test or treatment will benefit any individual patient, an impression of the baseline risk of the adverse event, and a sense of the cost to society. Thus, it gives perspective and hints at the “reasonableness” of a treatment. The value of this statistic has become appreciated in the last five years, and more journal articles are reporting it.

What is a reasonable NNT? In a perfect world, a treatment would have an NNT of 1, meaning that every patient would benefit from the treatment. Real life is not so kind (see “Examples of NNTs”). Clearly, an NNT of 1 is great and an NNT of 1,000 is terrible. Although it is hard to come up with firm guidelines, for primary therapies I am satisfied with an NNT of 10 or less and very pleased with an NNT less than 5. Our duct tape NNT of 4 is good, particularly since the treatment is cheap, easy and painless.

EXAMPLES OF NNTS

The number needed to treat (NNT) is one of the most useful statistics for physicians and patients. It calculates the number of patients that must be treated to prevent one adverse event or for one patient to benefit. Note that NNTs for preventive interventions will usually be higher than NNTs for treatment interventions. The lower the NNT, the better.

The following examples of NNTs are borrowed from an excellent list available through the Bandolier Web site at http://www.jr2.ox.ac.uk/bandolier/band50/b50-8.html.

View/Print Table

Therapy	NNT
Triple antibiotic therapy to eradicate H. pylori	1.1
Isosorbide dinitrate for prevention of exercise-induced angina	5
Short course of antibiotics for otitis media in children	7
Statins for secondary prevention of adverse cardiovascular outcomes	11
Statins for primary prevention of adverse cardiovascular outcomes	35
Finasteride to prevent one operation for benign prostatic hyperplasia	39
Misoprostol to prevent any gastrointestinal complication in nonsteroidal anti-inflammatory drug users	166

Note that NNTs for preventive interventions (e.g., the use of aspirin to prevent cardiac problems) will usually be higher than NNTs for treatment interventions (e.g., the use of duct tape to cure warts). Prevention groups contain both higher-risk and lower-risk individuals, so they produce bigger denominators, whereas treatment groups only contain diseased patients. Thus, an NNT for prevention of less than 20 might be particularly good.

When discussing a particular therapy, I explain the NNT to my patient. Since this statistical concept is easy to understand, it can help the patient be a more informed partner in therapeutic decisions.

You will soon start to see a similar statistic, the number needed to screen (NNS), which is the number of patients needed to screen for a particular disease for a given duration for one patient to benefit.6Although few NNSs have been calculated, they are likely to involve higher numbers, since the screening population consists of patients with and without the disease. For example, in the article on mammography screening mentioned above, the NNS was 961 for 16 years. In other words, you would need to screen 961 women for 16 years to prevent one breast cancer death.

The good news and the bad

Using PP-ICONS to assess the wart study, the problem, the patient/population, the intervention, the comparison and the outcome are all relevant to your patient. The number of subjects is on the small side, making you a little wary, but the intervention is cheap and low-risk. The statistics, particularly the NNT, are reasonable. On balance, this looks like a fair approach, so you call the patient’s mother and discuss it with her.

The PP-ICONS approach is an easy way to screen an article for validity and relevance, and the abstract often contains all of the information you need. Even the statistics can be done quickly in your head. You can apply PP-ICONS when searching for a particular article, when you come across an article in your reading, when data are presented at lectures, when a pharmaceutical representative hands you an article to support his or her pitch, and even when reading news stories describing medical breakthroughs.

Don’t be discouraged if you find that high-quality articles are rare, even in the most prestigious journals. This seems to be changing for the better, although many careers are still being built on questionable research. Nevertheless, screening articles will help you find the truth that is out there and will help you practice the best medicine. And as we become more discerning end-users of research, we might just stimulate improvements in clinical research in the process.

---

Dr. Flaherty is a family physician with the Student Health Service and WWAMI Medical Program, Montana State University-Bozeman, and a clinical professor in the Department of Family Medicine, University of Washington.

Read More

Empowering patients with genetic testing

More and more women like actress Angelina Jolie are stepping forward to undergo genetic testing for breast cancer – and for good reason.

As many as 5 percent to 10 percent of all cancers may be linked to an inherited risk, and approximately 30 percent of cancer cases occur in families with close relatives who have experienced similar cancers. Women with BRCA1 or BRCA2 gene mutations may have up to an 80 percent risk of being diagnosed with breast cancer during their lifetimes. Many other gene mutations have also been linked to hereditary cancer.

Hereditary cancer is the development of cancer due to an inherited gene mutation that has been passed from parent to child upon conception. Experts in the field of cancer genetics can help identify an individual’s risk of developing cancer and understand the options available to help manage related health issues, explained David Euhus, M.D., Professor of Surgery at UT Southwestern Medical Center and surgical oncologist at UT Southwestern’s NCI-designatedHarold C. Simmons Comprehensive Cancer Center.

“When we understand the causes of this disease and its triggers, we can empower patients to fight cancer,” said Dr. Euhus, co-director of the Mary L. Brown Breast Cancer Genetics and Risk Assessment Clinic in the Center for Breast Care at UT Southwestern.

Dr. Euhus helped develop the groundbreaking CancerGene software program that is used in more than 4,000 cancer centers throughout the world to help doctors work with patients to identify their inherited risk for breast cancer.

“Knowing that you carry a mutation in one of the BRCA genes opens up numerous options for early diagnosis and cancer prevention,” he said. “If you have been diagnosed with breast cancer recently, this information is critical for making treatment decisions.”

Clinical trials for patients with a BRCA gene mutation are available at UT Southwestern.

UT Southwestern’s genetic counselors have also partnered with FORCE, or Facing our Risk of Cancer Empowered, a support group for patients with hereditary breast cancer. This sentiment of cancer prevention is echoed through families facing hereditary breast and ovarian cancer.

One woman who recently discovered that she has a BRCA1 mutation spoke about her plans to undergo prophylactic bilateral mastectomy and removal of her ovaries. “At the end of the day, I want to do everything I can to make sure that I am here to watch my children grow up,” she said.

Dr. Euhus offered the following checklist to help individuals decide whether they should consider genetic testing for breast cancer:

• Breast cancer diagnosis before age 45
• Three or more blood relatives on the same side of the family diagnosed with breast cancer
• A family history of ovarian cancer
• A relative on either side of the family diagnosed with breast cancer before age 45
• Being of Ashkenazi Jewish heritage
• A family member who had triple negative breast cancer before age 60
• Any male in the family who has had breast cancer
• Female relatives who have had cancer in both breasts

An inherited risk for cancer also may be higher when:

• The same type of cancer occurs in multiple close relatives.
• You or a family member have a rare cancer/tumor such as a sarcoma, male breast cancer, medullary thyroid cancer, or a pheochromocytoma.
• You have a family history with a combination of breast and ovarian cancers, or colon and uterine cancers.
• You have more than 20 colon polyps.
• You or a family member have had multiple different kinds of cancer.

Read More

tips for doing research with schools, charities and NGOs

Conducting research on children, young people and learning often requires access to and help from schools, charities or NGOs. Alicia Blum-Ross draws on both struggles and successes from previous projects with learning institutions and presents five key strategies to build meaningful and mutually beneficial relationships.

You are a busy researcher, interested in children, young people and learning. They are a (probably even busier) NGO, charity or school working with children and young people (or vice versa). Possibly an ideal match; possibly a source of stress for both.

Partnering with schools, charities, NGOs and community organisations can make or break a research project. It may be how we observe learning practices, meet young people, or learn from facilitators, teachers and organisers. But often these potential partners are at best too busy to answer emails, or at worst sceptical or downright suspicious (sometimes rightly) about academic research.

In our current and previous projects we negotiated great (and sometimes not so great) research relationships with schools, charities and NGOs. These are our top tips for successful relationships

1. Research with

Researchers increasingly emphasise the importance of working with ‘ethical symmetry’ to do researchwith their participants – including organisations – instead of research on them.

Think of these relationships as partnerships, where sometimes your research goals may be called into question. This isn’t a hurdle to be overcome, but something that can enhance your research.

2. So you think you’re busy?

Bouncing from teaching to research to committees, you may have a lot going on. But try working in a (sometimes severely) under-resourced school, arts or learning organisation, or volunteering for a community group. Now that’s busy.

Make things as easy as you can – limit the emails you send, keep them to the point, make sure any action items / deadlines are clear, and get all your research materials (consent letters, project descriptions, any ethics approvals or police checks you might need) in order before you contact anyone.

Credit: J. Baxter, CC BY-NC-SA 2.0

3. Make it worth their while

In your memoranda of understanding (MoU), spell out what they can expect from you. Can you collect evidence (quotes, photos etc.) that they could use in an evaluation report or a future funding bid? How might your research help them answer their own questions about impact? Will you be collecting any data they can use as evidence of meeting key performance indicators (KPIs)?

There is a huge burden on organisations to ‘measure outcomes’, so your partners will love it if you can help them find evidence to do this, but be careful that you aren’t compromising the point of your research or confidentiality.

You can also be helpful in smaller ways – do they need more chairs for the meeting you’re sitting in on? Do they need an extra pair of hands to serve food? Helping out builds goodwill and also gives you opportunities to get to know both staff and participants.

4. Put it in writing

Okay, we said above to keep it simple. But also make sure that everyone is on the same page about what you need / want from each other. We suggest writing a document (sometimes called an MoU) that clarifies the following:

• The aims of your research. What are you trying to find out and why? Highlight why this would be interesting to them and how they might benefit from it.

• Your source of funding.

• Your research methods. What are you going to do, and where? Will you be asking to use an empty classroom for interviews? Will you be audio-recording or taking notes? How will you inform participants of the research? How will you get consent? Are there any research activities (like participant-observation) that may impact on their programmes? Also note your policy on anonymity – will you anonymise the school/organisation or just the names of the children/young people? What about in photos or video, if you are using them?

• Relationship with the organisation. Is the organisation recruiting participants on your behalf? Do you have a specific named contact in the organisation? How will you notify them if there are any problems that are raised in your research (e.g. evidence of criminal activity, feedback on the programmes)? Clarify what you would be expected to tell them and what you would keep confidential.

• Who will see this research? To whom and how will you be disseminating your research? Academic articles or in something more public-facing? Highlight especially if you are trying to create public impact or influence policy.

5. Don’t disappear

You’ve done your fieldwork, you’re back writing up the research, and the school or community organisation seems pretty far away (although not that far away if you’re still transcribing or coding interviews!). No matter how distant you may start to feel from the research, don’t disappear – find ways of involving the site in disseminating the research, or at least in giving them access to it. They may not take you up on it, but it’s good ethical practice to at least try.

Yes, this list does place a burden on you as a researcher, and adds a few more hoops to jump through. But in our experience, making sure that you honour the time you’re given, clarify expectations, and communicate about the process and the findings is not only good for building a relationship with a partner organisation, but it will also enhance the research itself.

Read More