Executive Summary
Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines
- COL Lisa K. Moores, MC, USA, FCCP*,
- Ed Dellert, RN, MBA,
- Michael H. Baumann, MD, MS, FCCP and
- Mark J. Rosen, MD, FCCP
- *From the Uniformed Services University of the Health Sciences (COL Moores), Gaithersburg, MD; the American College of Chest Physicians (Mr. Dellert), Northbrook, IL; the University of Mississippi Medical Center (Dr. Baumann), Jackson, MS; and Albert Einstein College of Medicine (Dr. Rosen), New Hyde, NY.
- Correspondence to: COL Lisa K. Moores, MC, USA, FCCP, Uniformed Services University of the Health Sciences, Gaithersburg, MD 20878; e-mail: lmoores{at}usuhs.mil
In 2005, the Education Committee and the Health and Science Policy Committee of the American College of Chest Physicians (ACCP) recognized the need to assess the ACCP medical education curriculum. During this assessment process, a proposal was made to evaluate the literature to determine what continuing medical education (CME) tools and techniques are most effective in improving our physician members' knowledge and skills. It quickly became clear that there was much more to be learned from this effort and that its potential impact could benefit not only the ACCP membership but also the medical education community as a whole. As a result, the ACCP Health and Science Policy and the Education committees developed collaboratively key literature search questions with a proposal that the search be conducted and synthesized by an independent evidence-based practice center (EPC) through an application process to the Agency for Healthcare Research and Quality (AHRQ). AHRQ awarded the Johns Hopkins EPC the task of performing a systematic review of the literature to answer six key questions regarding the effectiveness of CME (see “Methods and Definitions of Terms” article in this supplement for a detailed listing of the questions).
The purpose of these evidence-based guidelines, based on the Johns Hopkins systematic literature review, is to make recommendations to assist ACCP members, the ACCP Education Committee, and any others involved in CME to further the effectiveness of their CME programs. Although the level of evidence was generally of low quality, certain recommendations could be derived substantiating what is currently being conducted by CME providers. In addition, suggestions for further investigation and research were synthesized by the panel. During the Johns Hopkins' literature review, the following primary limitation became obvious: the differences in terminology used in CME activities and in conducting CME research. Examples of terminology variation include the terms used to define the educational interventions applied, the different target audiences, the multiple types of learning objectives, the diverse content areas, and the numerous educational teaching methodologies. This variation has led to a lack of standardized CME approaches and CME research including CME research controls. This clearly makes comparison difficult and quantitative syntheses impossible.
References to this and other limitations found in CME scientific literature are found throughout this document, but they are emphasized most explicitly in “Methods and Definitions of Terms”1 and “The Science of Continuing Medical Education: Terms, Tools, and Gaps.”2 Although not included as a key question, the guideline writing panel noted that a primary limitation in the literature review might be generated by the dearth of formalized training in CME to date. A review of educational skills and knowledge needed by CME providers throughout the United States noted very few studies that indicated or outlined the training needs necessary for CME providers.3 Of those studies that did describe educational training needs, most utilized the CME system used in Canada.4 The Quebec Federation of General Practitioners is an example of an organization attempting to determine the effectiveness of CME providers and to determine whether physician learners have subsequent improvement in educational outcomes.5 In understanding the transition of knowledge from physician instructor to the physician learner, one must consider that the nature of the expertise and approach to patient care may differ between the educator and physician learner, the former being more knowledge based and deductive, with the latter often being more practical and experiential.6 Day-to-day physician medical instructors and CME physician instructors must frame diagnostic and knowledge acquisition strategies of the physician (clinician) learners to maximize the retention and application to the bedside clinical environment.6–8
This guideline document is the result of the efforts of the guideline writing committee members and of the ACCP leadership and staff. This reflects not only the work of the Johns Hopkins EPC systematic review of the literature but also the collaborative endeavors among the ACCP, other major educational organizations, and thought leaders throughout the United States, all of whom had the common goal of improving the effectiveness of CME. The collaborative organizations include the American Medical Association, the Accreditation Council of Continuing Medical Education, the National Board of Medical Examiners, the AHRQ, and the Johns Hopkins EPC. ACCP organizational groups contributing to this document include the Education Committee and the Health and Science Policy Committee. Taking the ACCP evidence-based methodology for guideline development and systematic reviews, which is commonly used for clinical topics, and applying it to a nonclinical area was a learning exercise allowing us to identify areas of medical education that will strengthen approaches to educational events and educational research in the future for the ACCP and the medical education community at large.
Recommendations were made according to ACCP Health and Science Policy Committee grading system, adapted from the Grading of Recommendations Assessment, Development, and Evaluation (or GRADE) system, for the development of all ACCP clinical guidelines.9 For more details about the Johns Hopkins EPC review process and the ACCP grading system, the reader is directed to “Methods and Definitions of Terms.”1 The recommendations from each of the articles are listed below under their respective titles. The specific articles that follow provide an in-depth discussion and clarification of each recommendation.
The reader will note that, when possible, each article includes a general recommendation and then a more specific recommendation concerning educational media, techniques, and exposure. The term media refers to the format used to deliver the educational content (eg, printed articles, videotapes, and audiotapes) and the method of delivery (eg, live sessions vs internet or other technology-based sessions). The term technique refers to the specific educational tools used to deliver the media, such as small group learning, traditional lectures, or simulation. The exposure recommendation attempts to define the duration and frequency (ie, number of sessions) of the CME event.
Responses to key questions and terms used throughout each article may appear to overlap, but educational approaches to respond to these questions are significantly different. Due to time and financial constraints, the decision was made to include only reviews of medical simulation in the original search (Key Question 3). Thus, evidence-based recommendations could not be made in this area. However, the reader is strongly urged to read “Lessons for Continuing Medial Education from Simulation Research in Undergraduate and Graduate Medical Education”10 to better understand the educational potential of medical simulation technology in the field of CME and CME research.
The foundation of the field of CME is reviewed, and suggestions and research recommendations about the focus, terminology, process, and evaluation of future research in the field of CME are made in “The Science of Continuing Medial Education: Terms, Tools, and Gaps.”2 The guideline panel has also gone beyond the evidence-based review and made observations about the future direction of CME, “The Future of Continuing Medial Education.”11
We trust that you as a physician learner and educator will understand the significance that this document brings to the field of CME. There is much you can learn and contribute to the future of CME, and our hope is that you take away key areas of this document and implement them into your educational practice now and contribute to the future CME research that has been identified as a significant need.
Summary of Recommendations
Continuing Medical Education Effect on Physician Knowledge
-
General: We recommend that CME activities be used to improve physician knowledge (Grade 1C).
-
Instructional media: We suggest the use of multimedia CME interventions in preference to single medium interventions to improve physician knowledge (Grade 2C).
-
Instructional techniques: We suggest the use of CME interventions with multiple instructional techniques in preference to a single technique to improve physician knowledge (Grade 2C).
-
Frequency of exposure: We suggest the use of multiple exposures (sessions) to CME content in preference to a single exposure be used to improve physician knowledge (Grade 2C).
Continuing Medical Education Effect on Physician Knowledge Application and Psychomotor Skills
-
General: We recommend that CME activities be used to improve physician application of knowledge (Grade 1C).
-
Frequency of exposure: We suggest multiple CME exposures be used in place of a single exposure to maximize retention and improve physician application of knowledge (Grade 2C).
Continuing Medical Education Effect on Practice Performance
-
General: We recommend that CME interventions be used to improve physician practice performance (Grade 1C).
-
Instructional media
-
We recommend that both single live and multiple media be used to maintain or improve physician practice performance (Grade 1C).
-
We recommend that print media should not be used alone to improve physician practice performance (Grade 1C).
-
-
Instructional techniques: We recommend that multiple instructional techniques be used to improve or maintain physician practice performance (Grade 1C).
-
Frequency of exposure: We suggest that CME activities that include multiple exposures, as opposed to a single exposure, be used to improve physician practice performance (Grade 2C).
Continuing Medical Education Effect on Clinical Outcomes
-
General: We suggest that CME activities be used to improve clinical practice outcomes (Grade 2C).
-
Instructional media: We suggest the use of multiple media compared to single medium interventions to improve clinical outcomes (Grade 2C).
-
Instructional techniques: We suggest the use of multiple instructional techniques compared to single instructional techniques to improve clinical outcomes (Grade 2C).
-
Frequency of exposure: We suggest multiple exposures to content to meet instructional objectives intended to improved clinical outcomes (Grade 2C).
The Role of Audience Characteristics and External Factors in Continuing Medical Education and Physician Change
No recommendations could be made for this topic.
Guideline Panel Suggestions and Research Recommendations
The Science of Continuing Medical Education: Terms, Tools, and Gaps
Suggestions:
-
1a. We suggest that the AMA definition of CME and the terms articulated in this guideline (or their modifications) be consistently employed by CME practitioners and researchers as a basis for the development and study of CME interventions.
-
1b. We suggest widespread dissemination, elaboration and clarification of these terms by journal editors, professional societies, and by the research community.
-
2. We suggest that increased funding be made available for CME research, enabling use of the most rigorous methods in original studies and systematic reviews. We recommend that such funding be carefully determined by the scope and precision of the research question in each case.
-
3. We suggest that searches employ an information specialist and extend beyond the traditional medical educational literature to incorporate databases established to encompass the role of CME in quality improvement, guideline utilization, managed care, business and organizational development, informatics, and other domains.
-
4. We suggest that systematic review processes of CME interventions undertake rigorous efforts to ensure high levels of definitional agreement, independent data abstraction by more than one researcher, and assessment of interrater reliability.
-
5. We suggest that systematic reviews of studies of CME interventions define and employ well-described and commonly agreed-on constructs of what constitutes positive, negative, and mixed outcomes. In this process, careful attention should be paid, where methodologically feasible, to questions of statistical, educational, and clinical significance, and of the magnitude of the effect (eg, effect size, coefficient of determination).
-
6. We suggest standardized definitions, methods, and reporting structures be developed and used for future research, systematic reviews, and guidelines.
-
7. We suggest that researchers explicitly consider the inclusion and documentation of teaching and learning principles in the design and implementation of further trials of CME. In addition, we suggest that, whenever possible, trials be designed to study the educational outcomes of such variables.
-
8a. We suggest that comprehensive models of change such as those developed in knowledge translation be employed when studies of the effect of CME are undertaken, in order to consider and assess the role of unaccounted and dependent variables.
-
8b. We suggest that future studies of CME interventions incorporate full descriptions of elements expressed in the continuing health-care education study template (or CHEST).
-
8c. We suggest that randomized controlled studies (1) be performed with a clear definition of intervention and comparison or control groups, (2) have their effects measured at multiple points after intervention, and (3) pay close attention to issues of participation and dropout.
-
8d. We suggest that researchers consider the value of rigorous observational, ethnographic, and other qualitative study methods, and use them either separately or in conjunction with quantitative methods and designs.
-
9. We suggest that leaders in medical education and related fields (1) foster the identification of high priority research topics in CME research that would span the broad scope of CME and (2) conduct scientifically rigorous studies of the process and effectiveness of CME.
Conflict of Interest Disclosures
Dr. Moores has no conflicts of interest to disclose.
Mr. Dellert is the Vice President of Educational Resources for the ACCP. He has served as faculty for Accreditation Council for Continuing Medical Education, holds a committee position with Alliance for Continuing Medical Education, and is immediate past president of IACME.
Dr. Baumann has no conflicts of interest to disclose on the subject of this article.
Dr. Rosen has no conflicts of interest to disclose.
Footnotes
-
Reproduction of this article is prohibited without written permission from the American College of Chest Physicians (www.chestjournal.org/misc/reprints.shtml).
- Abbreviations::
- ACCP
- American College of Chest Physicians
- AHRQ
- Agency for Healthcare Research and Quality
- CME
- continuing medical education
- EPC
- evidence-based practice center
-
- Received October 20, 2008.
- Accepted December 10, 2008.
