For example, the Dannemiller Memorial Education Foundation (DMEF) includes an outcomes study for all of its activities. This study is directly tied into DMEF's certificate system, and is an e-mail and Internet-based process. Approximately 30-45 days after a clinician has completed a CME activity, he receives a survey designed to assess the participant's incorporation of the CME-activity information into his daily practice. As of January 2005, DMEF had sent more than 13,000 surveys, with a return rate of over 22 percent. An analysis of these surveys indicated:

• 86 percent of respondents had learned something new from completing the activity
• 71 percent of participants noted that they incorporated the new knowledge into their practices
• 86 percent reported having learned everything they expected to from the activity.

Medscape, a CME provider and leading distributor of online professional education, has developed Medscape CME Metrics. This is a quantitative evaluation designed to determine the effectiveness of educational material presented online.

The assessment measures retention of course content, adherence to clinical practice patterns, attitudes toward therapeutic management, and barriers to optimal clinical management. The survey instrument that is utilized consists of a series of case studies, which are then followed by questions designed to assess the clinician's motivation, confidence, knowledge, and barriers to treatment.

Grantors are taking steps to ensure that effectiveness measurement is an integral part of the CME learning process. Companies that fund CME endeavors are looking to include language in their requests for proposals and letters of agreement that clearly delineate the inclusion of effectiveness outcomes. For example, the Novartis Office of Grants and Education, in its grant-request submission process, states that one of the documents that need to be included for consideration for funding is an outcomes measurement plan.

Medical-education communication companies are also starting to include effectiveness and outcome measures into their programs. One very basic approach that is being utilized is the check survey for measuring readership of enduring materials. This process consists of the inclusion of a monetary check (for a token amount that can be donated to a charitable organization) in the body of a publication; this check is randomly inserted in a subsample of the publications mailed. SynerMed has conducted more than 40 check surveys (encompassing 95,000 clinicians) assessing a variety of physician/health provider specialties. The median response rate is approximately 20 percent (with a range of seven percent to 37 percent); this compares with a two-to-three percent response rate for other direct mail pieces.

Guiding Light

Although steps are being taken to improve outcomes measurement in CME activities, a clear definition of what specific measures need to be assessed—as well as guidance on the means and techniques for achieving those measures—is still lacking.

Much of medicine today is influenced by guidelines. Maybe it is time for those organizations most concerned with the provision of CME services and programs to take leadership roles in the development of the CME Program Outcome Assessment Guidelines.

The biggest challenge in gauging the effectiveness of CME activities is identifying and developing the means that will allow for the efficient incorporation of these measurement tools into the educational process. Demonstrating the effectiveness of one's services is necessary to ensure that the educational needs of healthcare clinicians are met, thereby facilitating their provision of services to their patients. In the end, it is the total healthcare system that will be the winner.

Robert Schneider is VP of professional relations at SynerMed. He can be reached at rschneider@ascendmedia.com