Evaluation of an ECHO Mental Health Pilot Program in Canada

The Extension of Community Health Outcomes (ECHO) constitutes a clinical care and practice improvement initiative intended to provide expertise to primary care providers (PCPs) in rural and underserved areas where access to specialists is limited.

PCPs are first-line responders for individuals experiencing mental health concerns — severe and complex mental health and addiction disorders are often managed within primary care, leaving some PCPs feeling under-resourced and isolated.

Sanjeev Sockalingam, MD, University Health Network, University of Toronto, Canada, delivered a talk on results of an evaluation of a pilot ECHO Mental Health program that was conducted by the Centre for Addiction and Mental Health (CAMH) and the University of Toronto.

“Specialists at CAMH and the University of Toronto have adopted the ECHO model to support PCPs in managing patients with complex mental health needs in primary care settings across Ontario. Project ECHO aims to address disparities in access to specialist care between urban centers and rural communities by building PCP capacity using a hub and spoke tele-education model,” said Sockalingam. In this case, the Hub was an interdisciplinary team with expertise in trauma, medical psychiatry, family medicine, addiction, and social work, including a mental health information specialist.

Spoke sites consisted of organizations that included community health centers, counseling organizations, family health teams, nurse practitioner-led clinics, and solo practitioners. The breakdown by profession of the Spoke participants was 29% social workers, 27% nurse practitioners, 15% community health workers, and 14% physicians. Sockalingam said this study focused on 26 sites with 131 total participants with a mean Hub to Spoke distance of about 400 miles, and added that, “the Spoke retention rates were over 90% with a mean of 34 providers representing 26 sites participating weekly.”

Regarding the structure and format of ECHO, Sockalingam explained there were weekly 2-hour teleconference sessions led by a Hub facilitator who conducted introductions and facilitated discussions throughout the session to encourage Spoke site participation.

There was also a didactic lecture of 20 to 30 minutes that covered curricula designed to promote best practices among the participants and to address previously-identified Spoke learning needs. In addition, the format featured anonymized case presentations of 90 minutes where real cases are brought forward by Spoke sites and are shared with the community of practice for discussion and recommendations.

Sockalingam also described the evaluation of the ECHO program’s performance. “The study’s aim was to evaluate the effectiveness of ECHO on primary care outcomes at CAMH and at University of Toronto. The primary outcomes measured were provider knowledge and self-efficacy. Secondary outcomes were participant satisfaction, participant engagement, and reduction of professional isolation.”

The study was intended to assess both perceived needs (based on the Ontario College of Family Physicians survey, Centre for Effective Practice) and unperceived needs (from population data on mental health and addictions).

Regarding the evaluation framework, Sockalingam said that there were multiple levels:

· Level 1 participation was tracking of the participant

· Level 2 was satisfaction as determined by weekly session questionnaires

· Level 3 measured learning knowledge before and after receiving the curriculum

Measures of higher levels (competence, performance, patient health, and community health are in progress.

Individual Spoke participants (at primary care sites) were included if they had attended at least 8 sessions. T-tests were conducted comparing the participants’ results before and after the 32-week curriculum, and there was a thematic analysis of curriculum comments.

Regarding the evaluations themselves, participant satisfaction was ranked on a 5-point Likert scale. Sockalingam said that weekly satisfaction ratings scored consistently greater than 4. Learning (knowledge) was evaluated by means of multiple-choice questions.

Learning cores improved significantly post-ECHO from about 48% to 60%, while perceived self-efficacy increased post-ECHO from about 50% to 60%, (but not significantly). Just over half (27 of 48) of Spoke participants completed questionnaires.

Summarizing, Sockalingam told the audience, “Our evaluation shows that an ECHO in mental health and addictions is feasible and acceptable, with high provider retention rates. Providers expressed satisfaction with the program and showed improvements in their knowledge and trends for improved self efficacy were observed.”