Key Considerations for Creating Evaluation Metrics for CHW Programs

Importance of Evaluating
Tracking evaluation metrics as part of the planning and implementation of a program can demonstrate the value of including a community health worker (CHW) as part of the intervention by monitoring successes and ensuring quality of program delivery. Process and outcome evaluation may also improve the integration of the CHW into a program and support the CHW in their work through ongoing quality improvement. On a broader level, incorporating evaluation metrics within a CHW program demonstrates the importance of CHWs in program implementation and underscores the unique expertise of the CHW in an organization. This document will provide a high-level overview of evaluation opportunities to consider for your program, as well as key take-aways including strategies and underlying evaluation principles that apply regardless of the level of evaluation effort you are undertaking. We will provide an overview of four levels of evaluation, which have been adapted from the Arizona Prevention Research Center CHW Evaluation Toolkit: 1) client-level impact, 2) CHW-level impact, 3) organization or program performance, 4) community-level change.

Key Principles for Evaluating CHW Programs
The Arizona Prevention Research Center developed a toolkit that describes key building blocks for evaluating CHW programs.^[i]  The content of the toolkit was developed in response to a need for practical methods to measure and evaluate CHW programs. These building blocks for evaluation include:

• Plan your program and evaluation simultaneously. Evaluation is a fundamental part of planning the program, not an afterthought.
• Ensure collaboration by including CHWs and staff, clients, funders, and other stakeholders in planning from the beginning of evaluation design.
• Make CHWs a focus of your evaluation. Keep the following questions in mind when designing an evaluation: How have CHWs contributed to the success of our program? What have CHWs done that no one else can do?
• Incorporate evaluation as part of CHW training. This step will help ensure that they are an effective part of the collaborative evaluation efforts.
• Keep your evaluation efforts simple. Answer the following questions when designing an evaluation: What are you trying to change? How are you documenting these changes? How can CHWs play a role in these changes?

Four Levels of Evaluation
The Arizona Preventions Research Center also outlines four levels on which evaluation can be conducted, which we discuss below.

Impact of CHWs on Individual Clients
Evaluating CHW’s impact on individuals can include both process and outcome measures. Examples of process measures include mode of interaction with clients, the specific number, type, and frequency of services a CHW provides, client satisfaction with CHW service, a client’s improved knowledge, beliefs, and behaviors, and a client’s improved health status (e.g., reduced blood pressure). Tracking individual-level interactions can be done in a variety of ways but is commonly completed through check-lists where a CHW documents their specific interactions with clients. In addition, supervisors may elect to accompany a CHW on a home visit or observe a CHW using role-play techniques. Regardless of the method of assessing a CHW-client interaction, evaluating the impact of a CHW on individual clients allows for the incorporation of fidelity monitoring to ensure that the intervention or program is being delivered as it was intended. It also allows for corrections and quality control throughout the life course of the program. If there are concerns or issues with the CHW-client interaction, the organization should identify the area that needs support and help improve the activity through additional training, supervision, and support.

Assessment of CHW Proficiencies in their Roles and Experiences at the Organization
Assessing CHW proficiencies in their roles and experience at an organization can help to regularly augment and improve supervision, training, and CHW experiences. These types of process-focused metrics can help ensure that CHWs are working at the top of their skillset. Self-reported metrics can be reviewed with a supervisor on a regular basis. The sample qualitative questions below could be modified to be Likert scale (needs improvement to excellent) if a qualitative approach is preferred. Sample questions for a CHW self-evaluation may include: length of time worked as a CHW, key tasks, initial training received, ongoing training (and additional elements needed for training), support for role (e.g., supplies and equipment necessary to do work), reflection on experience with supervision, opportunities for promotion or professional development, challenges as a CHW in current position, and what needs to change to improve job experience.

In addition to the CHW self-evaluation, supervisors may observe specific aspects of the CHWs work, which could align with the CHW self-evaluation listed above. This supervisor evaluation of a CHW may include additional elements such as assessing the CHW’s punctuality, level of knowledge about specific topics, and/or how well they perform in specific roles (e.g., providing health education).

Similar to evaluating the client-level impact of CHWs, regular evaluation at the CHW level can allow supervisors to identify areas where CHWs may need additional support, training, or supervision to help ensure that the program is being delivered with fidelity and that a the CHW is working at the top of their skillset and getting all of the resources they need.

Program-Level Evaluation
In addition to understanding CHW-level evaluation indicators, programs should consider indicators of success at the program-level. The University of Pennsylvania CHW Program offers a CHW Program Functionality Matrix, which assesses program-level indicators including recruitment, CHW role, initial training provided to CHW, continuous training, equipment and supplies availability, CHW supervision, individual performance assessment, incentives, CHW community involvement, referral system, opportunities for advancement, documentation and information management, and linkages to health systems. These internal program-level indicators of success are scored on a scale of 1-4 (non-functional to highly functional).^[ii]

Other program-level evaluation outcomes may include the number of clients that are part of the program, number of appointments kept, successful referrals, client satisfaction with the program, individual health status, improved access to services, and cost and benefit. These program-level indicators are specific to the intervention itself and can be evaluated on pre-determined timeframes.

Community Evaluation
Depending on the scope of a program or intervention, community-level evaluation can help to better understand the impact of an intervention on the community. Examples of these outcomes and impacts a program may have at the community level include: capacity building among partner organizations, improved access to community resources, partnerships, community mobilization, policy development across organizations, and improved range of health services available to the community.

Key Take-Aways
CHWs are vital members of health care teams, serving as a bridge between the community and health systems. Although CHWs are recognized as key members of care teams, it is important to evaluate their effectiveness, particularly in the context of a specific program. Linking specific CHW activities to program outcomes can help provide practice-based evidence for a CHW program. Below is a summary of the CDC’s six-step evaluation framework adapted to CHW programs that can help plan evaluation regardless of level described above.^[iii]

^[i]University of Arizona Rural Health Office and College of Public Health. The community health worker evaluation tool kit.
^[ii]Penn Center for Community Health Workers. IMPaCT Toolkit.
^[iii]Mirambeau, A. M. (2012). CDC Evaluation coffee break: Evaluating community health worker programs.