report

Attachment 3:

OWH WHLI Program Evaluation Survey Pilot Report

OWH WHLI Program Evaluation Survey Pilot Report

The Office on Women’s Health (OWH) Women’s Health Leadership Institute (WHLI) Program Evaluation Survey Pilot was conducted by NORC at the University of Chicago. The purpose of the pilot was to obtain feedback on a survey NORC developed to measure the intermediate and long-term impacts and outcomes of the WHLI training program.

NORC was responsible for all data collection activities, including contacting potential participants, providing surveys, and conducting debriefings. Fifteen Community Health Workers (CHWs) who participated in the WHLI training program, and for whom we have an email address on file, were randomly selected to take part in the pilot. The CHWs were invited by email with telephone follow-up, and the first three who agreed to participate were selected. Since participants in the pilot study will not take part in the main study, additional respondents were declined due to the small sample size of CHWs who participated in the WHLI training. They were, however, encouraged to participate in the main study. Three NORC staff members were also invited to participate in the pilot study to offer feedback on the design of the survey.

The survey for the main study will be web-based; for the pilot study, a paper-and-pencil format was used. A Microsoft Word version of the survey was emailed to participants, and they were asked to print it out and complete it. Once the survey was completed, they were invited to participate in a telephone or in-person debriefing with two members of the NORC evaluation team. Two CHWs and three NORC staff members completed the WHLI survey and the debriefing.

The completion time for the survey is displayed below in Table 1. The average amount of time it took participants to complete the survey is 26 minutes It is important to note that CHW #1 completed the entire survey without following the skip patterns appropriately, resulting in the extra time spent on the survey. In the main study, the web-based format will preclude these issues from occurring.

Table 1. Completion time for the OWH WHLI Survey

Participant	WHLI Survey Administration Time
CHW #1	40 minutes
CHW #2	N/A*
Staff Member #1	20 minutes
Staff Member #2	25 minutes
Staff Member #3	20 minutes

* CHW #2 returned to the survey over several hours instead of completing it in one sitting and could not recall the collective time it took to finish the survey.

Once participants completed the survey, they were scheduled to take part in a telephone debriefing (CHWs) or in-person debriefing (NORC staff). The goal of the debriefing was to discuss:

• Questions or instructions that were unclear or confusing,

• Terminology that needed to be better defined, and

• Response options that were unclear, confusing, overlapping, or missing.

Feedback from the pilot study participants (included as Appendix 1 to this summary) indicated the survey was comprehensive and that the instructions in the survey were generally easy to follow. Some respondents indicated the survey felt too long, although the time to complete the survey met our expectation of approximately 25 minutes. Because of the paper-and-pencil format, the survey appears longer than it will in the main study’s web-based survey. The skip patterns in the web-based survey will hide questions not relevant to the respondent. Another frequent comment was that many of the matrices felt a bit repetitive, even though the scopes of the questions and items were technically different. To address this, one matrix and several items throughout the survey were removed to reduce repetitiveness. Other questions were also modified according to feedback from the pilot study participants.

Appendix 1. Feedback on the OWH WHLI Survey

Debriefing Question	Detailed Suggestion/Comment	Solution/Justification
What are your general impressions about the survey?	Survey well-constructed, well-organized. Generally thought survey was comprehensive but a little too long.	The paper-and-pencil format of the pilot survey made it seem long and included items that may have been extraneous to the respondent. However, the time to complete the survey was within our expectations. Since the survey in the main study will be web-based, the skip patterns will hide question not relevant to the respondent.
In SECTION1_1, did you think the definition of CHW was clear and comprehensive? Did you think Section 1 as a whole was clear and comprehensive enough to describe a CHW’s work history at the time of and after the training? How easy or difficult was it to follow the questions about CHW experience?	Definition of CHW was clear and comprehensive. Thought it was inclusive of different types of CHWs. The section was clear and comprehensive. Encompassed a CHW’s work history. Generally easy to navigate the questions in this section. For SECTION1_9 (settings/organization), not sure if “Other” was a required question. For SECTION 1_13 (primary activities), two items seemed very similar - “Eligibility screening and enrollment” and “Conduct eligibility screening and enrollment”. For SECTION1_14 (populations served), not sure if more than one population could be selected.	For SECTION1_9 (settings/organization), the matrix was changed so that “Other” is an open-ended response. This change was also applied to all other matrices with an “Other” option. For SECTION 1_13 (primary activities), “Eligibility screening and enrollment” was deleted. For SECTION1_14 (populations served), “Check ALL that apply” was added to the instructions.
SECTION4_5a-c asks “To what extent do you think that your leadership activities since WHLI training have resulted in the following changes?” How easy or difficult was it to answer? Do you feel the items listed successfully capture possible systematic changes resulting from CHWs’ leadership activities? Are there any other items you think could be added? Is the answer scale from “Major impact” to “No impact” appropriate, given your CHW experience?	Easy to answer this question. Items capture the possible types of systematic changes. For policy change, one respondent suggested “state/national” could be added to better encompass policy changes outside one’s community. Answer scale is appropriate.	Revised the definition of policy change to “Resulted in policy changes at the local, state, or national level.”
[If respondent did not start a CAP] SECTION4_7a-i lists some common facilitators and barriers of CHWs’ leadership activities. Are there additional facilitators or barriers that you feel should be included?	No other facilitators or barriers were suggested.	N/A
[If respondent started a CAP] SECTION5_9a-9g aims to measures your CAP’s sustainable change in the community. Are these appropriate items? Do you have any suggestions for other items that should be included?	Items measuring a CAP’s sustainable changes in the community are appropriate. No other items were suggested.	N/A
[If respondent started a CAP] SECTION5_10a-10m lists some common facilitators and barriers to the CAP. Are there additional facilitators or barriers that you feel should be included?	No other facilitators or barriers were suggested.	N/A
Were there any other questions you had difficulty with?	None noted.	N/A
Were there any questions you were uncomfortable with answering?	For SECTION6_2a-2s (WHLI training), one CHW respondent was uncomfortable rating Master Trainers (MTs) in general since she had three MTs at the training and had varying experiences with them. The other CHW respondent confirmed that she also had more than one MT during her training, although this respondent did not share the same concern about rating the MTs in general.	In the revised survey, we separated the question asking about feedback on the training itself and that asking about feedback on the MTs. We also added a question asking the respondents how many MTs they had during their training. Based on their answer, the table asking for MT feedback will load up to three times to get feedback on each MT individually.
Were there any terms, words, or phrases in the survey that you were unfamiliar with or did not understand?	None noted.	N/A
Were there any questions, items, or answer categories that you felt were repetitive?	Some respondents felt the matrices, especially in Section 3 (Knowledge and Competencies) and Section 4 (Leadership Activities and Impact), were a little repetitive, even if the items were technically different.	The SECTION3_5a-e matrix was cut, as it was very similar to the previous matrices in the same section. We have also excluded several items in other matrices throughout the survey to reduce the burden.
Are there any additional questions, items, or answer categories you feel should be included in this survey to help us to understand your experience in this program and its impacts?	One CHW respondent mentioned she would like an item about Master Trainer (MT) follow-up since she would have liked her MT to have followed up with her more after the training was over.	We clarified Item D in the SECTION5_10a-10m matrix, which asks for feedback on the Community Action Project (CAP). The item was changed to “A Master Trainer provided sufficient support for the CAP (e.g., to follow up on my progress and/or to answer my questions).”

Page 4 of 4