B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
The survey and semi-structured interviews are being administered to collect information about the impact of the WHLI. The survey will be administered to the CHW population, while semi-structured interviews will be conducted with a total of five samples (two groups of CHWs, MT, supervisors, and stakeholders).
1. Respondent Universe and Sampling Methods
Exhibit 4 below presents the respondent groups and number of respondents from each group that will be sampled for both the survey and semi-structured interviews.
Online Survey
For the online survey, given the relatively small number of CHWs who participated in the WHLI training and the potential challenges of locating them, we propose to survey the entire population, i.e. all 422 CHWs who participated in the WHLI training. Contact information for approximately half of the 422 CHWs who were trained under the WHLI has been provided by OWH. We plan to use locating procedures, i.e., a batch search using LexisNexis Accurint (http://www.accurint.com/) followed by case-level locating for non-responders, to identify contact information for as many of the remaining CHWs as possible.
Semi-structured Interviews
Interviews will be conducted with five different respondent groups: two groups of CHWs, MTs, CHW worksite supervisors, and community stakeholders. This section describe the sampling strategy for each group, the process for recruiting these groups to participate in interviews; and the process for follow-up with non-responders.
Proposed sampling strategy and content of the interviews for each of the respondent groups varies. Exhibit 4 provides information about the population and sample for each group.
Group 1: Community Health Workers (CHWs). We plan to contact all 30 CHWs who successfully completed a Community Action Project (CAP), according to our record, (regardless whether they completed the online survey) and invite them to participate in an interview.
Group 2: CHWs with non-CAP leadership activities. We will also contact an additional 10 CHWs who were unable to complete a CAP, but have made significant changes in their home organization and/or community, as identified through reviewing their responses to the online survey.
Group 3: Master Trainers (MTs). A total of 18 individuals completed the WHLI Master Trainer training, and were certified as WHLI MTs. We will contact all 18 WHLI MTs and invite them to participate in an interview.
Group 4: CHW Worksite Supervisors. For semi-structured interviews with CHW worksite supervisors and community stakeholders, we will use snowball sampling, asking the 40 CHWs we interview to identify suitable participants. Worksite supervisors will be contacted if they supervised the CHW during the time of the training and/or post-training. We expect to interview up to 20 worksite supervisors.
Group 5: Community Stakeholders. CHWs will help us to identify community stakeholders. Community stakeholders will be contacted if they were directly impacted by the CHW’s CAP or other leadership activity. We expect to interview up to 20 community stakeholders identified by CHWs.
Exhibit 4. Overview of Sample
Respondent group |
Population |
Sample |
Online survey |
||
Community Health Workers trained by WHLI |
422 |
422 |
Semi-structured interviews |
||
Groups 1 and 2: CHWs with and without CAP |
422 |
40 |
Group 3: WHLI master trainers |
18 |
18 |
Group 4: CHW worksite supervisors |
40 |
20 |
Group 5: Community stakeholders identified by CHWs |
40 |
20 |
2. Procedures for the Collection of Information
Online Survey
The online survey will be conducted using a CAWI (computer-assisted web interview) system and the fielding of the survey will take approximately three months. Respondents will be recruited for the online survey using a sequential multi-mode contact approach that utilizes email, mail and telephone. Recruitment will first entail emailing an online survey invitation to CHWs that provides an introduction to the evaluation and contains a link to the online survey (Attachment 12). Two reminder email invitations will then be sent to non-responders over a week’s time (Attachment 13 and 14). We expect that approximately 25 percent of the CHWs with viable email addresses will complete the survey without additional prompting. Those CHWs who do not respond to the email invitations or do not have a valid email address will be sent a letter through the mail (Attachment 15 and 16) followed by a reminder postcard (Attachment 17). The mail materials will also provide an introduction to the evaluation and a link to the online survey.
Additional targeted telephone prompting of non-respondents will be used to increase participation. We will make a maximum of five calls to CWHs with available telephone numbers that did not respond to either the email or mail contacts. Telephone calls will be made approximately three days apart. Telephone interviewers will use standard scripts for answering machine messages and when initial contact is made with a CHW or associated household member. Respondents contacted by telephone will be given the choice to complete the survey by telephone.
Exhibit 5 below provides an overview of the timeline to be followed in fielding the survey.
Exhibit 5. Timeline for Survey Data Collection
Action |
Timeframe |
OMB approval |
Dependent upon OMB’s timeline |
First email invitation sent (Attachment 12) |
Start of data collection; beginning of the week |
Second email invitation sent (Attachment 13) |
End of the week (3 days after first email) |
Third email invitation sent (Attachment 14) |
Start of the following week (7 days after first email) |
First letter sent via USPS (Attachment 15) |
End of the following week (11 days after first email) |
Reminder postcard sent via USPS (Attachment 17) |
Approximately a week after first letter is sent (21 days after first email) |
Telephone prompting begins (Attachments 18 and 19) |
A week after reminder postcard is sent (28 days after first email) |
Case-level locating beings |
Locate non-responders after first telephone call does not reach the desired respondent |
Semi-structured Interviews
Semi-structured interviews will be conducted over a period of four months. Interviews will be recorded and conducted by telephone by one team member, with a second member taking notes. At the start of each interview, respondents will be provided with information on the purpose of the interview and the confidentiality terms, so they may give verbal informed consent for participation (Attachment 5). To ensure consistency, interviewers will use a standardized protocol to direct the conversation during semi-structured interviews (Attachments 8, 9, 10, and 11).
The following paragraphs describe our recruitment strategy for the semi-structured interviews:
Group 1: CHWs who completed a CAP project: Once the CHWs complete the online survey we will begin the recruitment process for the interview. The final page of the survey will inform them that they will be contacted to complete a phone interview and they will be given the opportunity to enter their most current phone number and email address. A member of the research team will contact the CHW via phone (Attachment 20) or email (Attachment 21) to schedule a time for the interview.
Group 2: CHWs with non-CAP leadership activities: Once the CHWs complete the online survey they will be contacted to complete an interview. The final page of the survey will inform them that they may be contacted to complete a phone interview and they will be given the opportunity to enter their most current phone number and email address. Researchers will review the survey responses to select CHWs who will be invited to participate in an interview. Up to 10 CHWs who report that they have made significant impact in the organization or community will be contacted for an interview. A member of the research team will contact the CHW via phone (Attachment 20) or email (Attachment 21) to schedule a time for the interview.
Group 3: Master Trainers: A member of the research team will begin contacting Master Trainers by phone (Attachment 22) or email (Attachment 23) to set-up a time for the interview.
Group 4: CHW Worksite Supervisors: During interviews with CHWs, we will ask CHWs for their permission to contact their worksite supervisor (i.e., the supervisor at the time of their training and/ or shortly after). If permission is granted, we will ask the CHW to provide the most recent contact information for their supervisor. A member of the research team will then contact the supervisor via phone (Attachment 24) or email (Attachment 25) to schedule a time for the interview.
Group 5: Community Stakeholders: During interviews with CHWs and CHW worksite supervisors, we will ask for recommendations of stakeholders (community members) that we could interview about the impact of CHWs’ CAP projects or other leadership activities. We will ask the CHW and the CHW supervisor to provide the most recent contact information for the stakeholder. A member of the research team will then contact the stakeholder via phone (Attachment 26) or email (Attachment 27) to schedule a time for the interview.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Online Survey
Our primary method to maximize response rates and deal with nonresponse is our sequential, multi-mode contact approach described above (email, mail, telephone), which is aimed at directing respondents to complete the survey via a secure web-based reporting system. As shown in Exhibit 5 above, if CHWs have not completed the online survey 11 days after the first of three email invitations is sent, other contact methods (mail and telephone) will be utilized. The first mail invitation will be sent at the point where response to the initial email invitation begins to fall off and encourages initial non-responders to complete the questionnaire, as well as reaches respondents who did not receive the email invitations. Similarly, telephone calls will begin after the mail materials have had time to reach CHWs and response begins to taper off. Telephone scripts will be developed by the study team and will be standardized across interviewers. To maximize response rates, telephone interviewers will attempt to complete the survey with respondents over the phone on that call, or at another time that is convenient for the respondents. This approach is based on previous project experience and recommendations made by Dillman and colleagues.1 We have found a combination of contacting methods to be an effective and efficient means for achieving high response rates. In addition, using multiple contact modes creates opportunities to provide information that a potential respondent can use to confirm that the survey request is legitimate.
The contact protocol is based on having email addresses, mailing addresses, and telephone numbers for the CHWs; however, if information is missing for specific CHWs, the contact protocol will be tailored appropriately. If a CHW cannot be reached by email, mail or telephone, case-level locating will be conducted in order to identify updated contact information for the CHW. NORC uses a Windows-based locating case management system to manage and track the progress of all cases that are identified as needing locating. Any new contact information returned from locating will undergo similar contact attempts as described above.
NORC will also attempt to maximize response rates by attempting to re-engage CHWs that completed the WHLI program using recruitment materials with familiar images (e.g., the WHLI logo) and thanking them for their participation in the program. Emails, letters and postcards will include these consistent images and messaging.
As discussed in Section A9, another way that NORC will address non-response is by offering monetary incentives to CHWs that participate in the online survey. Research suggests that incentives increase response rates in surveys and that, as the value of the incentive increases, so do the number of completes.23 Offering $20 to CHWs for completing the online survey is expected to increase response rates in a similar fashion.
Semi-structured Interviews
We expect to interview up to 98 participants, including 40 CHWs, 18 MTs, 20 CHW worksite supervisors, and 20 community stakeholders. Potential participants will be offered $20 for completing the in-depth telephone interview. Similar to the rationale provided for the online survey incentive, offering $20 for completing the interview is expected to increase response rates and will allow us to interview our desired number of respondents. Efforts to maximize response rates and address nonresponse for the semi-structured interviews are also similar to those used for the online survey. A multi-mode contact protocol, tailored based on the availability of contact information for each potential interviewee, will be used to increase response rates.
Follow-up methods for each respondent group are described below:
Groups 1 and 2: Community Health Workers (with or without a CAP)
CHWs for whom we have a phone number. CHWs who do not respond to our initial invitation call will receive an email (if an email is provided) directly after the call and then a follow-up call 2-4 days later. If the CHW does not answer the second call we will leave a voicemail. If they do not respond to the follow-up call and voicemail, they will receive another email within 1 week after the voicemail is left. If they do not respond to the email, they will receive two follow-up emails spaced 1 week apart.
CHWs for whom we not have a phone number. CHWs will be sent an email invitation to participate. CHWs who do not respond to our initial email will be sent a follow-up email 4 days after the initial email. If they still do not respond, they will receive two follow-up emails spaced 1 week apart.
Group 3: Master Trainers
If MTs do not answer our initial call, we will leave a voicemail and send a follow-up email directly after leaving the voicemail. If they do not respond we will make a follow-up phone call 3-5 days after leaving the voicemail. If they do not answer or respond, we will send another email 2 days later. If they do not respond to the email, we will send two follow-up emails spaced 1 week apart.
Groups 4 and 5: Workplace Supervisors & Stakeholders
Supervisors & Stakeholders for whom we have a phone number. Supervisors and stakeholders who do not respond to our initial invitation call will receive an email (if an email is provided) directly after the call and then a follow-up call 2-4 days later. If they do not answer on the second call we will leave a voicemail. If they do not respond to the follow-up call and voicemail, they will receive another email within 1 week after the voicemail is left. If they do not respond to the email, they will receive two follow-up emails spaced 1 week apart.
Supervisors & Stakeholders for whom we not have a phone number. Supervisors and stakeholders will be sent an email invitation to participate. Supervisors and stakeholders who do not respond to our initial email will be sent a follow-up email 4 days after the initial email. If they still do not respond, they will receive two follow-up emails spaced 1 week apart.
4. Test of Procedures or Methods to be Undertaken
Questions on CHW competencies incorporated into the survey have been used previously in the short-term evaluation of the WHLI. Some questions about the impact of the WHLI on CHW work activities, CHW leadership qualities, skills, and activities, and CHW engagement in social change were also previously utilized in an evaluation conducted nine months after the CHWs were trained. An NORC senior survey methodologist, as well as several members of OWH, reviewed the survey questions and interview protocols and their comments were incorporated into the final versions.
In addition, the survey was pilot tested by a group of CHWs and NORC staff. Please see Attachment 3 which contains the Pilot Report. Key elements of the pilot testing and the results are summarized here. A random sample of the CHWs who participated in the WHLI training were contacted and invited to participate in the pilot study. A Microsoft Word version of the survey was emailed to participants, and they were asked to print it out and complete it. Once the survey was completed, they were invited to participate in a telephone or in-person debriefing with two members of the NORC evaluation team. Two CHWs and three NORC staff members completed the WHLI survey and the debriefing.
The pretest allowed OWH and NORC to confirm CHW’s understanding of the survey questions, the flow of the survey, and provided an opportunity for them to make suggestions on survey questions and answer categories. Feedback from the pilot study participants indicated the survey was comprehensive and that the instructions in the survey were generally easy to follow. The time to complete the survey met our expectations of approximately 25 minutes. Feedback provided by pretest responders has been incorporated into the survey. Questions deemed to be repetitive were removed, and questions reported to be ambiguous were clarified.
The CHW survey and semi-structured interview protocols for the CHWs, workplace supervisors, and community stakeholders were reviewed by one of the WHLI Master Trainers. Feedback and suggestions for increasing clarity and readability were incorporated.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Program Development Contact
Adrienne M. Smith, Ph.D. M.S., CHES
U.S. Department of Health and Human Services
Office on Women's Health
202-205-0270
Instrument Development/Data Collection/Data Analysis Contacts
Weiwei Liu, Ph.D.
Senior Research Scientist
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9559
Britta Anderson, Ph.D.
Research Scientist
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9356
Caitlin Oppenheimer, M.P.H.
Senior Vice President
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9322
Felicia Cerbone, M.A.
Principal Research Scientist
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9328
Michael Stern, Ph.D.
Fellow, Center for Excellence in Survey Research
NORC at the University of Chicago
55 East Monroe, 30th Floor
Chicago, IL 60602 (312) 357-3891
Jacquelyn George, B.A.
Survey Director II
NORC at the University of Chicago
55 East Monroe, 30th Floor
Chicago, IL 60602
(312) 325-2573
Amy Rosenfeld, M.P.H.
Senior Research Analyst
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9549
Rachel Weingart, B.A.
Research Assistant
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9377
Katherine Groesbeck, B.A.
Research Assistant
NORC at the University of Chicago
4350 East-West Highway, 8th Floor
Bethesda MD 20814
(301) 634-9450
1 Dillman, DA, Smyth JD, Christian LM. 2009. Mail and Internet Surveys: The Tailored Design Method, Third edition. New York: John Wiley and Sons
2 Stern, MJ, Bilgen I, Wolter KM. 2014. Do Sequence and Mode of Contact Impact Response Rates for Web Only Surveys? Presented at the 69th Annual Conference of the American Association for Public Opinion Research (AAPOR) in Anaheim, CA.
3 Lewis D, Creighton K. 2005. The Use of Monetary Incentives in the Survey of Income and Program Participation. The American Association for Public Opinion Research (AAPOR) 60th Annual Conference.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Windows User |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |