Multi-Site Implementation Evaluation of Tribal Home Visiting
OMB Information Collection Request
New Collection
Supporting Statement
Part B
Non-Substantive Change Request February 2019
Initial Request Approved December 2018
Submitted July 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, S.W.
Washington, D.C. 20201
Project Officer:
Aleta Meyer
Part B. Collections of Information Employing Statistical Methods 1
B1. Respondent Universe and Sampling Methods 1
B2. Procedures for Collection of Information 6
B3. Methods to Maximize Response Rates and Deal with Nonresponse 9
B4. Tests of Procedures or Methods to be Undertaken 13
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 13
Attachments:
Attachment E. Caregiver Interviewee Selection Guide
Attachment F. Caregiver Informational Flyer
Attachment G. Staff Survey Invitation Email
Attachment H. Staff Interview Invitation Email and Phone Script
Attachment I. Caregiver Interview Invitation Phone Script
Attachment J. Program Data Collection Reminder Emails
Attachment L . Introduction to MUSE Video Script
List of Exhibits
Exhibit B.1. Data Collection Activities that Include Caregiver Data, by Caregiver Group 2
Exhibit B.2. Minimum Detectable Associations for Caregiver Surveys 3
Exhibit B.3. Qualitative Interview Estimated Sample Size by MUSE Staff Role 6
Seventeen tribal communities received Tribal MIECHV Implementation and Expansion grants and are eligible to participate in the Multi-Site Implementation Evaluation of Tribal Home Visiting (MUSE). Based on enrollment data reported to ACF, we estimate that 1,441 caregivers receiving home visiting could be recruited for MUSE over an 18-month participant recruitment period. We will aim to successfully recruit and consent 75 percent of these caregivers for a total of 1,081 caregivers (see below for explanation of burden estimates in Supporting Statement A). Data collection within the context of established home visitor-caregiver relationships and a service delivery model that collects routine data for purposes outside of this study are expected to support high consenting rates among caregivers. Ongoing communication, training, and engagement with grantee program staff are also expected to result in a high-quality informed consent process and a greater likelihood of positive communication about the study from grantee staff to caregivers. Due to the relatively small universe of caregivers receiving services from Tribal MIECHV programs, we do not plan to sample caregivers and will instead recruit all caregivers to participate in MUSE. Our sampling strategy for each data collection method corresponds to the unique purpose of each data collection activity.
There are two universes of caregivers receiving Tribal MIECHV services that will be recruited for MUSE: (1) Group 1: caregivers who have started home visiting services before MUSE data collection begins and (2) Group 2: caregivers who start home visiting services after MUSE data collection begins. These groups only reflect when caregivers enrolled in home visiting services relative to the beginning of the MUSE study; they do not reflect different assignment to groups for the purposes of comparison or delivery of different types or amount of services. Caregivers in Group 1 will be recruited into MUSE during the first two months of the study only. We estimate that there will be 812 caregivers in Group 1 who will be recruited to participate in MUSE and 609 that consent to participate in the study (75%). Caregivers in Group 2 will be recruited into the study over a 17-month period as they enroll in home visiting services. We estimate that there will be 629 caregivers in Group 2 who will be recruited to participate in MUSE and 472 that will consent to participate (75%).
In Supporting Statement A, we have estimated additional burden for each instrument beyond the sample estimates in case grantees enroll more caregivers, complete more home visits, or retain caregivers at a higher rate than we estimated based on available data. For our sample estimates we project that 75% of eligible caregivers will consent to participate in MUSE; whereas for our burden estimates, we project that 95% of caregivers will consent to participate. Exhibit B.1 shows in which data collection activities caregiver information will be included, for each group.
Caregivers |
Caregiver Enrollment Form |
Caregiver Surveys |
Rapid Reflect Self-Completed Questionnaires |
Qualitative Interviews |
Administrative Program Data |
Group 1 |
X |
|
X |
X (subsample) |
X (Some) |
Group 2 |
X |
X |
X |
X (subsample) |
X (All) |
We have chosen to collect a more complete set of data on caregivers in Group 2 for the following reasons:
We will only be able to collect baseline data on key indicators from caregivers in Group 2 because they will have begun home visiting after MUSE data collection begins, whereas Group 1 will have begun home visiting services prior to MUSE data collection.
We will be requesting additional administrative data from this group in order to correlate these data with responses from the Caregiver Surveys.
We will be administering Rapid Reflect questionnaires to caregivers in Group 2 and Group 1 in order to get a more complete picture of what happens in home visits, including those home visits provided to caregivers who have been receiving home visiting for more than a year.
We are requesting that grantees provide a limited number of administrative data elements from caregivers in Group 1 to help us interpret data collected with the Rapid Reflect for caregivers in this group.
In addition to the reasons cited above, we felt that limiting data collection for caregivers in Group 1 resulted in a more manageable level of burden for participating grantees.
Caregiver Enrollment Form and Caregiver Surveys
Home visitors will complete a Caregiver Enrollment Form (Instrument 1), entering basic information that the program has previously collected on the caregiver who consented to participate in MUSE. We estimate that 93 home visitors will complete Caregiver Enrollment Forms for 1,081 caregivers. Only caregivers who start home visiting after MUSE data collection has begun (Group 2) will be invited to take the Caregiver Surveys (Instruments 2 and 3). We have chosen to restrict the universe for the Caregiver Surveys to caregivers in Group 2, because the study design requires a baseline measure of items within the Caregiver Survey obtained when the caregiver begins home visiting services. We anticipate that 472 caregivers from Group 2 will consent to participate in MUSE and will be asked to take the Baseline Caregiver Survey. Of those invited to take the survey, we estimate that 95 percent of the 472 consented will complete the baseline survey (n=448).We estimate that 75 percent (n=354) of the 472 caregivers in Group 2 who consented to participate in MUSE will still be participating in home visiting at 6 months and will be asked to take the 6-month survey. If 95% of the 354 caregivers complete the survey, we will have a sample of 336 caregivers for the 6-month Caregiver Survey. Caregivers who begin MUSE in the final five months of caregiver enrollment into the study will not become eligible to take the 12-month follow-up survey before the end of the MUSE data collection period, which leaves a total of 336 eligible caregivers at 12 months. We estimate that 60 percent (n=202) of these 336 caregivers will still be participating in home visiting at 12 months, and 95% of those caregivers will agree to take the Caregiver Survey, which produces a sample size of 192 caregivers for the 12-month survey.
The first two aims of the MUSE study are descriptive; the third aim – Explore supports and challenges to home visiting implementation in tribal communities – will involve estimating statistical relationships among variables. To understand the sufficiency of our sample to address this aim and detect significant associations among study variables, we conducted power analyses. Those analyses are presented here because they are most relevant to the Caregiver Surveys. For most other participant groups, the size of the sample is constrained by the small number of staff in these roles across the 17 grantees, limiting power to detect associations to larger effects than we would expect to find (i.e., correlations > .5). Analyses of those surveys thus will largely be limited to descriptive goals under Aims 1 and 2.
Exhibit B.2 shows the minimum effects we will be able to detect within each wave of Caregiver Survey data collection, at α <= .05 with power >= .80, as statistically significant bivariate correlations among variables collected from caregivers. Power analyses indicate that the estimated sample size at baseline (N=448) will be sufficient to detect a small effect (r=.131), and will remain sufficient to detect relatively small effects at both the 6-month (N=336; r=.152) and 12-month follow-ups (N=201, r=.195). In Supporting Statement A, we estimate higher retention rates at 6 months (85%) and 12 months (70%), to guarantee that we have requested enough burden for the study. The higher retention rates and consenting rates used to calculate burden means we estimate burden for an additional 117 caregivers at baseline, 145 caregivers at 6-months, and 90 caregivers at 12-months. As indicated in Exhibit B.2, if these sample sizes are achieved, slightly smaller associations will be detectible (r=.164 at 6-month follow-up and .188 at 12-month follow-up).
Exhibit B.2. Minimum Detectable Associations1 for Caregiver Surveys
|
|||||
|
Estimated Retention Rate (%) |
Estimated No. Eligible Participants |
Projected Response Rate (%) |
No. of Responses |
Minimum Detectable Association |
Baseline |
n/a |
472 |
95% |
448 |
.131 |
6-month follow-up |
|
|
|
|
|
conservative estimate for power |
75% |
354 |
95% |
336 |
.152 |
estimate for burden |
85% |
301 |
95% |
286 |
.164 |
12-month follow-up |
|
|
|
|
|
conservative estimate for power |
60% |
212 |
95% |
201 |
.195 |
estimate for burden |
75% |
226 |
95% |
215 |
.188 |
1 Smallest bivariate correlation coefficients detectable at α = .05 and power = .80 |
|||||
Examples of relationships we will examine in the Caregiver Survey data, based on the MUSE conceptual model co-created with Tribal Home Visiting grantees and other stakeholders include associations of variables within each time point, such as perceptions of the relationship with the home visitor and satisfaction with home visiting, with parenting self-efficacy, and with social support. We will also examine associations across time, such as perceptions of the home visitor relationship at 6-months with parenting self-efficacy and social support at 12-months. The fact that we will have the sensitivity to detect even small associations among these variables, both within and across time, will be essential to the success of MUSE in providing an unprecedented look at how home visiting is being implemented in tribal communities and how it is experienced by caregivers served by those programs.
Rapid Reflect Self-Completed Home Visit Questionnaires
Home visitors and caregivers both (Group 1 and Group 2) will complete a Rapid Reflect Self-Completed Questionnaire (Instruments 4 and 5) for every home visit conducted during one week out of each month. This is an effective sampling rate of 23% of home visits conducted. The assigned Rapid Reflect week will rotate to ensure that we capture home visits happening with families on different home visit schedules. This sampling strategy will allow us to collect data on some home visits for most families participating and should result in data on a sample of home visits conducted with a wide variety of families, throughout the course of home visiting, and throughout the calendar year. This sampling strategy should result in a representative sample of home visits with approximately 919 caregivers completing at least one Rapid Reflect. Because the selected weeks will rotate, not all caregivers who consent to participate in MUSE will be asked to complete a Rapid Reflect; some will not have a visit that falls during the selected weeks. All home visitors will complete Rapid Reflect questionnaires, along with additional staff who are completing home visits on a temporary basis (N=93). As with the other instruments, we have estimated additional burden in case response rates are higher than anticipated or we experience a fewer number of home visits that are ineligible for the Rapid Reflect than we expect to occur. We have estimated burden for an additional 217 caregivers to complete the Rapid Reflect.
Staff Surveys
Staff from Tribal MIECHV grantees participating in MUSE will be invited to take the staff survey. Managers will also be invited to take an additional survey—the Program Implementation Survey—and complete that survey together, if needed. Staff that fit into the following four MUSE staff roles will be surveyed: program directors (Instrument 8), program coordinators/managers (Instruments 7 and 10), home visitors (Instrument 6), and local evaluators (Instrument 9). Prior to recruiting staff for the survey, the MUSE Team will clearly define each MUSE staff role and work with grantees to determine which staff members fit within each MUSE staff role.
Because of the expected universe size, we will invite all Tribal MIECHV grantee staff and local evaluators who fit into the four MUSE staff roles and whose contact information is provided to the MUSE Team to participate. To estimate the number of individuals that will be recruited to take the staff survey, we reviewed information gathered by ACF on the number of full time equivalent positions (FTE) funded in each category at each participating grantee. We also reviewed data on staff attrition rates. The MUSE Team will send survey invitations to approximately: 17 program directors, 17 program coordinators/managers, 70 home visitors, and 25 local evaluators. We will send the Program Implementation Survey to approximately 17 program coordinators/mangers and they will be instructed to complete the survey with their program director. We estimate that we will invite a total of 129 program staff and evaluators to take the surveys, but we estimate burden for up to 153 staff and evaluators to account for any unexpected staff turnover. We anticipate an 85 percent response rate for the staff surveys.
Qualitative Interviews (Instruments 11-14)
The qualitative component of MUSE uses a purposive sampling of multiple research participant groups including managerial staff (program coordinators/managers and directors), evaluators, home visitors, and caregivers receiving home visiting services in both Groups 1 and 2.
Program Staff Sampling
Each Tribal MIECHV grantee has a unique staffing structure, which typically includes a small leadership team (between 1-2 administrators). Additionally, each grantee includes evaluation staff, but the structure of the evaluation team varies widely across programs (i.e. evaluator on staff, sub-contractor). Given the small size and unique structure of leadership and evaluation teams across these grantees, MUSE will use a universal approach to sampling directors, coordinators/managers and evaluators for interviews. These individuals will be asked to reflect on program level phenomena, such as staff supports, program design, and institutional resources.
Due to the highly variable nature of the home visitor teams across grantees, we will select home visitors based on some basic criteria and then use emergent sampling, incorporating quantitative data when possible.
For grantees that employ two home visitors or less, we will interview each home visitor.
For grantees with more than two home visitors, we will select possible interviewees based on the following criteria: home visitor tenure, FTE level, and case load. We will prioritize interviewing home visitors who have worked longer than 6 months, have full FTE, and who maintain full caseloads. Using these criteria will help ensure that the home visitors who are asked to reflect on their experience can draw from ample evidence across multiple families over a longer period of time. Additionally, we will consider the gender of the home visitor as well as his/her tribal affiliation during the selection process to ensure that we achieve maximum variation within these discrete sets of characteristics. Depending on the results and timing of initial quantitative assessments, it may be possible to incorporate stratified purposive sampling techniques.
Caregiver Sampling
Caregiver interviewees will be selected in collaboration with grantee personnel. We will provide a Caregiver Interviewee Selection Guide (Attachment E) to assist grantees in identifying potential caregiver interviewees, which will include a list of the various characteristics and any prioritizations. Caregiver interviewees will be individuals who: serve as a primary caregiver (e.g., mother, father, grandparent) and actively participate as a primary caregiver in home visits. When appropriate and feasible, we will prioritize selection of male caregivers, first-time parents, and grandparent caregivers as well as consider tribal affiliation, age, and number of children to help ensure maximum variation across caregiver types. As with sampling home visitors, it may be possible to establish rigorous strata through quantitative assessments to conduct stratified purposive sampling techniques for caregiver interviewees as well.
Approximately 157 participants will be included in the interview component across 17grantees. We anticipate a response rate of 90% for the qualitative component. Exhibit B.2 provides the estimated sample size by MUSE staff role.
Exhibit B.3. Qualitative Interview Estimated Sample Size by MUSE Staff Role
MUSE Staff Role |
Sample Size |
Home visitors |
42 |
Managerial staff (program coordinators/managers or directors) |
34 (2 per site) |
Local evaluators (lead evaluators, data support staff) |
30 (1-3 per site) |
Caregivers participating in home visiting |
51 (3 per site) |
Total |
157 |
Implementation Logs
Implementation Log data (Instrument 15) will be collected from all 17 grantees participating in MUSE. A designated staff person, most likely the program coordinator/manager, at each grantee will complete the Implementation Logs each month over the two-year data collection period. The logs collect information on staff hires and departure, training and supervision received by all home visitors at each grantee, as well as family group activities. We expect that 85 percent of the monthly Implementation Logs will be submitted.
Administrative Program Data
Administrative Program Data (Instrument 16) will be submitted by all 17 grantees participating in MUSE. Some Administrative Program Data will be submitted for all caregivers who consent to participate in MUSE (Groups 1 and 2). The MUSE consent forms (See Attachment A) seek consent for both retrospective and prospective data collection of Administrative Program Data. We estimate that 1,081 caregivers will consent to participate in MUSE and agree to share some of the data they provided to the home visiting program with the MUSE study. The MUSE Team will ask grantees to provide more extensive demographic and service utilization data from an estimated 472 caregivers who start home visiting services after MUSE begins and consent to participate in the study (Group 2).
Caregiver Recruitment, Consent, and Enrollment into MUSE
Caregivers who are enrolled in one of the 17 Tribal MIECHV programs will be invited to participate in the study by their home visitor. As described above, caregivers will be invited to participate in each data collection activity on the basis of whether they started receiving home visiting services before the beginning of MUSE data collection (Group 1) or started receiving home visiting services after MUSE data collection began (Group 2). See Exhibit B.1 above for data collection activities involving caregiver data by caregiver group.
All recruitment will take place through one-on-one, in-person or virtual communication. Prior to formally inviting caregivers to participate, home visitors will introduce caregivers to the study during a regularly scheduled home visit with the aid of a video (Attachment L) and informational flyer (Attachment F). During the following home visit, home visitors will invite caregivers to participate in the study, explain all data collection activities, and lead them through the informed consent process. Consent can be obtained in-person or over the telephone or videoconference. The MUSE study has obtained an IRB-approved waiver of parental consent for minor caregivers. Home visitors will use a tablet to complete an electronic consent form (Attachment A). Once a caregiver has consented to participate in MUSE, the home visitor will complete the Caregiver Enrollment Form. The Caregiver Enrollment Form contains basic demographic information about the caregiver, already collected by the program. This information is used to track data collection for each caregiver within the MUSE data system.
Caregiver Surveys
Surveys of Group 2 caregivers (i.e. caregivers who start home visiting services after MUSE data collection begins) will be administered by a member of the home visiting program staff during regularly scheduled home visits at baseline, 6-month follow-up, and 12-month follow-up. The Caregiver Survey will typically be administered at the location the caregiver has chosen for their home visit (most often, this location is the caregiver's home). Home visitors will provide a tablet to the caregiver with the surveys pre-loaded and answer any of the caregiver’s questions. Home visitors will not ask caregivers the survey questions, they will be trained in protocols to ensure that caregivers are given privacy to complete surveys, and all staff collecting these data will be certified in human subjects’ research protections. Caregivers receiving virtual home visits will be able to take the survey over the phone or on a personal device using a web survey link. In addition, surveys will be collected on tablets or via the web survey link using secure software that ensures caregiver responses cannot be accessed by home visiting program staff. Caregivers with limited literacy or visual impairments can choose to complete the surveys over the phone with a member of the MUSE study team. Caregivers who are not able to use a study tablet to complete the survey (because they are receiving virtual home visits or are experiencing limited literacy or visual impairments) will receive a $5 reimbursement for the use of their personal cell phone minutes or data needed to complete the MUSE survey. MUSE informed consent and Baseline Caregiver Surveys will take place after caregivers complete the grantee’s local program enrollment and initial data collection processes. The MUSE study team will work with grantees to finalize exact MUSE data collection and consent timeframes.
Rapid Reflect Self-Completed Home Visit Questionnaires
Home visitors and caregivers will complete Rapid Reflect short surveys for every in-person home visit conducted with caregivers who have consented to participate in MUSE during one designated week per month. During the designated Rapid Reflect week, all participating caregivers will be asked to take a five-question survey about the home visit that was just completed. Home visitors will be trained to hand caregivers a MUSE data collection tablet and provide privacy for the caregiver to independently answer the questions on the tablet. Caregivers with limited literacy or visual impairments can choose to hear the questions by selecting to play an audio recording of the questions and answer options.
Immediately following the home visit, home visitors will complete a Rapid Reflect survey about the same visit using the tablet. Throughout the twenty-two-month Rapid Reflect data collection period, the assigned Rapid Reflect week will rotate to ensure that MUSE captures home visits happening with caregivers on different home visit schedules. Home visitors will be intensively trained to administer the caregiver Rapid Reflect and to complete the home visitor Rapid Reflect during MUSE data collection training. Home visitors will use a tablet to complete Rapid Reflect surveys about home visits that occur virtually, but caregivers will not be asked to complete Rapid Reflects on virtual home visits.
Staff Surveys
Surveys of home visiting program staff-- including home visitors, program coordinators/managers, program directors, and local evaluators-- will be administered electronically through a web-based data system once, during the first month of the study, and on a rolling basis for new hires. Each staff person will receive via email (Attachment G) a survey tailored to their role. Grantee staff in a managerial role will also be invited to complete a second survey focused on program implementation via email (Attachment G) as a management team. Informed consent will be obtained electronically before the respondent proceeds to the survey questions (Attachment A). The MUSE Team will send four email reminders (Attachment G) and will consider instituting telephone reminders if necessary. While some of the staff surveys are lengthy (30-70 minutes), respondents can complete the survey at a time that is convenient for them, including over multiple sittings, if needed. The MUSE Team elected to administer staff surveys once instead of multiple times due to the burden associated with responding to the surveys.
Qualitative Interviews
In-person qualitative interviews with home visiting program staff and enrolled caregivers will be conducted by members of the research team either virtually or during a single site visit for each grantee. For in-person interviews, the investigator and/or other members of the MUSE Team will partner with an individual at each grantee to reserve a room or rooms at the grantee program office to conduct the interviews. Virtual interviews will be conducted using a secure telephone or videoconferencing platform.
Approximately 2 home visitors, 1 program director, 1 program coordinator/manager, and 1-3 local evaluators will be invited to participate in a qualitative interview at each grantee. Specific interviewees will be determined on a site-by-site basis depending on local staffing structure. In the case of grantees with more than 2 home visitors, purposive sampling criteria such as tenure, FTE, caseload, and gender will be considered in selecting potential interview participants. The MUSE Team will directly contact potential staff interview participants by phone or email (Attachment H), provide basic information about the study and the interview specifically, and invite the individual to participate in an interview. Individuals who agree to move forward will be scheduled for an interview. Consent will be obtained by the MUSE Team.
Three caregivers at each grantee who have consented to participate in MUSE will be invited to participate in qualitative interviews. Potential participants will be identified based on purposive sampling criteria to ensure representativeness and robust findings. The MUSE Team will work with local grantee staff to identify caregivers who fit the sampling criteria and may be interested in participating in an interview. Once a caregiver has been identified, local grantee staff will contact him or her to provide the study flyer (if they have not already received it) and ask for initial permission to provide their name and contact information to the MUSE study team. If the caregiver agrees, a MUSE Team member will contact them directly to review the study (Attachment I) and, if they agree to an interview, schedule them for an interview conducted virtually or during a site visit. On a case-by-case basis, the MUSE Team member(s) conducting an interview with a caregiver may elect to travel to the caregiver's home to conduct the interview to allow flexibility or accommodate participant preferences. Consent to participate in the interview will be obtained by a member of the study team (Attachment A).
Implementation Logs
Monthly Implementation Logs will be completed by a grantee’s program coordinator/manager to report data on staff hires and departures, training, individual and group supervision, and family group activities. A web-based data system will allow grantees to access data entry for each component of the Implementation Logs. Program coordinators/managers will be directly oriented to the Implementation Logs during MUSE data collection training.
Administrative Program Data
Tribal MIECHV grantees collect a substantial amount of data for Federal reporting, model reporting, and local quality assurance processes that can also be used to answer the MUSE study questions. To reduce burden and avoid collecting the same data twice from caregivers receiving home visiting, the MUSE study design includes analyses of existing Administrative Program Data collected and maintained by Tribal MIECHV grantees.
Home visitors will be trained to administer informed consent to caregivers receiving home visiting services. During the informed consent process, home visitors will explain the data collection activities that caregivers are being asked to participate in, and that participating in MUSE involves sharing some of the data their program collects about them, their child, and the services they received with the MUSE study (i.e., Administrative Program Data). Grantees participating in MUSE will only share Administrative Program Data about caregivers who have consented to participate in MUSE.
Four types of Administrative Program Data will be reported for the MUSE study: participant demographics, screener data, home visit participation data, and group activity participation data (Instrument 16). Home visitors already collect this information from caregivers and will continue to the collect the information following their local data collection protocols. Grantees will be asked to submit individual level data to support analyses that examine associations between participant characteristics and implementation factors including: participation in home visiting, home visit content, relationship with their home visitor and satisfaction with home visiting. Grantees will upload the data to a secure, password protected cloud-based data storage website. Grantees will submit Administrative Program Data semi-annually for a total of four times.
Expected Response Rates
For the past 18 months, the MUSE Team has conducted extensive stakeholder engagement activities with all 17 Tribal MIECHV grantees who have agreed to participate in MUSE as well as other stakeholders. We have attempted to design a study that will answer questions of interest to participating grantees, utilize existing data where possible, and involve a feasible amount of data collection burden. For these reasons, we believe we can expect a relatively high participation rate from grantees and the home visiting staff at each grantee.
Based on the success of grantee recruitment into MUSE discussed in section B1, we anticipate response rates above 90 percent for the qualitative interviews conducted during each grantee’s site visit. Ongoing partnership within the community-engaged MUSE study design process has included direct grantee involvement in qualitative instrument development. Prior to the beginning of MUSE data collection, each participating grantee will have reviewed the study design and agreed to facilitate the MUSE study in their local program to the extent appropriate and feasible. This agreement addresses the site visit component where interview data will be collected, and all participating sites will have endorsed their willingness to facilitate site visits, supporting a high response rate. We will tailor our interviewing plan based on each grantee’s unique staffing structure and availability.
As stated in section B1, we anticipate a staff survey response rate of 80 percent, a Home Visitor Rapid Reflect response rate of 80 percent, and an Implementation Log response rate of 85 percent. Grantee staff engagement in the study design process and familiarity with the study through participation in extensive virtual and in-person training is expected to support high response rates on staff surveys. In addition to stakeholder engagement and intensive training and support, the following will support high rates of completion for the Home Visitor Rapid Reflect: ability to complete Rapid Reflects immediately following home visits using the tablets, reminders ahead of the Rapid Reflect week, and mobilizing support and energy to complete Rapid Reflects for just one week out of each month. Higher response rates are expected for the Implementation Logs because they are only collected once per month and will be easily accessible through the MUSE web-based data collection system.
Caregivers will be recruited to participate in MUSE by home visitors or other program staff. Home visitors have already built rapport with the caregivers, so we anticipate higher response rates than if MUSE Team members were conducting caregiver recruitment. We have worked to keep the burden of the Caregiver Surveys and Caregiver Rapid Reflect low. We anticipate a response rate of 85 percent on the Caregiver Surveys, due to its short length, administration by home visitors, $10-15 incentives (See SSA, section A9 for additional information), and convenient timing during a regularly scheduled home visit. We anticipate a response rate of 85 percent on the Caregiver Rapid Reflect due to its short length, administration by home visitors, and convenient timing at the close of a regularly scheduled home visit.
Dealing with Nonresponse and Nonresponse Bias
While we have utilized community engagement strategies that should produce higher respondent cooperation rates, we know that some caregivers may have concerns about participating in research due to a lengthy history of abuses where researchers have conducted unethical research resulting in harm to tribal communities (Tribal Evaluation Workgroup, 2013). We also know that there is a fair amount of mistrust amongst tribal communities towards the federal government, and research has shown that AIAN research participants are significantly less likely to participate in a research study sponsored by the federal government (Buchwald et al., 2006). Offering incentives to caregivers for completing the Caregiver Surveys is an important demonstration of our respect for cultural protocols and acknowledgment of the historical misuse of research in tribal communities. In addition to an appropriate incentive plan, we have developed contextualized recruitment and informed consent strategies to increase our chance of obtaining a high response rate. These same strategies should assist us in mitigating non-response bias. By providing incentives for Caregiver Surveys and interviews we aim to offset the opportunity costs that may be disproportionally experienced by families who are not as financially disadvantaged as other potential respondents (Emanuel, 2004).
The MUSE Team will closely monitor response rates of all data collection activities. We anticipate some nonresponse to occur, given the complexity of data collection and the challenges of program implementation across the 17 participating grantees. Random nonresponse (for example, a home visitor periodically failing to complete a Rapid Reflect survey) will not be a problem, as long as it does not greatly exceed the response rate estimates we have delineated elsewhere in this document. We will be able to statistically adjust for random nonresponse using missing data analytic procedures. Non-random patterns of nonresponse, however, will be more problematic. If we observe such patterns during data collection (for example, if Rapid Reflect surveys are not being completed at home visits occurring on Mondays or by particular home visitors), we will quickly take corrective action, such as providing feedback to program managers, requiring booster trainings for staff, or instituting additional data collection reminders.
Maximizing Response Rates
The MUSE Team has worked with Tribal MIECHV grantee staff, Federal Tribal MIECHV staff, and Tribal MIECHV technical assistance providers to establish and embed a community-engaged approach into the design of the MUSE study. Throughout this process, grantee representatives have informed study design, instrument selection and development, as well as study protocol development. Significant changes to the study have been made based on these partnerships and we anticipate these changes to increase the local appropriateness and feasibility of instruments and study procedures. Response rates are expected to benefit from these processes.
The MUSE study will also offer four periodic honoraria over a two-year period to each participating grantee ranging from a total amount of $1,560 to $3,110 depending on the size of the grantee team and number of caregivers enrolled in home visiting. These honoraria recognize the extra effort required of grantees participating in MUSE and are intended to be used to celebrate grantee teams’ successes in MUSE data collection. These honoraria should contribute to higher data collection completion rates. Each Tribal MIECHV grantee will have regularly scheduled individualized phone calls with a MUSE Team member and their evaluation TA provider to check-in on MUSE study activities. In addition, the MUSE Team has developed protocols particular to each data collection activity to maximize response rates:
Secondary Data Analysis
We are asking grantees to share implementation plans that already exist and are part of the official grantee file. Implementation plans are covered under OMB control number 0970-0389, expiration date 8/31/2019.
Caregiver Surveys
Home visitors who have already built rapport with caregivers will most often be facilitating the surveys.
The surveys have been kept short to reduce the burden on caregivers.
Caregivers will receive an incentive to show our respect for cultural protocols and to minimize nonresponse bias.
We will develop a process to remind program coordinators/managers and home visitors of upcoming Caregiver Surveys. We will also send program coordinators/managers and home visitors multiple reminders (Attachment J) near the close of a data collection window for each data collection timepoint.
Rapid Reflect Self-Completed Home Visit Questionnaires
We will develop processes to remind program coordinators/managers and home visitors of upcoming Rapid Reflect reporting weeks and the caregivers who have consented to participate in MUSE and are eligible to complete a Rapid Reflect. We will monitor Rapid Reflect completion rates. If they appear to be unexpectedly low, we will partner with the evaluation TA provider and grantees to identify any challenges and develop plans to address them.
Data entry can be completed on a tablet. The Caregiver Rapid Reflects are conveniently collected at the end of a regularly scheduled home visit. Home visitors will complete the Home Visitor Rapid Reflects at the end of, or shortly after, the same home visits. We have limited the number of Caregiver Rapid Reflect questions to 5 to reduce the burden on caregivers. We have limited the Home Visitor Rapid Reflect questions to approximately 12 to minimize the burden on home visitors.
Staff Surveys
We have elected to survey staff once instead of multiple times to limit burden on staff associated with this study component. Additionally, staff are able to complete the survey at a time that is convenient for them and return to the survey to complete it, should they need to.
We will email staff individual survey reminders (Attachment G) after the initial survey consent request. To address potential non-responsiveness to the reminders, we will generate and send a report to program coordinators/managers after the initial survey consent request to inform them of staff who have not responded to the consent request. To protect the privacy of staff, we will not provide information regarding the content of a staff member’s response to the consent request, only whether a response has been received. Five weeks following the initial consent request, we will email staff a final individual survey reminder.
The training for grantee staff participating in MUSE will emphasize the value of hearing directly from staff about their experience with home visiting in the staff survey. The survey will also be administered at the beginning of MUSE data collection to take advantage of initial excitement around the MUSE study.
Qualitative Interviews
We will schedule the interviews at times that are convenient for the respondents.
We will provide incentives to caregivers to show our understanding of cultural protocols and to offset potential incidental costs as a result of participation (ex. child care, travel, etc.).
Implementation Logs
We designed the Implementation Logs to be completed by the program coordinators/managers to reduce the data collection burden on home visitors.
We will remind program coordinators/managers of upcoming Implementation Logs and clearly identify Implementation Logs that have not been completed. We will email program coordinators/managers weekly individual Implementation Log reminders (Attachment J) three times throughout the following month to identify Implementation Logs that have not been completed.
Administrative Program Data
We are requesting data that grantees already collect for MIECHV reporting, model reporting, or local quality assurance monitoring. We will work with local evaluators who are already familiar with these data. We will accept the data in a format that is least burdensome to the grantees while still meeting our needs.
Intensive technical assistance on compiling and submitting the administrative data elements will be available to grantees, and grantees will be able to quickly and easily upload files to a secure cloud-based file sharing website.
The MUSE Team conducted pre-testing for most quantitative instruments. For any given instrument, fewer than 10 people participated in the pretesting. Volunteers from participating grantees and some MUSE Technical Workgroup members reviewed the instruments for appropriateness for the population, understandability, and importance. This feedback was used to finalize the list of questions included and refine the wording of those questions. Once draft instruments were finalized, additional volunteers from participating grantees were asked to self-administer the instruments and record the amount of time it took to complete the instrument. Grantee staff also identified typos or question wording that needed further clarification. This feedback was used to inform the final burden estimates and final instrument documents.
The following staff from OPRE, James Bell Associates, Inc. and the University of Colorado led the design of the project and will lead all analysis efforts:
Aleta Meyer, ACF federal contracting officer’s representative (COR)
Nancy Rumbaugh Whitesell, Principal Investigator
Kate Lyon, Project Director
Michelle Sarche, Co-Investigator
These individuals will lead all data collection, supported by qualified junior staff and grantee personnel trained in data collection.
Additional staff consulted on methodological and other design considerations included:
Nancy Asdigian, quantitative methods specialist
Teresa Abrahamson-Richards, tribal review specialist
Melina Salvador, qualitative methods specialist
Expert external consultants and project stakeholders
Buchwald, D., V. Mendoza-Jenkins, C. Croy, H. McGough, M. Bezdek, and P. Spicer. 2006. Attitudes of urban American Indians and Alaska Natives regarding participation in research. J Gen Intern Med 21:648-51.
Emanuel, E. J. (2004), Ending Concerns About Undue Inducement. The Journal of Law, Medicine & Ethics, 32: 100-105.
Tribal Evaluation Workgroup. (2013, September). A roadmap for collaborative and effective evaluation in tribal communities. Washington, DC: Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Zeno, Tia (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |