MUSE OMB Part B PASSBACK 1.17.19 FINAL CLEAN

Multi-Site Implementation
Evaluation of Tribal Home
Visiting
OMB Information Collection Request
New Collection
Supporting Statement
Part B
Non-Substantive Change Request February 2019
Initial Request Approved December 2018
Submitted July 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, S.W.
Washington, D.C. 20201
Project Officer:
Aleta Meyer

Table of Contents
Part B. Collections of Information Employing Statistical Methods ............................. 1
B1. Respondent Universe and Sampling Methods ..................................................... 1
B2. Procedures for Collection of Information ............................................................ 6
B3. Methods to Maximize Response Rates and Deal with Nonresponse ................. 9
B4. Tests of Procedures or Methods to be Undertaken ........................................... 13
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data................................................................................................ 13
References .................................................................................................................... 14

Attachments:
Attachment E.

Caregiver Interviewee Selection Guide

Attachment F.

Caregiver Informational Flyer

Attachment G.

Staff Survey Invitation Email

Attachment H.

Staff Interview Invitation Email and Phone Script

Attachment I.

Caregiver Interview Invitation Phone Script

Attachment J.

Program Data Collection Reminder Emails

Attachment L .

Introduction to MUSE Video Script

i

List of Exhibits
Exhibit B.1. Data Collection Activities that Include Caregiver Data, by
Caregiver Group ......................................................................................................... 2
Exhibit B.2. Minimum Detectable Associations for Caregiver Surveys ................ 3
Exhibit B.3. Qualitative Interview Estimated Sample Size by MUSE Staff
Role .............................................................................................................................. 6

ii

Part B. Collections of Information Employing Statistical Methods
B1. Respondent Universe and Sampling Methods
Seventeen tribal communities received Tribal MIECHV Implementation and Expansion grants and
are eligible to participate in the Multi-Site Implementation Evaluation of Tribal Home Visiting
(MUSE). Based on enrollment data reported to ACF, we estimate that 1,441 caregivers receiving
home visiting could be recruited for MUSE over an 18-month participant recruitment period. We
will aim to successfully recruit and consent 75 percent of these caregivers for a total of 1,081
caregivers (see below for explanation of burden estimates in Supporting Statement A). Data
collection within the context of established home visitor-caregiver relationships and a service
delivery model that collects routine data for purposes outside of this study are expected to support
high consenting rates among caregivers. Ongoing communication, training, and engagement with
grantee program staff are also expected to result in a high-quality informed consent process and a
greater likelihood of positive communication about the study from grantee staff to caregivers. Due
to the relatively small universe of caregivers receiving services from Tribal MIECHV programs,
we do not plan to sample caregivers and will instead recruit all caregivers to participate in MUSE.
Our sampling strategy for each data collection method corresponds to the unique purpose of each
data collection activity.
There are two universes of caregivers receiving Tribal MIECHV services that will be recruited for
MUSE: (1) Group 1: caregivers who have started home visiting services before MUSE data
collection begins and (2) Group 2: caregivers who start home visiting services after MUSE data
collection begins. These groups only reflect when caregivers enrolled in home visiting services
relative to the beginning of the MUSE study; they do not reflect different assignment to groups for
the purposes of comparison or delivery of different types or amount of services. Caregivers in
Group 1 will be recruited into MUSE during the first two months of the study only. We estimate
that there will be 812 caregivers in Group 1 who will be recruited to participate in MUSE and 609
that consent to participate in the study (75%). Caregivers in Group 2 will be recruited into the
study over a 17-month period as they enroll in home visiting services. We estimate that there will
be 629 caregivers in Group 2 who will be recruited to participate in MUSE and 472 that will
consent to participate (75%).
In Supporting Statement A, we have estimated additional burden for each instrument beyond the
sample estimates in case grantees enroll more caregivers, complete more home visits, or retain
caregivers at a higher rate than we estimated based on available data. For our sample estimates we
project that 75% of eligible caregivers will consent to participate in MUSE; whereas for our burden
estimates, we project that 95% of caregivers will consent to participate. Exhibit B.1 shows in which
data collection activities caregiver information will be included, for each group.

1

Exhibit B.1. Data Collection Activities that Include Caregiver Data, by Caregiver Group
Caregivers
Group 1
Group 2

Caregiver
Enrollment
Form
X
X

Caregiver
Surveys

X

Rapid Reflect
Self-Completed
Questionnaires
X
X

Qualitative
Interviews

Administrative
Program Data

X (subsample)
X (subsample)

X (Some)
X (All)

We have chosen to collect a more complete set of data on caregivers in Group 2 for the following
reasons:
1. We will only be able to collect baseline data on key indicators from caregivers in Group 2
because they will have begun home visiting after MUSE data collection begins, whereas
Group 1 will have begun home visiting services prior to MUSE data collection.
2. We will be requesting additional administrative data from this group in order to correlate
these data with responses from the Caregiver Surveys.
3. We will be administering Rapid Reflect questionnaires to caregivers in Group 2 and Group
1 in order to get a more complete picture of what happens in home visits, including those
home visits provided to caregivers who have been receiving home visiting for more than a
year.
4. We are requesting that grantees provide a limited number of administrative data elements
from caregivers in Group 1 to help us interpret data collected with the Rapid Reflect for
caregivers in this group.
5. In addition to the reasons cited above, we felt that limiting data collection for caregivers in
Group 1 resulted in a more manageable level of burden for participating grantees.
Caregiver Enrollment Form and Caregiver Surveys
Home visitors will complete a Caregiver Enrollment Form (Instrument 1), entering basic
information that the program has previously collected on the caregiver who consented to
participate in MUSE. We estimate that 93 home visitors will complete Caregiver Enrollment
Forms for 1,081 caregivers. Only caregivers who start home visiting after MUSE data collection
has begun (Group 2) will be invited to take the Caregiver Surveys (Instruments 2 and 3). We have
chosen to restrict the universe for the Caregiver Surveys to caregivers in Group 2, because the
study design requires a baseline measure of items within the Caregiver Survey obtained when the
caregiver begins home visiting services. We anticipate that 472 caregivers from Group 2 will
consent to participate in MUSE and will be asked to take the Baseline Caregiver Survey. Of those
invited to take the survey, we estimate that 95 percent of the 472 consented will complete the
baseline survey (n=448).We estimate that 75 percent (n=354) of the 472 caregivers in Group 2
who consented to participate in MUSE will still be participating in home visiting at 6 months and
will be asked to take the 6-month survey. If 95% of the 354 caregivers complete the survey, we
will have a sample of 336 caregivers for the 6-month Caregiver Survey. Caregivers who begin
MUSE in the final five months of caregiver enrollment into the study will not become eligible to
take the 12-month follow-up survey before the end of the MUSE data collection period, which
leaves a total of 336 eligible caregivers at 12 months. We estimate that 60 percent (n=202) of these
336 caregivers will still be participating in home visiting at 12 months, and 95% of those caregivers
will agree to take the Caregiver Survey, which produces a sample size of 192 caregivers for the
12-month survey.

2

The first two aims of the MUSE study are descriptive; the third aim – Explore supports and
challenges to home visiting implementation in tribal communities – will involve estimating
statistical relationships among variables. To understand the sufficiency of our sample to address
this aim and detect significant associations among study variables, we conducted power analyses.
Those analyses are presented here because they are most relevant to the Caregiver Surveys. For
most other participant groups, the size of the sample is constrained by the small number of staff
in these roles across the 17 grantees, limiting power to detect associations to larger effects than
we would expect to find (i.e., correlations > .5). Analyses of those surveys thus will largely be
limited to descriptive goals under Aims 1 and 2.
Exhibit B.2 shows the minimum effects we will be able to detect within each wave of Caregiver
Survey data collection, at α <= .05 with power >= .80, as statistically significant bivariate
correlations among variables collected from caregivers. Power analyses indicate that the estimated
sample size at baseline (N=448) will be sufficient to detect a small effect (r=.131), and will remain
sufficient to detect relatively small effects at both the 6-month (N=336; r=.152) and 12-month
follow-ups (N=201, r=.195). In Supporting Statement A, we estimate higher retention rates at 6
months (85%) and 12 months (70%), to guarantee that we have requested enough burden for the
study. The higher retention rates and consenting rates used to calculate burden means we estimate
burden for an additional 117 caregivers at baseline, 145 caregivers at 6-months, and 90 caregivers
at 12-months. As indicated in Exhibit B.2, if these sample sizes are achieved, slightly smaller
associations will be detectible (r=.164 at 6-month follow-up and .188 at 12-month follow-up).
Exhibit B.2. Minimum Detectable Associations1 for Caregiver Surveys

Baseline
6-month follow-up
conservative estimate for power
estimate for burden
12-month follow-up
conservative estimate for power
estimate for burden
1

Estimated
Minimum
Estimated
No.
Projected
Detectable
Retention
Eligible
Response
No. of
Association
Rate (%) Participants Rate (%) Responses
n/a
472
95%
448
.131
75%
85%

354
301

95%
95%

336
286

.152
.164

60%
75%

212
226

95%
95%

201
215

.195
.188

Smallest bivariate correlation coefficients detectable at α = .05 and power = .80

Examples of relationships we will examine in the Caregiver Survey data, based on the MUSE
conceptual model co-created with Tribal Home Visiting grantees and other stakeholders include
associations of variables within each time point, such as perceptions of the relationship with the
home visitor and satisfaction with home visiting, with parenting self-efficacy, and with social
support. We will also examine associations across time, such as perceptions of the home visitor
relationship at 6-months with parenting self-efficacy and social support at 12-months. The fact
that we will have the sensitivity to detect even small associations among these variables, both
within and across time, will be essential to the success of MUSE in providing an unprecedented

3

look at how home visiting is being implemented in tribal communities and how it is experienced
by caregivers served by those programs.
Rapid Reflect Self-Completed Home Visit Questionnaires
Home visitors and caregivers both (Group 1 and Group 2) will complete a Rapid Reflect SelfCompleted Questionnaire (Instruments 4 and 5) for every home visit conducted during one week
out of each month. This is an effective sampling rate of 23% of home visits conducted. The
assigned Rapid Reflect week will rotate to ensure that we capture home visits happening with
families on different home visit schedules. This sampling strategy will allow us to collect data on
some home visits for most families participating and should result in data on a sample of home
visits conducted with a wide variety of families, throughout the course of home visiting, and
throughout the calendar year. This sampling strategy should result in a representative sample of
home visits with approximately 919 caregivers completing at least one Rapid Reflect. Because the
selected weeks will rotate, not all caregivers who consent to participate in MUSE will be asked to
complete a Rapid Reflect; some will not have a visit that falls during the selected weeks. All home
visitors will complete Rapid Reflect questionnaires, along with additional staff who are completing
home visits on a temporary basis (N=93). As with the other instruments, we have estimated
additional burden in case response rates are higher than anticipated or we experience a fewer
number of home visits that are ineligible for the Rapid Reflect than we expect to occur. We have
estimated burden for an additional 217 caregivers to complete the Rapid Reflect.
Staff Surveys
Staff from Tribal MIECHV grantees participating in MUSE will be invited to take the staff survey.
Managers will also be invited to take an additional survey—the Program Implementation Survey—
and complete that survey together, if needed. Staff that fit into the following four MUSE staff roles
will be surveyed: program directors (Instrument 8), program coordinators/managers (Instruments
7 and 10), home visitors (Instrument 6), and local evaluators (Instrument 9). Prior to recruiting
staff for the survey, the MUSE Team will clearly define each MUSE staff role and work with
grantees to determine which staff members fit within each MUSE staff role.
Because of the expected universe size, we will invite all Tribal MIECHV grantee staff and local
evaluators who fit into the four MUSE staff roles and whose contact information is provided to the
MUSE Team to participate. To estimate the number of individuals that will be recruited to take
the staff survey, we reviewed information gathered by ACF on the number of full time equivalent
positions (FTE) funded in each category at each participating grantee. We also reviewed data on
staff attrition rates. The MUSE Team will send survey invitations to approximately: 17 program
directors, 17 program coordinators/managers, 70 home visitors, and 25 local evaluators. We will
send the Program Implementation Survey to approximately 17 program coordinators/mangers and
they will be instructed to complete the survey with their program director. We estimate that we
will invite a total of 129 program staff and evaluators to take the surveys, but we estimate burden
for up to 153 staff and evaluators to account for any unexpected staff turnover. We anticipate an
85 percent response rate for the staff surveys.

4

Qualitative Interviews (Instruments 11-14)
The qualitative component of MUSE uses a purposive sampling of multiple research participant
groups including managerial staff (program coordinators/managers and directors), evaluators,
home visitors, and caregivers receiving home visiting services in both Groups 1 and 2.
Program Staff Sampling
Each Tribal MIECHV grantee has a unique staffing structure, which typically includes a small
leadership team (between 1-2 administrators). Additionally, each grantee includes evaluation staff,
but the structure of the evaluation team varies widely across programs (i.e. evaluator on staff, subcontractor). Given the small size and unique structure of leadership and evaluation teams across
these grantees, MUSE will use a universal approach to sampling directors, coordinators/managers
and evaluators for interviews. These individuals will be asked to reflect on program level
phenomena, such as staff supports, program design, and institutional resources.
Due to the highly variable nature of the home visitor teams across grantees, we will select home
visitors based on some basic criteria and then use emergent sampling, incorporating quantitative
data when possible.
•
•

For grantees that employ two home visitors or less, we will interview each home visitor.
For grantees with more than two home visitors, we will select possible interviewees based
on the following criteria: home visitor tenure, FTE level, and case load. We will prioritize
interviewing home visitors who have worked longer than 6 months, have full FTE, and
who maintain full caseloads. Using these criteria will help ensure that the home visitors
who are asked to reflect on their experience can draw from ample evidence across multiple
families over a longer period of time. Additionally, we will consider the gender of the home
visitor as well as his/her tribal affiliation during the selection process to ensure that we
achieve maximum variation within these discrete sets of characteristics. Depending on the
results and timing of initial quantitative assessments, it may be possible to incorporate
stratified purposive sampling techniques.

Caregiver Sampling
Caregiver interviewees will be selected in collaboration with grantee personnel. We will provide
a Caregiver Interviewee Selection Guide (Attachment E) to assist grantees in identifying potential
caregiver interviewees, which will include a list of the various characteristics and any
prioritizations. Caregiver interviewees will be individuals who: serve as a primary caregiver (e.g.,
mother, father, grandparent) and actively participate as a primary caregiver in home visits. When
appropriate and feasible, we will prioritize selection of male caregivers, first-time parents, and
grandparent caregivers as well as consider tribal affiliation, age, and number of children to help
ensure maximum variation across caregiver types. As with sampling home visitors, it may be
possible to establish rigorous strata through quantitative assessments to conduct stratified
purposive sampling techniques for caregiver interviewees as well.
Approximately 157 participants will be included in the interview component across 17grantees.
We anticipate a response rate of 90% for the qualitative component. Exhibit B.2 provides the
estimated sample size by MUSE staff role.

5

Exhibit B.3. Qualitative Interview Estimated Sample Size by MUSE Staff Role
MUSE Staff Role

Sample Size

Home visitors

42

Managerial staff (program coordinators/managers or directors)

34 (2 per site)

Local evaluators (lead evaluators, data support staff)

30 (1-3 per site)

Caregivers participating in home visiting

51 (3 per site)

Total

157

Implementation Logs
Implementation Log data (Instrument 15) will be collected from all 17 grantees participating in
MUSE. A designated staff person, most likely the program coordinator/manager, at each grantee
will complete the Implementation Logs each month over the two-year data collection period. The
logs collect information on staff hires and departure, training and supervision received by all home
visitors at each grantee, as well as family group activities. We expect that 85 percent of the monthly
Implementation Logs will be submitted.
Administrative Program Data
Administrative Program Data (Instrument 16) will be submitted by all 17 grantees participating in
MUSE. Some Administrative Program Data will be submitted for all caregivers who consent to
participate in MUSE (Groups 1 and 2). The MUSE consent forms (See Attachment A) seek consent
for both retrospective and prospective data collection of Administrative Program Data. We
estimate that 1,081 caregivers will consent to participate in MUSE and agree to share some of the
data they provided to the home visiting program with the MUSE study. The MUSE Team will ask
grantees to provide more extensive demographic and service utilization data from an estimated
472 caregivers who start home visiting services after MUSE begins and consent to participate in
the study (Group 2).

B2. Procedures for Collection of Information
Caregiver Recruitment, Consent, and Enrollment into MUSE
Caregivers who are enrolled in one of the 17 Tribal MIECHV programs will be invited to
participate in the study by their home visitor. As described above, caregivers will be invited to
participate in each data collection activity on the basis of whether they started receiving home
visiting services before the beginning of MUSE data collection (Group 1) or started receiving
home visiting services after MUSE data collection began (Group 2). See Exhibit B.1 above for
data collection activities involving caregiver data by caregiver group.
All recruitment will take place through one-on-one, in-person communication. Prior to formally
inviting caregivers to participate, home visitors will introduce caregivers to the study during a
regularly scheduled home visit with the aid of a video (Attachment L) and informational flyer
(Attachment F). During the following home visit, home visitors will invite caregivers to participate
in the study, explain all data collection activities, and lead them through the informed consent

6

process. The MUSE study has obtained an IRB-approved waiver of parental consent for minor
caregivers. Home visitors will use a tablet to complete an electronic consent form (Attachment A).
Once a caregiver has consented to participate in MUSE, the home visitor will complete the
Caregiver Enrollment Form. The Caregiver Enrollment Form contains basic demographic
information about the caregiver, already collected by the program. This information is used to
track data collection for each caregiver within the MUSE data system.
Caregiver Surveys
Surveys of Group 2 caregivers (i.e. caregivers who start home visiting services after MUSE data
collection begins) will be administered by a member of the home visiting program staff during
regularly scheduled home visits at baseline, 6-month follow-up, and 12-month follow-up. The
Caregiver Survey will typically be administered at the location the caregiver has chosen for their
home visit (most often, this location is the caregiver's home). Home visitors will provide a tablet
to the caregiver with the surveys pre-loaded and answer any of the caregiver’s questions. Home
visitors will not ask caregivers the survey questions, they will be trained in protocols to ensure that
caregivers are given privacy to complete surveys, and all staff collecting these data will be certified
in human subjects’ research protections. In addition, surveys will be collected on tablets using
secure software that ensures caregiver responses cannot be accessed by home visiting program
staff. Caregivers with limited literacy or visual impairments can choose to complete the surveys
over the phone with a member of the MUSE study team. MUSE informed consent and Baseline
Caregiver Surveys will take place after caregivers complete the grantee’s local program enrollment
and initial data collection processes. The MUSE study team will work with grantees to finalize
exact MUSE data collection and consent timeframes.
Rapid Reflect Self-Completed Home Visit Questionnaires
Home visitors and caregivers will complete Rapid Reflect short surveys for every home visit
conducted with caregivers who have consented to participate in MUSE during one designated
week per month. During the designated Rapid Reflect week, all participating caregivers will be
asked to take a five-question survey about the home visit that was just completed. Home visitors
will be trained to hand caregivers a MUSE data collection tablet and provide privacy for the
caregiver to independently answer the questions on the tablet. Caregivers with limited literacy or
visual impairments can choose to hear the questions by selecting to play an audio recording of the
questions and answer options.
Immediately following the home visit, home visitors will complete a Rapid Reflect survey about
the same visit using the tablet. Throughout the twenty-two-month Rapid Reflect data collection
period, the assigned Rapid Reflect week will rotate to ensure that MUSE captures home visits
happening with caregivers on different home visit schedules. Home visitors will be intensively
trained to administer the caregiver Rapid Reflect and to complete the home visitor Rapid Reflect
during MUSE data collection training.
Staff Surveys
Surveys of home visiting program staff-- including home visitors, program coordinators/managers,
program directors, and local evaluators-- will be administered electronically through a web-based
data system once, during the first month of the study, and on a rolling basis for new hires. Each
staff person will receive via email (Attachment G) a survey tailored to their role. Grantee staff in

7

a managerial role will also be invited to complete a second survey focused on program
implementation via email (Attachment G) as a management team. Informed consent will be
obtained electronically before the respondent proceeds to the survey questions (Attachment A).
The MUSE Team will send four email reminders (Attachment G) and will consider instituting
telephone reminders if necessary. While some of the staff surveys are lengthy (30-70 minutes),
respondents can complete the survey at a time that is convenient for them, including over multiple
sittings, if needed. The MUSE Team elected to administer staff surveys once instead of multiple
times due to the burden associated with responding to the surveys.
Qualitative Interviews
In-person qualitative interviews with home visiting program staff and enrolled caregivers will be
conducted by members of the research team during a single site visit for each grantee. The
investigator and/or other members of the MUSE Team will partner with an individual at each
grantee to reserve a room or rooms at the grantee program office to conduct the interviews.
Approximately 2 home visitors, 1 program director, 1 program coordinator/manager, and 1-3 local
evaluators will be invited to participate in a qualitative interview at each grantee. Specific
interviewees will be determined on a site-by-site basis depending on local staffing structure. In the
case of grantees with more than 2 home visitors, purposive sampling criteria such as tenure, FTE,
caseload, and gender will be considered in selecting potential interview participants. The MUSE
Team will directly contact potential staff interview participants by phone or email (Attachment H),
provide basic information about the study and the interview specifically, and invite the individual
to participate in an interview. Individuals who agree to move forward will be scheduled for an
interview. Consent will be obtained face-to-face by the MUSE Team.
Three caregivers at each grantee who have consented to participate in MUSE, will be invited to
participate in qualitative interviews. Potential participants will be identified based on purposive
sampling criteria to ensure representativeness and robust findings. The MUSE Team will work
with local grantee staff to identify caregivers who fit the sampling criteria and may be interested
in participating in an interview. Once a caregiver has been identified, local grantee staff will
contact him or her to provide the study flyer (if they have not already received it) and ask for initial
permission to provide their name and contact information to the MUSE study team. If the caregiver
agrees, a MUSE Team member will contact them directly to review the study (Attachment I) and,
if they agree to an interview, schedule them for an interview during the site visit. On a case-bycase basis, the MUSE Team member(s) conducting an interview with a caregiver may elect to
travel to the caregiver's home to conduct the interview to allow flexibility or accommodate
participant preferences. Consent to participate in the interview will be obtained face-to-face by a
member of the study team (Attachment A).
Implementation Logs
Monthly Implementation Logs will be completed by a grantee’s program coordinator/manager to
report data on staff hires and departures, training, individual and group supervision, and family
group activities. A web-based data system will allow grantees to access data entry for each
component of the Implementation Logs. Program coordinators/managers will be directly oriented
to the Implementation Logs during MUSE data collection training.

8

Administrative Program Data
Tribal MIECHV grantees collect a substantial amount of data for Federal reporting, model
reporting, and local quality assurance processes that can also be used to answer the MUSE study
questions. To reduce burden and avoid collecting the same data twice from caregivers receiving
home visiting, the MUSE study design includes analyses of existing Administrative Program Data
collected and maintained by Tribal MIECHV grantees.
Home visitors will be trained to administer informed consent to caregivers receiving home visiting
services. During the informed consent process, home visitors will explain the data collection
activities that caregivers are being asked to participate in, and that participating in MUSE involves
sharing some of the data their program collects about them, their child, and the services they
received with the MUSE study (i.e., Administrative Program Data). Grantees participating in
MUSE will only share Administrative Program Data about caregivers who have consented to
participate in MUSE.
Four types of Administrative Program Data will be reported for the MUSE study: participant
demographics, screener data, home visit participation data, and group activity participation data
(Instrument 16). Home visitors already collect this information from caregivers and will continue
to the collect the information following their local data collection protocols. Grantees will be asked
to submit individual level data to support analyses that examine associations between participant
characteristics and implementation factors including: participation in home visiting, home visit
content, relationship with their home visitor and satisfaction with home visiting. Grantees will
upload the data to a secure, password protected cloud-based data storage website. Grantees will
submit Administrative Program Data semi-annually for a total of four times.

B3. Methods to Maximize Response Rates and Deal with Nonresponse
Expected Response Rates
For the past 18 months, the MUSE Team has conducted extensive stakeholder engagement
activities with all 17 Tribal MIECHV grantees who have agreed to participate in MUSE as well as
other stakeholders. We have attempted to design a study that will answer questions of interest to
participating grantees, utilize existing data where possible, and involve a feasible amount of data
collection burden. For these reasons, we believe we can expect a relatively high participation rate
from grantees and the home visiting staff at each grantee.
Based on the success of grantee recruitment into MUSE discussed in section B1, we anticipate
response rates above 90 percent for the qualitative interviews conducted during each grantee’s site
visit. Ongoing partnership within the community-engaged MUSE study design process has
included direct grantee involvement in qualitative instrument development. Prior to the beginning
of MUSE data collection, each participating grantee will have reviewed the study design and
agreed to facilitate the MUSE study in their local program to the extent appropriate and feasible.
This agreement addresses the site visit component where interview data will be collected, and all
participating sites will have endorsed their willingness to facilitate site visits, supporting a high
response rate. We will tailor our interviewing plan based on each grantee’s unique staffing
structure and availability.

9

As stated in section B1, we anticipate a staff survey response rate of 80 percent, a Home Visitor
Rapid Reflect response rate of 80 percent, and an Implementation Log response rate of 85 percent.
Grantee staff engagement in the study design process and familiarity with the study through
participation in extensive virtual and in-person training is expected to support high response rates
on staff surveys. In addition to stakeholder engagement and intensive training and support, the
following will support high rates of completion for the Home Visitor Rapid Reflect: ability to
complete Rapid Reflects immediately following home visits using the tablets, reminders ahead of
the Rapid Reflect week, and mobilizing support and energy to complete Rapid Reflects for just
one week out of each month. Higher response rates are expected for the Implementation Logs
because they are only collected once per month and will be easily accessible through the MUSE
web-based data collection system.
Caregivers will be recruited to participate in MUSE by home visitors or other program staff. Home
visitors have already built rapport with the caregivers, so we anticipate higher response rates than
if MUSE Team members were conducting caregiver recruitment. We have worked to keep the
burden of the Caregiver Surveys and Caregiver Rapid Reflect low. We anticipate a response rate
of 85 percent on the Caregiver Surveys, due to its short length, administration by home visitors,
$10-15 incentives (See SSA, section A9 for additional information), and convenient timing during
a regularly scheduled home visit. We anticipate a response rate of 85 percent on the Caregiver
Rapid Reflect due to its short length, administration by home visitors, and convenient timing at
the close of a regularly scheduled home visit.
Dealing with Nonresponse and Nonresponse Bias
While we have utilized community engagement strategies that should produce higher respondent
cooperation rates, we know that some caregivers may have concerns about participating in research
due to a lengthy history of abuses where researchers have conducted unethical research resulting
in harm to tribal communities (Tribal Evaluation Workgroup, 2013). We also know that there is a
fair amount of mistrust amongst tribal communities towards the federal government, and research
has shown that AIAN research participants are significantly less likely to participate in a research
study sponsored by the federal government (Buchwald et al., 2006). Offering incentives to
caregivers for completing the Caregiver Surveys is an important demonstration of our respect for
cultural protocols and acknowledgment of the historical misuse of research in tribal communities.
In addition to an appropriate incentive plan, we have developed contextualized recruitment and
informed consent strategies to increase our chance of obtaining a high response rate. These same
strategies should assist us in mitigating non-response bias. By providing incentives for Caregiver
Surveys and interviews we aim to offset the opportunity costs that may be disproportionally
experienced by families who are not as financially disadvantaged as other potential respondents
(Emanuel, 2004).
The MUSE Team will closely monitor response rates of all data collection activities. We anticipate
some nonresponse to occur, given the complexity of data collection and the challenges of program
implementation across the 17 participating grantees. Random nonresponse (for example, a home
visitor periodically failing to complete a Rapid Reflect survey) will not be a problem, as long as it
does not greatly exceed the response rate estimates we have delineated elsewhere in this document.
We will be able to statistically adjust for random nonresponse using missing data analytic

10

procedures. Non-random patterns of nonresponse, however, will be more problematic. If we
observe such patterns during data collection (for example, if Rapid Reflect surveys are not being
completed at home visits occurring on Mondays or by particular home visitors), we will quickly
take corrective action, such as providing feedback to program managers, requiring booster
trainings for staff, or instituting additional data collection reminders.
Maximizing Response Rates
The MUSE Team has worked with Tribal MIECHV grantee staff, Federal Tribal MIECHV staff,
and Tribal MIECHV technical assistance providers to establish and embed a community-engaged
approach into the design of the MUSE study. Throughout this process, grantee representatives
have informed study design, instrument selection and development, as well as study protocol
development. Significant changes to the study have been made based on these partnerships and we
anticipate these changes to increase the local appropriateness and feasibility of instruments and
study procedures. Response rates are expected to benefit from these processes.
The MUSE study will also offer four periodic honoraria over a two-year period to each
participating grantee ranging from a total amount of $1,560 to $3,110 depending on the size of the
grantee team and number of caregivers enrolled in home visiting. These honoraria recognize the
extra effort required of grantees participating in MUSE and are intended to be used to celebrate
grantee teams’ successes in MUSE data collection. These honoraria should contribute to higher
data collection completion rates. Each Tribal MIECHV grantee will have regularly scheduled
individualized phone calls with a MUSE Team member and their evaluation TA provider to checkin on MUSE study activities. In addition, the MUSE Team has developed protocols particular to
each data collection activity to maximize response rates:
1. Secondary Data Analysis
a. We are asking grantees to share implementation plans that already exist and are part
of the official grantee file. Implementation plans are covered under OMB control
number 0970-0389, expiration date 8/31/2019.
2. Caregiver Surveys
a. Home visitors who have already built rapport with caregivers will most often be
facilitating the surveys.
b. The surveys have been kept short to reduce the burden on caregivers.
c. Caregivers will receive an incentive to show our respect for cultural protocols and
to minimize nonresponse bias.
d. We will develop a process to remind program coordinators/managers and home
visitors of upcoming Caregiver Surveys. We will also send program
coordinators/managers and home visitors multiple reminders (Attachment J) near
the close of a data collection window for each data collection timepoint.
3. Rapid Reflect Self-Completed Home Visit Questionnaires
a. We will develop processes to remind program coordinators/managers and home
visitors of upcoming Rapid Reflect reporting weeks and the caregivers who have
consented to participate in MUSE and are eligible to complete a Rapid Reflect. We
will monitor Rapid Reflect completion rates. If they appear to be unexpectedly low,

11

we will partner with the evaluation TA provider and grantees to identify any
challenges and develop plans to address them.
b. Data entry can be completed on a tablet. The Caregiver Rapid Reflects are
conveniently collected at the end of a regularly scheduled home visit. Home visitors
will complete the Home Visitor Rapid Reflects at the end of, or shortly after, the
same home visits. We have limited the number of Caregiver Rapid Reflect
questions to 5 to reduce the burden on caregivers. We have limited the Home
Visitor Rapid Reflect questions to approximately 12 to minimize the burden on
home visitors.
4. Staff Surveys
a. We have elected to survey staff once instead of multiple times to limit burden on
staff associated with this study component. Additionally, staff are able to complete
the survey at a time that is convenient for them and return to the survey to complete
it, should they need to.
b. We will email staff individual survey reminders (Attachment G) after the initial
survey consent request. To address potential non-responsiveness to the reminders,
we will generate and send a report to program coordinators/managers after the
initial survey consent request to inform them of staff who have not responded to
the consent request. To protect the privacy of staff, we will not provide information
regarding the content of a staff member’s response to the consent request, only
whether a response has been received. Five weeks following the initial consent
request, we will email staff a final individual survey reminder.
c. The training for grantee staff participating in MUSE will emphasize the value of
hearing directly from staff about their experience with home visiting in the staff
survey. The survey will also be administered at the beginning of MUSE data
collection to take advantage of initial excitement around the MUSE study.
5. Qualitative Interviews
a. We will schedule the interviews at times that are convenient for the respondents.
b. We will provide incentives to caregivers to show our understanding of cultural
protocols and to offset potential incidental costs as a result of participation (ex.
child care, travel, etc.).
6. Implementation Logs
a. We designed the Implementation Logs to be completed by the program
coordinators/managers to reduce the data collection burden on home visitors.
b. We will remind program coordinators/managers of upcoming Implementation Logs
and clearly identify Implementation Logs that have not been completed. We will
email program coordinators/managers weekly individual Implementation Log
reminders (Attachment J) three times throughout the following month to identify
Implementation Logs that have not been completed.

12

7. Administrative Program Data
a. We are requesting data that grantees already collect for MIECHV reporting, model
reporting, or local quality assurance monitoring. We will work with local evaluators
who are already familiar with these data. We will accept the data in a format that is
least burdensome to the grantees while still meeting our needs.
b. Intensive technical assistance on compiling and submitting the administrative data
elements will be available to grantees, and grantees will be able to quickly and
easily upload files to a secure cloud-based file sharing website.

B4. Tests of Procedures or Methods to be Undertaken
The MUSE Team conducted pre-testing for most quantitative instruments. For any given
instrument, fewer than 10 people participated in the pretesting. Volunteers from participating
grantees and some MUSE Technical Workgroup members reviewed the instruments for
appropriateness for the population, understandability, and importance. This feedback was used to
finalize the list of questions included and refine the wording of those questions. Once draft
instruments were finalized, additional volunteers from participating grantees were asked to selfadminister the instruments and record the amount of time it took to complete the instrument.
Grantee staff also identified typos or question wording that needed further clarification. This
feedback was used to inform the final burden estimates and final instrument documents.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data
The following staff from OPRE, James Bell Associates, Inc. and the University of Colorado led
the design of the project and will lead all analysis efforts:
• Aleta Meyer, ACF federal contracting officer’s representative (COR)
• Nancy Rumbaugh Whitesell, Principal Investigator
• Kate Lyon, Project Director
• Michelle Sarche, Co-Investigator
These individuals will lead all data collection, supported by qualified junior staff and grantee
personnel trained in data collection.
Additional staff consulted on methodological and other design considerations included:
• Nancy Asdigian, quantitative methods specialist
• Teresa Abrahamson-Richards, tribal review specialist
• Melina Salvador, qualitative methods specialist
• Expert external consultants and project stakeholders

13

References
Buchwald, D., V. Mendoza-Jenkins, C. Croy, H. McGough, M. Bezdek, and P. Spicer. 2006.
Attitudes of urban American Indians and Alaska Natives regarding participation in research. J
Gen Intern Med 21:648-51.
Emanuel, E. J. (2004), Ending Concerns About Undue Inducement. The Journal of Law, Medicine &
Ethics, 32: 100-105.
Tribal Evaluation Workgroup. (2013, September). A roadmap for collaborative and effective
evaluation in tribal communities. Washington, DC: Children’s Bureau, Administration for
Children and Families, U.S. Department of Health and Human Services.

14