NonSub Change - Modifications to Incentives

MEMORANDUM

To: Josh Brammer

Office of Information and Regulatory Affairs

Office of Management and Budget (OMB)

From: Samantha L. Illangasekare

Office of Planning, Research, and Evaluation

Administration for Children and Families (ACF)

Date: May 12, 2020

Subject: Request for changes to incentive amount for Strengthening Relationship Education and Marriage Services (OMB Control Number 0970-0481)

Background

Type of Request: Non-substantive change to incentive amount for the second follow-up surveys for two of the four sites in Strengthening Relationship Education and Marriage Services (STREAMS).

Study Features Salient to Request: The purpose of STREAMS (OMB control number 0970-0481) is to identify and evaluate strategies for improving the delivery and effectiveness of healthy marriage and relationship education (HMRE) programs. The project includes a process study and a randomized controlled trial (RCT) of HMRE interventions in four sites: (1) MotherWise in Denver, CO, (2) Family Workforce Centers of America (FWCA) in St. Louis, MO, (3) The Parenting Center in Fort Worth, TX, and (4) More Than Conquerors, Inc. (MTCI) in Atlanta, GA. MotherWise, FWCA, and The Parenting Center provide services for adults. MTCI provides services to high school students. The information from this study is expected to be of use to policymakers, grantees, technical assistance providers, and researchers. The information could inform policy making, program improvement, decision-making, program design, funding decisions, and the HMRE research field.

In each site, people eligible for HMRE services were randomized into one or more treatment groups who were offered services and a control group who were not offered services. In MTCI, students were randomly assigned into two different treatment groups that differ in how the program was administered, as well as a control group.

A follow-up survey at 12 months after random assignment was administered in all sites; these surveys will be completed by May 2020. A second follow-up survey at 30 months after random assignment is being conducted for the MotherWise program and at 24-36 months after random assignment for the MTCI program.

The second follow-up survey for MotherWise study participants began in May 2019 and will continue through September 2021. Study participants are released for this second follow-up survey on a rolling basis based on their date of study enrollment. Study participants can respond to the survey either by phone or online. Before March 2020, survey field staff attempted to locate nonresponders in-person and offer them the use of a cell phone to complete the survey. In March 2020, all field work stopped because of COVID-19.

Study participants at MTCI fall into two cohorts. The second follow-up survey for the first cohort of MTCI study participants began in October 2019; the survey for the second cohort began in March 2020. The second follow-up survey at MTCI will continue through December 2020. Surveys for the first cohort were attempted first in schools. If the surveys were not completed in schools, study participants were asked to conduct the surveys by phone or online. Before March 2020, survey field staff would attempt to locate first cohort survey nonresponders and let them use a cell phone to complete the survey. Because of the COVID-19 restrictions and school closures, survey staff were unable to administer the survey to the second cohort students in the schools. Additionally, field locating efforts had to be discontinued in March 2020 because of COVID-19 restrictions.

Progress to Date: First follow-up data collection for all four sites will be completed by May 2020.

Response rates to the second follow-up survey in MotherWise and MTCI were already lower than anticipated and, due to the inability to conduct field work because of COVID-19, we are increasingly concerned about the anticipated overall response rate and the treatment-control difference in response rates to the second follow-up survey in both sites.

MotherWise

The response rate to the second follow-up survey in MotherWise is low currently. The overall response rate for the second-follow-up survey in MotherWise for the 200 cases released before December 2019 is currently 64.0 percent and the difference in response rates between the treatment and control group is 3.8 percentage points (Table 1).

Table 1: Response rates for the second follow-up survey in MotherWise

		Response rates
Total number of cases releaseda	Overall response ratesb	Treatment group	Control group	Treatment-control difference
Including responses occurring because of field work
200	64.0%	65.4%	61.6%	3.8 percentage points
Excluding responses occurring because of field work
200	50.5%	54.3%	43.8%	10.5 percentage points

^a Includes sample members who were released before December 2019 and were worked in the field until March 2020. A further 212 cases were released between December 2019 and April 2020. An additional 557 cases are still waiting to be released.

^b Target response rate is 80 percent.

Without the ability to do field work because of the COVID-19 restrictions, the response rate level is likely to be lower and the treatment-control differential in response rates is likely to be higher for the 769 cases released between December 2019 and May 2021 (when the final MotherWise cases will be released). Without the ability to conduct field work, we expect that the response rate for the later releases will be similar to those obtained for the earlier releases in which no in-person contact by survey field staff was made. The overall response rates for the earlier cohorts without the field work would have been 50.5 percent with a 10.5 percentage point differential in the response rates between treatment and control groups (Table 1). The overall response rate for the cases released between December 2019 and April 2020 is currently 35.8 percent with a 6.7 percentage point difference between the response rates in the treatment and control groups.

MTCI

Similarly, the response rate for the MTCI second follow-up survey is lower than anticipated and the differential between the research groups is higher than anticipated. For the first cohort released in October 2019, the overall response rate is 63.0 percent and the treatment-control response rate differential is 6.9 percentage points (Table 2). The response rate for the second cohort, for whom school administration and field location was not feasible due to COVID-19 restrictions, is likely to be lower. For the first cohort study participants who completed the survey by telephone or on-line without in-person contact by survey field staff, the response rate is only 34.7 percent and the differential between treatment and control groups is 5.0 percentage points (Table 2). The overall response rate for cohort 2 (762 cases) released in March 2020 is currently 44.0 percent with a 2.9 percentage point difference in response rates between research groups.

Table 2: Response rates for the second follow-up survey in MTCI

		Response rates
Number of cases in cohort 1	Overall response ratesb	T1 group	T2 group	Control group	Largest difference between groups
Including responses occurring because of field work
1,100	63.0%	64.8%	58.9%	65.8%	6.9 percentage points
Excluding responses occurring because of field work
1,100	34.7%	37.8%	32.8%	33.9%	5.0 percentage points

^a All cohort 1 sample members were released in October 2019. An additional 762 cases were release in cohort 2 in March 2020.

^b Target response rate is 80 percent.

Previous Terms of Clearance: The following tokens of appreciation for both the first and second follow-up surveys were approved by OMB under Control Number 0970-0481 (April 2019):

• Participants responding to the MotherWise survey: $25 gift card

• Participants responding to the MTCI survey: In school, $15 gift card; at home, $20 gift card

Time Sensitivity: This request has high time sensitivity. To improve our chances of achieving our target response rate for the full sample and to avoid bias from differential response rates across research groups, it is essential to increase the response rate among the cases currently in the field.

Request Overview

ACF requests a non-substantive modification of the survey incentive amount for the second follow-up survey for two sites.

Due to the inability to conduct field work because of COVID-19, we are concerned about the anticipated overall response rate and the treatment-control difference in response rates to the second follow-up survey in both MotherWise and MTCI. We are concerned that the inability to conduct field work in both these sites will limit the ability to increase the response rates for newly released cases as well as the cases that have already been released. We will also be unable to use targeted field locating efforts to help balance any differences across research groups in response rates that may emerge.

These changes to the survey incentive amounts are proposed as part of OPRE’s ongoing efforts to address lower-than-expected response rates and potential non-response bias caused by treatment-control differentials in response rates in these sites. ACF is concerned that without these interventions, data will produce biased estimates of program impacts and will underrepresent participants in key analytic groups.

Mitigation to Date

From the beginning of survey fielding, ACF has actively monitored survey production and response rates. To date, the contract team has taken the following steps to improve respondent location and encourage participation:

Initial Design for MotherWise

• The follow-up surveys were offered to respondents in two modes: web and telephone, allowing study participants to complete the surveys how and when it is convenient for them.

• Survey outreach used multiple channels and multiple contact attempts. Prior to attempted telephone contact, all participants were notified about the data collection through an advance letter. Participants who provided an email address were sent email messages providing login information for the survey web form. Participants with an email address or assent to text were sent two email messages or two text messages prior to telephone contact attempts.

• Nonresponding participants received multiple telephone calls over several weeks on different days and at different times asking them to complete the survey.

• Non-working telephone numbers or those that did not result in contact were subject to locating, through contact with individuals provided by the participant at the time of study enrollment.

• A specialized letter was sent to participants who made non-adamant refusals, followed by contact from a specially-trained and experienced telephone interviewer.

• If the participants could not be contacted by telephone, field locators made in-person contact with them and provided them with a cell phone to complete the survey with a telephone interviewer.

Initial Design for MTCI

• Survey staff worked closely with participating schools to coordinate data collection. The follow-up surveys were administered to students in a classroom setting whenever possible. Opportunities to administer the survey were scheduled on different days to account for absences. Participants who did not complete in school were contacted at home via telephone.

• Survey outreach used multiple channels and multiple contact attempts. Prior to attempted telephone contact, all participants were notified about the data collection through an advance letter. Participants who provided an email address were sent email messages asking them to call a toll free number.

• Nonresponding participants received multiple telephone calls over several weeks on different days and at different times asking them to complete the survey.

• Non-working telephone numbers or those that did not result in contact were subject to locating.

• A specialized letter was sent to participants who made non-adamant refusals, followed by contact from a specially-trained and experienced telephone interviewer.

• If the participants could not be contacted by telephone, field locators made in-person contact with them and provided them with a cell phone to complete the survey with a telephone interviewer.

Additional Mitigation

As it became apparent that survey production in the two affected sites would likely be insufficient, the contractor took additional steps to address non-response in these sites.

• The survey fielding period was lengthened to provide additional time to locate participants and complete surveys.

• Site staff began providing updated participant contact information for nonresponding sample members.

Plans for Future Mitigation

• (The subject of this request) ACF requests a change to the survey incentive amount in two sites.

• Site staff will continue to provide updated participant contact information for nonresponding sample members.

• For MTCI, contractor staff will begin direct messaging via Instagram to participants who provided consent and Instagram user information. No participants will be allowed to follow the contractor’s account, in order to protect participants privacy.

• As described in Supporting Statement B of the OMB package approved in April 2019, following the conclusion of data collection, we will use demographic information from the baseline survey to (1) statistically adjust for observed differences between the characteristics of the treatment and control group respondents and (2) produce sample weights so that the weighted baseline characteristics of the respondents in the treatment and control group are similar to those of the full sample (respondents and nonrespondents).

Proposed Intervention for OIRA Approval

We will continue our increased efforts in tracing and field location, described above.

We also propose to replace the currently-approved $25 incentive with a $50 incentive to all study participants completing a second follow-up survey in the MotherWise site. In the MTCI site, we propose replacing the currently-approved $20 incentive with a $40 incentive to all participants completing a second follow-up survey. This increased incentive would be offered to all nonresponders to the second follow-up survey.

ACF’s proposed change to the incentives is aimed at increasing the likelihood that sample members who are successfully contacted respond to the survey. It is also intended to increase the likelihood that friends and relatives of the respondents who are contacted by the survey team will tell respondents about the opportunity to complete the survey.

Expected Benefits and Proposed Assessment

We hypothesize that increasing the incentives will increase response rates conditional on successful contact, particularly for the control group that is disproportionately underrepresented among survey respondents. This would reduce the risk of bias in our experimental impact estimates.

While recent RCTs with similar populations do not provide causal evidence about the impact of incentives on survey participation, several data collections provide observational data suggesting that the amount may be sufficient to support desired outcomes. Recent RCTs with similar populations have used incentives in the $40 to $50 range to conduct longitudinal surveys of equal or lower respondent burden; these data collection efforts had lower non-response bias than the current study. Examples include the Evaluation of Adolescent Pregnancy Prevention Approaches study (OMB control number 0990-0382), which offered a $50 incentive in two of its six sites, and obtained an overall response rate of 80 percent or higher and low treatment-control differentials in those sites. The evaluation of YouthBuild (OMB control number 1205-0503) similarly achieved high overall response rates with low treatment-control differentials and offered incentives of $40 to $50. Given these recent experiences, we propose that a $50 incentive is an appropriate amount for adults for completing a 45-minute survey and a $40 incentive is an appropriate amount for youth completing a 30-minute survey.

This proposal was designed to maximize improvements in data quality given the results of ACF’s efforts to date and the constraints of remaining time and budget for the data collection. ACF is interested in using this opportunity to contribute to the body of evidence on the role of incentives in mitigating non-response bias. Response rates before and after the change in incentives will be calculated for all study participants in each site as well as for each research group. In addition, the demographic characteristics of respondents before and after the increase in incentives will be compared with the full baseline study sample. This analysis will yield an estimate of the impact of our changes to fielding protocol on observable non-response bias.

Given the widespread changes caused by COVID-19 and the fact that we propose to pair the incentive with additional outreach, this analysis will not isolate the causal effect of increased incentives alone. ACF anticipates that the information resulting from the survey administration efforts, including the proposed change in incentive amount, will be of sufficient quality to meaningfully contribute to ongoing learning about strategies for improving the quality, efficiency, and effectiveness of our survey data collections.