Appendix F_MIHOPE-K Maximizing response rates

Appendix F

MIHOPE-K: Maximizing Response Rates

Minimizing sample attrition is of utmost importance to any longitudinal study. Experience with this sample has demonstrated that many MIHOPE families have been highly mobile, and therefore are at continued risk of attrition at follow-up.

Several strategies have been adopted to mitigate the risk of attrition at the kindergarten follow-up:

1. Implementing a multi-pronged tracing effort to minimize attrition from outdated contact information

We will continue to use the detailed information collected in MIHOPE 1, MIHOPE 2, MIHOPE Check-in, and the first rounds of MIHOPE-K data collection (including names, dates of birth, Social Security numbers, addresses and phone numbers [home and work], and email addresses for the family, as well as addresses and phone numbers for up to three relatives or friends who will know how to reach the family) and employ Mathematica’s highly effective locating techniques to reach families.

Updating Participant Contact Information. Mathematica’s Sample Management System (SMS) is the central clearinghouse for all contact information on MIHOPE families, and is also used to track structured interview response rates. Contact between rounds of the structured interview increase sample retention and reduce the level of effort needed to locate families. To reduce the loss of families between follow-up points, we send families a study information packet that contains a newsletter with updates about the study. Additionally, we send a birthday card to each child on a yearly basis (Appendix A) and a seasonal greeting (either in the winter or spring) (Appendix A). An example of the format and topics covered in the newsletter is included in Appendix C.

If any updated contact information is provided after the mailing of the letters, postcards, birthday cards, or holiday cards, or they are returned from the post office with an updated address, we document the new address for the family in the SMS and re-mail the materials to the updated address.

Locating Participants. Although the outlined strategies to track participants between follow-up rounds will likely result in lower attrition rates, additional techniques are used to ensure a high response rate is achieved at each follow-up round from this mobile population. Mathematica has extensive experience conducting studies with mobile and hard-to-reach populations and has developed several techniques to locate these populations. Locating can be costly, depending on which methods are used. In general, mailing letters and receiving updated information via returned mail is less expensive than electronic database searches; electronic database searches are less expensive than locators calling neighbors or other contacts; and telephone locating is less expensive than in-person field locating. The least expensive methods (mailing and electronic locating) are used before moving to more expensive methods (telephone and in-person locating). As preparations to conduct follow-up data collection get underway, the following process for locating participants are employed: (1) multiple pre-field mailings, (2) in-house locating, and as needed, (3) field locating. All materials used for locating and contacting participants are included in Attachment 7.

1. Pre-Field Mailings. Any letters or postcards that are returned to Mathematica with updated information are re-mailed to the new address and the new address is entered into the SMS. Families are then sent an additional mailing, an invitation letter, directly before calling for the structured interview begins. (We also send an email version of the letter.)

2. In-House Locating. Custom database searches and telephone calls to contacts provided by the family during prior rounds of MIHOPE data collection are conducted when the existing contact information we have for a family is not accurate and pre-field mailing does not yield an updated telephone number or address. Mathematica’s specialized locating staff uses searchable databases, directory assistance services, reverse directories, and contacts with neighbors and community organizations to obtain current contact information. Mathematica’s locating staff willsearch the Web and social networks such as Facebook and Instagram to find sample member contact information.

3. Field locating. Some families are not locatable using in-house locating methods. These families are assigned to field locators who employ proven techniques for finding hard-to-find populations. For instance, field staff may approach neighbors residing in close proximity to the families’ last known address or the contact persons provided during prior structured interview rounds. They also rely on neighborhood resources such as local post offices, churches, bars, homeless shelters, or community centers as sources of information. Field staff are trained not to reveal any private information about the participant to any informants, including the study’s name or unique details about the study.

2. Training telephone interviewers and field workers on techniques for building participant buy-in and converting caregivers to participation. Field staff and assessors are trained to establish rapport with families so that they have a positive impression of the study and are more willing to participate in the future.

3. Utilizing multimodal reminders based on behavioral science principals. We plan to use the email addresses and cellular telephone numbers of participants that have been collected in MIHOPE 1, MIHOPE 2, and MIHOPE Check-in to send email and text message reminders about the follow-up data collection during the fielding period.

4. Providing tokens of appreciation, as discussed in Supporting Statement A.

Tokens of appreciation are intended to address the following concerns:

Reducing nonresponse bias, differential attrition, and overall attrition to ensure that the study has enough statistical power and a sufficiently representative sample to answer its key research questions. A high response rate makes it more likely that interview respondents are representative of the initial sample (including ensuring equal representation among the program and control groups), which is important when estimating effects of home visiting for the study population. As has previously been communicated to OMB, MIHOPE struggled with overall attrition and attrition across subgroups of families in the MIHOPE Check-in 2.5 year old survey.

In the 2.5 year sample, we conducted an experiment examining a pre-pay and an early bird token of appreciation strategy (for additional information about the experiment and the results, please see the memorandum detailing this experiment). Overall attrition was 45.2% for the portion of the 2.5 year follow-up sample that participated in our token of appreciation experiment (N = 1,705). As previously communicated to OMB, we also found some statistically significant differences in important baseline characteristics between respondents and nonrespondents of the 2.5 year old survey. For example, as shown in TableF.1, nonrespondents were:

• more likely to have entered the study while they were pregnant, which we expect to be an important predictor of the effectiveness of home visiting services

• more likely to have moved in the year prior to entering the study, so survey responses might not accurately represent the effects for the most mobile part of the sample

• less likely to live in a household with their child’s father figure, and are

• less likely to be married to the biological father of their child.

Table F.1: Differential response to the 2.5 year old survey: Significant differences at end of token of appreciation experiment period^{1, 2}

Characteristics (at study entry)	Respondents (%)	Nonrespondents (%)	Difference
Pregnant	47.2	56.6	-9.4
Moved in the prior year	17.1	25.1	-8.0
Child’s father figure does not live in household	54.2	61.5	-7.3
Not married to biological father of child	77.9	85.3	-7.4

Differential attrition across key subgroups is also a major concern for the kindergarten time point of MIHOPE, as subgroup representativeness is necessary to address a primary research question: Are the long-term effects of home visiting larger for some types of families than for others?

Ensure the study meets quality rating standards set out by the What Works Clearinghouse (WWC) ³ and by HHS’s Home Visiting Evidence of Effectiveness review (HomVEE).⁴ It is especially important for MIHOPE to meet HomVEE’s standards, as this is the primary evidence review for the home visiting field. Study rating criteria include low overall attrition and low differential attrition. Meeting these standards is necessary for the study to be maximally useful for policymakers and practitioners.

5. Providing a study web page to relay information about the study to participating families. (The website, included as Instrument 6, allows families to provide consent for the study to contact their child’s teacher.)

3