Justification for study change memo

F

memorandum

Social and Economic Policy

Date	November 1, 2007
To	Ivelisse Martinez-Beck, Mary Bruce Webb, Naomi Goldstein
From	Ann Collins and Charles Michalopoulos
Subject	Request for Changes in Evaluation of Child Care Subsidy Strategies: Massachusetts, Illinois, and Washington (OMB NO.: 0970-0306)

As you know, we are requesting two changes to the study approach that require OMB clearance. One relates to the Massachusetts study; the other to the Illinois study. Neither results in additional respondent burden. Each is summarized below:

Massachusetts Requested Change: Substituting Standardized Child Assessment Measures:

The Massachusetts study tests the effects of LearningGames, a developmental curriculum, in family child care homes using a random assignment design. The original plan called for using the PLS-4 as the tool for measuring impacts of LearningGames on child development for all children, regardless of age. The PLS-4 was selected for three reasons:

• Psychometric properties. The PLS-4 has strong psychometric advantages, including its overall reliability and validity and the fact that it has been normed. It is appropriate for the age range of children in the study and has a Spanish version.

• Consistency with another major being funded by the ACF. One, the “QUINCE” study, uses the PLS-4 to assess child development outcomes; having parallel measures across the two studies will have important advantages when outcomes of the two family child care intervention studies are presented.

• Validity as an assessment tool for the LearningGames program. LearningGames is particularly focused on children’s oral language development and the PLS-4 includes subtests for receptive and expressive language.

We planned to use both the auditory (receptive language) and expressive (expressive language) components of the assessment. However, when we consulted with the QUINCE study team about their experiences with the PLS-4, they advised that they had decided to use only the Auditory subscale and not the Expressive subscale. The primary reasons given were (a) they believed that the Auditory subscale did a better job of addressing children's knowledge vs. children's productive speech skills; children’s knowledge was more central as an outcome of interest; and (b) the Expressive subscale includes a substantial number of items based on provider report, which they wanted to minimize because of problems with reliability with provider or parent reports.

We then decided to recommend substituting the Bracken Basic Concept Scale--Revised (BBCS-R) for the preschool children (ages 30 months and older) with the Expressive subscale of the PLS-4. On this test, we would use only the first 6 subtests, which form the School Readiness Scale.¹ The recommendation to use the Bracken is based on the same reasons that we gave for using the PLS-4:

• Psychometric properties. The Bracken has similar psychometric advantages to the PLS-4, including its overall reliability and validity and the fact that it has been normed. It is appropriate for the age range of children in the study and has a Spanish version.

• Consistency with the QUINCE study. Since the QUINCE study is now using the Bracken with children 30 months and older to assess school readiness; having parallel measures across the two studies again will have important advantages when outcomes of the two family child care intervention studies are presented.

• Validity for the LearningGames program. LearningGames also focuses on children’s concept development, and the Bracken is a well-known measure of the child’s understanding of the basic concepts that are important for school performance.

Effects on Burden

Administering both parts of the PLS-4 would require 30 to 45 minutes, depending on the child’s level of ability (children who respond correctly are administered more items). By eliminating the Expressive subscale, we will cut the assessment time in half. The first 6 subtests of the Bracken take 10 to 15 minutes to administer. Therefore, replacing the Expressive subscale of the PLS-4 with the Bracken does not increase the assessment burden on the child.

Illinois: Increasing Incentives from $20 to $50 to Improve Response Rate

As described to OMB in 2005, approximately 1,900 parents in Cook County, Illinois, with incomes between 50 and 65 percent of state median income (SMI) and who applied for child care subsidies were asked to participate in a random assignment study. Without the study, these families would not have been eligible for child care subsidies because their income placed them above state income guidelines. Half of the families who agreed to be in the study were approved to receive subsidies for two years. The study will measure the effects of child care subsidies on the use of public assistance programs, earnings and employment stability, as well as child care section, satisfaction, and stability.

Outcome data come from administrative records and from participant interviews. Administrative records provide information on child care subsidy use, employment and earnings, and receipt of TANF and food stamps benefits. A survey is being administered to gain additional information on job characteristics, child care characteristics, and the stability of employment and child care. To date, however, the survey has achieved a lower than optimal response rate. Our goal is to obtain an 80 percent response rate to maximize the likelihood that survey responses will provide results that are representative of the full sample.

The survey began in January 2007. Between then and May 2007, survey respondents were located solely by telephone with no in-person tracking. At that point, a small pilot effort indicated that in-person tracking would locate enough study participants to allow the survey to achieve our response rate. And, after the in-person tracking was started, survey response rates initially grew rapidly.

However, it has become clear that the problem goes beyond locating sample members. Although in-person tracking initially increased the response rate, at this point without additional changes we will not reach our 80 percent goal for another eight months, if we can reach it at all. Our field staff say that they are able to locate individuals but have not been able to persuade many of them to respond to the survey. For instance, as of early October, they had located 195 potential survey respondents who still would not complete the survey. Field staff report that many of these individuals have indicated that the $20 incentive is not “worth their time” to participate in the interview. It seems as if this is the sentiment especially for control group members who “didn’t get child care anyway.” Field staff therefore believe that many of the 195 who have been located but who have not yet completed or refused to complete interviews could be converted if the incentive was changed from $20 to $50. Other recent experiences indicate that there are similar issues using a $20 incentive and having difficulties with response rates in samples of low-income families in urban areas. One recent study made approximately 39 attempts for each complete and was only able to obtain a 58 percent response rate. One of the reasons anecdotally cited as a reason for non-response was that the 30-minute interview was too much of a hassle, given the incentive amount.

Therefore, in order to increase response rates, and minimize the nonresponse bias, we are requesting an increase in payments to survey respondents from $20 to $50 per completed survey. Our experience is that higher incentives are likely to produce higher response rates. One example from a recent MDRC study comes from the Opening Doors site at Kingsborough Community College in Brooklyn. There, MDRC was able to generate 146 additional survey responses among 811 non-completers by increasing the cash incentive from $20 to $50. Finally, we note that OMB has previously approved incentives of $50 or higher for this group. For example, the Moving To Opportunity study conducted by Abt Associates, completed in 2004, paid survey respondents $55 per completion.

Burden

The requested change in the incentive amount from $20 to $50 will not affect respondent burden.

1 The six subtests include: colors, letters, numbers, shapes, size, and quantity.

Abt Associates, Inc. October 31, 2007 1