Document

Abt Single-Sided Body Template

ICR 200803-0970-008 · OMB 0970-0306 · Object 6396401.

Document Viewer [doc]

Status: The document is viewable through fallback artifacts while conversion repair continues in the background.

Queue: A repair attempt failed. The document remains flagged for retry and verification. (priority 90 · attempts 13 · next 2026-10-15 21:11:16)

Missing: html

Download: Source copy (doc) | pdf

Primary: htmlSource: application/msword

Loading document viewer…

Document Metadata

Record metadata
File Type	application/msword
File Title	Abt Single-Sided Body Template
Keywords	Single-Sided body Templates
Author	stewartg
Last Modified By	Writer
File Modified	2006-07-10
File Created	2026-10-14
Conversion State	failed_conversion

Extracted Text

Evaluation of
Child Care
Subsidy Strategies
Request for OMB
Clearance:
Massachusetts, Illinois,
and Washington

June 2006

Prepared for
Ivelisse Martinez-Beck
Child Care Bureau
Administration for Children and
Families, HHS
1250 Maryland Avenue, SW
Portals Building, Suite 800
Washington, DC 20024

Prepared by
Ann Collins
Charles Michalopoulos
Jean I. Layzer

Abt Associates Inc.

OMB Clearance Request

Table of Contents
Summary................................................................................................................................................. i
Part A Justification............................................................................................................................... 1
A1 Explanation of the Circumstances That Make the Collection of Information Necessary. .1
a.Massachusetts……………………………………...…………………………………...3
b. Illinois……….…..………………………………………………..……………...…….3
c. Washington……….………………………………………………...………………….5
A2 How the Information Will Be Used, by Whom, and for What Purpose.............................6
a. Massachusetts………………………………………………………………………….6
b. Illinois………………………………………………………………………………...10
c. Washington…………………………………………………………………………...15
A3 Use of Improved Technology to Reduce Burden……………………………………….23
A4 Efforts to Avoid Duplication............................................................................................24
A5 Efforts to Minimize Burden on Small Businesses or Other Small Entities.....................24
A6 Consequences if the Information is Not Collected..........................................................24
A7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent
with Section 1320.5 (D) (2)............................................................................................. 26
A8 Efforts to Consult with Persons Outside the Agency.......................................................26
A9 Payments to Respondents.................................................................................................27
A10 Assurances of Confidentiality Provided to Respondents.................................................28
A11 Justification for Questions of a Sensitive Nature.............................................................30
A12 Estimates of Respondent Burden.....................................................................................30
A13 Estimates of the Cost Burden to Respondents.................................................................33
A14 Estimates of the Cost to the Federal Government............................................................34
A 15 Reasons for Any Program Changes or Adjustments........................................................35
A 16 Plans for Tabulation, Statistical Analysis and Publication..............................................35
a. Massachusetts...............................................................................................................35
b. Illinois………………………………………………………………………………..38
c. Washington…………………………………………………………………………..47
A17 Display of Expiration for OMB Approval.......................................................................53
A18 Exception to the Certification Statement Identified in Item 19.0 of Form OMB 83-I....53
Part B Collection of Information Using Statistical Methods...........................................................54
B1 Sample Universe, Sampling Method and Expected Response Rates...............................54
a. Massachusetts..........................................................................................................................54
b. Illinois...........................................................................................................................55
c. Washington…………………………………………………………………………...56
B2 Data Collection Strategy.................................................................................................. 59
a. Massachusetts............................................................................................................... 59
b. Illinois...........................................................................................................................61
c. Washington................................................................................................................... 64
B3 Methods to Maximize Response Rates............................................................................ 68
B4 Tests of Procedures.......................................................................................................... 71
B5 Individuals Consulted on the Statistical Aspects of the Design.......................................71
References……………………………………………………………………………………………
73
Abt Associates Inc.

OMB Clearance Request

Summary
The Evaluation of Child Care Subsidy Strategies is designed to provide Federal, state, and local
policymakers with information about the role of subsidy programs and policies in helping low-income
families obtain and retain work and in improving outcomes for children. The goal of the study is to
determine how differences in certain aspects of child care subsidy policies or quality-improvement
efforts are related to outcomes for parents, children, and child care providers. The Evaluation of Child
Care Subsidy Strategies consists of three studies, one in Massachusetts, one in Illinois, and one in
Washington. OMB clearance is sought for each study site.
a. Massachusetts
The Massachusetts experiment will test the effects of a curriculum designed to address language
development and the development of pre-literacy skills for very young children in family child care
settings. Study participants are approximately 350 family child care providers who belong to one of
16 state-supported family child care networks and care for subsidized children. Each of these
providers has approximately two children under the age of three in their homes. Half of the providers
(i.e., the treatment group) will be randomly assigned to use Learningames, a research-based early
childhood curriculum that can be easily adapted for use in family child care homes. These providers
will be supported by network home visitors who are trained in the Learningames approach. The other
half of the providers (i.e., the control group) will continue to offer care in their usual manner and be
supported by the network’s standard training and technical assistance.
The major research questions include:


What is the effect of a research-based developmental curriculum (Learningames) on
provider’s behavior and interactions with children on the language and literacy
environment of the home?



What is the impact of the intervention on children’s language and pre-literacy skills?

The study will include an implementation analysis and an impact analysis. Data sources include
observations of the family child care setting, at baseline and three other points in time, assessments of
children taken at two points in time, questionnaires for providers at two points in time, and a one-time
questionnaire for home visitors. Providers began using Learningames in the fall of 2005. The
intervention and evaluation will be completed in the fall of 2007. Reports will be issued in 2008.
b. Illinois
The Illinois experiment will test the effects of subsidies on parental employment and selection of
child care for families with incomes between 50% and 65% of State Median Income (SMI). Such
families currently have incomes slightly over the state’s income eligibility ceiling. Approximately
2,000 families living in Cook County will be in the study; half will receive subsidies (i.e., the
treatment group) and the other half will remain ineligible for subsidies (i.e., the control group).
Families in the treatment group will be eligible for subsidies for a two-year period. These families
will be randomly assigned to a six-month certification period, at which point they must reapply for
subsidies in order to continue receiving them, or a 12-month certification period.
The major research questions are:
Abt Associates Inc.

OMB Clearance Request

iii



What is the impact of receiving a child care subsidy on parental employment, family
income, and the receipt of public benefits?



What is the impact of receiving a subsidy on the type of care chosen, the stability of the
arrangement and on parents’ satisfaction with the child care?



Does the length of the certification period for subsidies affect how long families receive a
subsidy?

The study will include process analysis, impact analysis, and cost-benefit analysis. Data sources
include (1) a 35-minute telephone interview of parents, conducted three times, (2) administrative
records from the child care subsidies, other public benefits, and unemployment insurance programs,
(3) 30-minute interviews, asking the same questions to fewer than nine state and local staff to
understand contextual issues for the process analysis, and (4) policy manuals and other documents.
Recruitment for the study began in the spring of 2005 and will be concluded in the spring of 2006.
The intervention and evaluation will be completed in the spring of 2008. Reports will be issued in
2008 and early 2009.
c. Washington
The Washington state experiment focuses on parental co-payments for families receiving child care
subsidies. In the fall of 2005, approximately 5,150 families who applied or reapplied for subsidies
throughout the state during a three-week time period were assigned to either the state’s standard copayment scale or an alternative scale. The alternative scale requires either the same or a lower copayment amount for families at every income level. The intervention will continue for two years. If
families leave the subsidy system and return to it, they will still be assigned co-payments according to
the standard co-payment scale (if they are in the control group) or the alternative scale (if they are in
the treatment group).
The major research questions are:


What is the effect of the alternative co-payment scale on parental employment, family
income, length of receipt of subsidies and the receipt of other public benefits?



What is the effect of the alternative scale on the type of child care chosen, the stability of
the arrangement, and on parents’ satisfaction with child care?



Is the effect of the alternative co-payment amount on employment and child care
outcomes different for families who are new subsidy recipients than for families who are
already receiving a subsidy and wish to continue?



Does the impact of the alternative co-payment differ for families at different income
levels at the beginning of the study?

OMB Clearance Request

Part A
Justification
A1

Explanation of the Circumstances That Make the Collection of
Information Necessary

Investment in child care by the Federal government and by individual states increased substantially in the
years after the passage of the Personal Responsibility and Work Opportunity Reconciliation Act. The
legislation provided greatly increased Federal resources to states to provide child care assistance,
authorizing some funds specifically for child care and also allowing states to transfer Temporary
Assistance for Needy Families (TANF) funds to the Federal child care program (and to spend TANF
funds directly on child care subsidies). The Child Care and Development Fund (CCDF), created as part
of the legislation, combined four of the existing child care funding programs into a single block grant to
states, giving them much more flexibility to decide how child care funds should be expended. In FY
2004, Federal and state spending on child care totaled $9.4 billion: Federal CCDF funding (including
TANF funds transferred into CCDF) reached $6.9 billion; state spending totaled $2.5 billion. In addition,
direct TANF spending on child care was $1.4 billion. As a result of increased Federal funding combined
with steady increases in the states’ contributions to the subsidy program, many more low-income families
with working parents are able to receive help in paying for child care. In addition, the CCDF stipulates
that states must set aside 4% of their CCDF funding for efforts to expand the supply or improve the
quality of child care; many states allocate more than this minimum amount. States face a considerable
challenge in trying to use their child care funds as effectively as possible, both to support parent’s
employment and to improve child care quality to ensure children’s safety and enhance their development.
In related efforts, states are working to meet the goals of President Bush’s initiative, Good Start, Grow
Smart, to enhance the school readiness of young children. Universal pre-kindergarten (UPK) is one of the
strategies that many states use to meet the presidential mandate. Typically, states are implementing UPK
through the existing system of schools and child care centers. However, much of the care for young
children, especially infants and toddlers, is provided in family child care homes. States can and do use
the CCDF quality-set aside funds to attempt to improve the quality of family child care, but they do so
with scant information about the effectiveness of their efforts. The majority of the research available
about efforts to enhance the school readiness of children from low-income families focuses on centerbased early childhood programs that serve primarily preschool-age children. A research focus on centerbased programs for three- and four-year old children does not reflect the widespread recognition that the
very early years are also a critical period of development. A focus on the school readiness of children in
center and pre-school classrooms leaves out the many children who are cared for in family child care,
before they reach the age where they may be in settings supported by UPK. The work of Hart and Risley
(1996), among others, suggests that efforts need to be made early in children’s lives to enrich their
language environments because this are of development is key to school readiness and later success in
school.
Furthermore, despite the substantial increases in funding for subsidies over the last seven years, more
recently some states have faced budget constraints that have had an impact on the subsidy program.
Regardless of their fiscal situation, states must still make choices about how to allocate resources and
target subsidies to meet multiple objectives. They do so directly and indirectly through a host of
decisions about child care policies and their implementation. These include: determining the level of state
Abt Associates Inc.

OMB Clearance Request

resources and matching requirements for counties (where applicable); setting eligibility guidelines and
setting priorities for subsidies (including priorities attached to serving TANF vs. non-TANF families);
deciding how and to what extent the availability of subsidies will be publicized; developing co-payment
scales; and developing fee schedules and payments for providers. None of the child care research
conducted over the past three decades has systematically examined the effectiveness of different child
care subsidy policies or programs.
Recognizing the need for carefully-designed research that would provide useful information to states and
communities, the Child Care Bureau and the Office for Planning, Research and Evaluation of the
Administration for Children and Families (ACF) within the US Department of Health and Human
Services commissioned a research effort designed to expand our knowledge about child care subsidies
and quality-improvement efforts. In September 2001, a contract was awarded to Abt Associates Inc. to
conduct a multi-site, multi-year Evaluation of Child Care Subsidy Strategies. To carry out the study, Abt
Associates and its subcontractors—Manpower Demonstration Research Corporation (MDRC), Columbia
University’s National Center for Children in Poverty (NCCP), and Moore & Associates, Inc.— worked
closely with state and local partners in four sites to design and implement experimental studies that are
tailored to their needs and interests, as well as the interests of policymakers in general.
The Evaluation of Child Care Subsidy Strategies will provide Federal, state, and local policymakers with
information about the role of subsidy programs and policies in helping low-income families obtain and
retain work and in improving outcomes for children. The goal of the study is to determine how
differences in certain aspects of child care subsidy policies or quality-improvement efforts are related to
outcomes for parents, children, and child care providers/caregivers. Outcomes of interest include the
stability of parental employment and earnings, parent and child well-being (especially the development of
children’s language development and literacy skills), availability of care, and child care quality. The
study will address this goal through rigorous evaluation using a set of three random assignment
experiments that will test aspects of subsidy policies or evaluate quality-improvement efforts. Three
states and one locality have agreed to participate in the study—Illinois, Washington, Massachusetts, and
Miami-Dade County in Florida. This request for OMB approval is for the experiments that will be
conducted in Massachusetts, Illinois, and Washington.1
Authorization for the CCDF is part of the Personal Responsibility and Work Opportunity Reconciliation
Act (PRWORA), Public Law 104-193 (42 USC 1305). Authorization for research related to the CCDF is
part of the appropriations legislation for the Departments of Labor and Health and Human Services (PL
109-149).2

a. Massachusetts
1

In the fourth site, Miami-Dade County, we are using extant data on children and teachers collected by the Early
Learning Coalition of Miami-Dade/Monroe Counties as part of its ongoing Assessment and Improvement
Initiative to address research questions about the impact of the experimental treatment. Abt is collecting
observation data on each study classroom, but there is no data collection burden and thus no request for
clearance.

The Labor-HHS appropriations bill states that “$9,920,000 shall be for use by the Secretary for child care
research, demonstration, and evaluation activities.”

Abt Associates Inc.

OMB Clearance Request

In Massachusetts, many licensed family child care providers who accept subsidies for children in their
care operate under the auspices of family child care networks (called “systems” by the state). There are
more than 50 such networks in the state, varying greatly in size and geographic coverage. Networks in
Massachusetts receive state funds to provide technical assistance to their member providers in part
through regular visits by home visitors/mentors. These visits are intended to provide general support and
information to providers to improve the quality of the care they offer. In 2003, the state developed family
child care regulations that stipulated that all licensed family child care providers must use a
developmental curriculum. However, little guidance has been offered about how to identify and select a
curriculum, and state agency staff believes that the mandate has largely been ignored or overlooked,
although mentors could, in principle, help providers do so. Further, little research, thus far, has focused
on the linkages between the use of CCDF quality funds and state activities aimed at improving the quality
of programs and of the care children experience.
To address the need for tested approaches that enhance children’s language development and pre-literacy
skills, the Commonwealth of Massachusetts has identified a strong research-based curriculum—
Learningames— that can easily be adapted for use in a family child care setting. The state is interested in
experimentally testing the effectiveness of Learningames in family child care homes linked to family
child care networks. The state views the proposed experimental test of Learningames as an opportunity to
provide both system administrators and providers with evidence-based guidance on how to address the
state mandate as well as complement their current efforts on a new UPK initiative, by focusing on
younger children in family child care settings. The need for such a study is clear. In Massachusetts the
most recent research on the quality of child care (Marshall, et al, 2001 and 2003) indicates that, on
average, the quality of care in family child care is considerably lower that of center-based programs.
Indeed, only 30% of family child care homes met accepted standards for good quality child care.
The evaluation in Massachusetts includes 350 child care providers who are members of 16 family child
care networks. Each provider is caring for at least two children under the age of 26 months at the outset
of the study. Half of the providers were assigned to receive training on the implementation of
Learningames, materials and mentoring support; the remaining providers will continue to receive the
usual ongoing training and support. Recruitment for the study began in the spring of 2005 and random
assignment was completed by June 30th. Outcomes of interest are changes in caregiver behavior and
changes in children’s language development and literacy skills. The data collection includes observations
of the family child care home environment and child assessments. It will occur at three points in time: in
June 2006 (after receiving OMB approval); in January 2007; and January/February 2008. In addition, a
short caregiver survey will be administered twice; in June 2006 and January 2008. Baseline data will
include observations of homes and extant child assessment data collected by family child care network
staff. Finally, home visitors will be asked to complete a brief questionnaire once, in June 2006.
b. Illinois
The study in Illinois will consist of a random assignment evaluation to assess the effects of subsidies on
parental employment and selection of child care for families with incomes between 50% and 65% of the
state median income (SMI). Such families currently have incomes slightly over the state’s income
eligibility ceiling. Approximately 2,000 families residing in Cook County will be in the study; half will
receive subsidies (i.e., be in the treatment group) and the other half will continue to remain ineligible for
subsidies (i.e., serve as the control group). Recruitment for the study began in March 2005 and is
expected to occur over a 12-month period. The study has three components: an Implementation Study, an
Impact Study, and Cost-Benefit Study. We are seeking clearance for a parent interview that will be used
to inform the impact and cost-benefit studies (this interview will also be used in the Washington study).
Abt Associates Inc.

OMB Clearance Request

The survey will be conducted three times over the two-year period. Baseline data on study participants
will be obtained through extant administrative records. The Implementation Study will rely on
documents, researcher notes made during the planning and start-up period, and brief interviews conducted
with fewer than nine people in the state.
The State of Illinois currently sets the income eligibility ceiling for initial and continuing receipt of
subsidy at 50% of State Median Income (SMI), and is committed to serving all eligible families that
apply. In 2003, all families receiving subsidies in Illinois had an average annual income of $10,744, and
50% of SMI was equivalent to an annual income of $21,819 (Anderson, Ramsburg, Rothbaum, 2003).
The CCDBG eligibility limit is set at 85% of SMI and the average state subsidy eligibility ceiling is set at
62 % of SMI (Collins, et al., 2000), thus the new Illinois income limit of 65% is comparable to the state
SMI average. Allowing a group of families between 50 and 65% of SMI to receive subsidies would allow
us to examine the impact of child care subsidy receipt on employment, on achieving self-sufficiency, and
on child care outcomes for low-income families currently not receiving TANF assistance, but who
include former TANF recipients or families at risk for TANF receipt.
A study of the dynamics of child care subsidy receipt, conducted in Illinois and four other states, found
that the median length of the first spell of subsidy receipt was six months in Illinois (Meyers et. al., 2002).
Because most families currently are certified to receive subsidies for six months, the state is interested in
knowing whether a longer certification period would increase the use of subsidies and the stability of
care, as well as make subsidy administration more efficient. Results regarding the impact of changing
recertification periods will serve to inform administrative decisions aiming to increase administrative
efficiency and strengthen fiscal and administrative accountability.
The study will test the impact of subsidies on the employment outcomes, child care choice and stability,
and child care satisfaction. The study group will be composed of two subgroups. Subgroup 1 will include
those families who are new applicants to the subsidy system.3 For this subgroup, families who apply for
subsidies with incomes over the current eligibility ceiling but under 65% of SMI will be randomly
assigned to a treatment group (approved to receive subsidies) or a control group (not approved to receive
subsidies). The second subgroup will be composed of families who were using subsidies, but, at redetermination, have incomes between the current eligibility ceiling and 65% of SMI. As with the first
group, these families will be randomly assigned to a treatment group (recertified to receive subsidies) or a
control group (not recertified to receive subsidies). With the subgroups, it will be possible to test the
effect of subsidy receipt on families with incomes between 50% and 65% of median income and whether
there is a difference in impacts for a family newly receiving a subsidy (Subgroup 1) versus a family that
had a subsidy but lost it (Subgroup 2).
In addition, the study will test the effects of longer certification periods by randomly assigning those
approved to receive subsidies in the treatment group into two groups: a) eligibility certified for six
months; and b) eligibility certified for one year. Families in the two treatment groups will retain
eligibility for subsidies over the two-year study period, provided their income remains below the
experimental limit and they comply with other requirements (e.g., continue to be employed, have children
under 13 years of age, etc.). Outcomes will be measured through administrative records and interviews
with parents.

“New” is defined as having a break of one month or longer in the use of subsidies.

Abt Associates Inc.

OMB Clearance Request

The experiments will result in three related sets of analyses: an implementation evaluation, an analysis of
impacts on families, and a cost-benefit analysis. The experiment began in spring 20054; the final report
for the evaluation will be completed in early 2008. Near the end of the 24-month period during which
families in both experiments receive subsidies under enhanced eligibility guidelines, a letter will be
mailed to families reminding them of the date that the enhanced eligibility guidelines will no longer be in
effect.
c. Washington
The CCDF is a major source of funds for child care subsidies in Washington. The law requires states to
establish sliding fee scales for subsidies but gives little further instruction. Little is known about the
effects of varying co-payment amounts on outcomes related to parents’ employment and on their selection
of and use of child care.
In 2000, the average co-payment amount, among those families that had a co-payment, was
approximately six percent of family income (Administration for Children and Families, 2003).
However, individual state co-payment policies vary greatly. Differences in co-payments across states are
quite pronounced for eligible families who are at the higher end of the income scale. For instance, in
1999, at 33 percent of state median income, among the 17 states in the National Study of Child Care for
Low-Income Families, the required co-payment ranged from 2 percent of a family’s income (Minnesota)
to 17 percent (Massachusetts) (Collins et al, 2000). In addition, in their 2002-2003 CCDF state plans, 14
states reported that they prohibit child care providers from charging fees in addition to the co-payments
established by the state (Administration for Children and Families, 2002).
In some states, the co-payment schedule produces large jumps, or “notches,” as families earned more. In
Washington State, for instance, the 1999 co-payment was 6 percent of a family’s income when its income
was 33 percent of the state median, but climbed to 20 percent of the family’s income when it reached 50
percent of the state median. In other states, the co-payment was consistent across income levels; for
example, in North Carolina, the co-payment always accounted for nine percent of a family’s income
(Collins et al., 2000).
How do co-payments affect families’ work and child care decisions? A large literature in economics has
tried to address how child care subsidies affect employment and child care use by examining the effects
of reduced child care costs on these outcomes. Two useful summaries of this literature are Chaplin et al.
(1999) and Blau (2000). Chaplin et al. (1999) summarize studies of the effects of child care costs on child
care use patterns, and find that the average results of 13 studies of child care use indicated that a 1 percent
reduction in child care costs would cause a 0.5 percent increase in the number of families using market
forms of child care. This suggests that a lower co-payment would help families use non-parental care,
which might help them stay at work.
Blau (2000) summarizes studies of the effects of child care costs on maternal employment. He finds that
results from 12 studies of the effects of child care costs on employment imply that a 1 percent reduction
in child care costs would encourage .4 percent of parents to work. However, few of these studies focus on
low-income families and Blau points out that in studies that do focus on such families, the price
elasticities of employment tend to be larger. Despite the large number of studies, there is a critical need
for experimental studies of employment responses of low-income families in the new environment
created by welfare reform.
4

Baseline data comes from extant administrative records so there is no respondent burden.

Abt Associates Inc.

OMB Clearance Request

The need for such a study of the effects of different child care co-payments on parental employment and
child care choice is clear. To date, there is little research to guide states in structuring co-payment
schedules. The only research on co-payments is descriptive and often shows how states co-payment
policies compare. In Washington State, the current parent co-payment schedule begins with $15 per
month for families whose incomes are below 82% of the federal poverty level (FPL), jumps to $50 for
families with incomes between 82% and 137.5% of the FPL, and then rises very steeply. There is concern
that these “notch” effects in the schedule, as well as relatively high co-payments for some families, may
cause parents to drop the subsidy, change their child care arrangements to more informal arrangements in
which the co-payment may not be demanded, or refuse additional hours of work if that would raise their
co-payment so much that they would see no net benefit.
The study in Washington will test the effects of an alternative co-payment structure that decreases the copayment burden for some families. The alternative co-payments were formulated so that amounts required
are either the same as or less than current co-payment amounts at every point on the fee scale. That is, all
participating families will either pay less or the same amount than they would under the current schedule
for the same level of income. From October 18, 2005 through November 7, 2005, 5,142 families
throughout the state who were applying for or reapplying for subsidies were randomly assigned to the
alternative co-payment schedule (approximately 2,000) or to the standard schedule (approximately 3,142).
The study will include an Implementations Study, an Impact Study, and a Cost-Benefit Study. Baseline
data will come from extant records. Data collection will include 35-minute parent interviews at 8, 16 and
24 months after random assignment to gather detailed child care and employment information. The study
will also rely on a 30-minute interview with the state’s coordinators from regional offices as well as
eligibility staff. Finally, the study will draw from extant administrative records on use of public benefits
and to obtain wage data reported to the unemployment insurance system.

How the Information Will Be Used, by Whom, and for What Purpose

a. Massachusetts
In cooperation with the State Lead Child Care Agency, the Massachusetts experiment will test the effects
of the Learningames curricula to address language development and the development of pre-literacy skills
for very young children in family child care settings. Learningames is an approach built on evidence that
children learn best in one-on-one interactions with a caregiver who is nurturing and responsive to the
child and who also provides rich language stimulation. If caregivers in home-based settings are given
appropriate tools, the intimate interactions that are possible in home-based care provide an ideal
opportunity for promoting children’s oral language, communication skills, and early phonological
awareness. The curriculum is well suited to support the caregiving practices that have been linked to
improving and providing good quality care, a goal of the CCDF. An earlier version of Learningames was
first used in the highly-successful Abecedarian program, as an approach for teachers to use with children
and their parents. It has been adapted for use in Even Start, a two-generation family literacy program,
both with the teachers in the center-based early childhood programs and with parents to use at home with
their children from birth to eight years. In its revised form, Learningames expands the number of games
and activities to reflect the newest research on the importance of language and emergent literacy. It is
appropriate for use with parents, family child care providers and early childhood teachers.
What makes the Learningames curriculum suitable for family child care is the fact that it is less
concerned with a particular set of lessons and content than with the relationship between the caregiver and
Abt Associates Inc.

OMB Clearance Request

the children. It focuses on teaching caregivers to be (a) responsive and nurturing and, (b) capable of using
observation of the child and appropriate stimulation to move the child from his/her developmental level to
a higher level of functioning. The stimulation is organized around a set of simple games that the provider
can play with the child one-on-one, as a way to encourage the provider to listen to the child, talk to the
child, respond to the child’s questions and actions, and help the child develop.
Learningames supports caregivers in providing an effective learning environment for children. The model
is organized around three strategies that are used sequentially, forming a responsive interaction between
caregiver and child. First, the caregiver notices what the child is doing as well as the child’s interests and
developmental level. Second, based on what he or she notices, the caregiver then nudges the child in an
appropriate way to help move his/her understanding to a higher level. Once the child responds, the adult
begins to narrate, to track what the child is doing and/or to guide the child’s behavior in new directions.
As the adult notices changes in the child’s behavior the cycle begins again and may repeat itself many
times even during a short interaction. To help caregivers find opportunities to engage in these types of
interactions with children, Learningames has a set of about 200 simple activities (each presented on a
card with both a picture of a caregiver and child in that activity and easily-understood suggestions about
how to initiate and extend the activity). The 200 “Learningames” focus on simple, everyday activities to
provide opportunities for the Notice-Nudge-Narrate cycle. In addition, the activities are carefully
constructed to promote children’s oral language development, communication skills, and early
understanding of sounds and letters.
Mentors are trained over a three-day period in how to work with providers both to implement
Learningames and to address other problems and needs they may have. Providers and mentors each attend
separate, initial group training sessions after which mentors visit the homes every two weeks, spending
one to two hours with each provider. During this visit, mentors introduce new games, model their use and
encourage providers to try using the approach with children. They answer questions and discuss problems
or issues the provider may have and suggest solutions.
Giving providers simple games to play individually with children, and training providers about the
importance of using language and of encouraging children to use language in an interactive cycle,
Learningames incorporates the most recent research on what supports children’s development and
delivers it in a way that seems well-suited to the family child care environment .
Study Components
The study includes two components:


An Implementation Study to document the process in which Learningames is implemented in
family child care homes in Massachusetts and describe challenges and barriers to implementation
that are encountered in this setting; and



An Impact Study to estimate the effects of Learningames on the quality of care provided in
family child care homes and children’s language and pre-literacy skills.

Implementation Study

Abt Associates Inc.

OMB Clearance Request

When combined with a well-designed impact study, a comprehensive Implementation study is an
indispensable evaluation component. In the overall context of the experiments, the research goals of the
Implementation analysis are to:
 Describe the intervention;
 Describe the degree to which the intervention was implemented as planned;
 Document relevant contextual factors; and
 Help interpret the findings of the impact study.
The principal task of the Implementation Study is to describe the intervention. Because observed impacts
are the result of differences in services actually experienced by treatment and control group providers
and children (i.e., the fidelity of the implementation), those experiences must be documented. There will
be three sources of information on fidelity/degree of implementation: home visitors that are implementing
Learningames will use a five-point scale with definitions at each point; as part of the implementation,
providers will keep feedback logs on which games they used with which children during the week; and, as
part of the independent data collection, Abt observers will also complete a simple observational measure
of fidelity.
In addition, we will investigate challenges to implementation and possible reasons for differences in
implementation through informed discussions during regular meetings with child care systems staff and
home visitors and through a Home Visitor Questionnaire. The Questionnaire, which home visitors for
both the treatment and control groups will be asked to complete, will collect information on caseload size,
frequency duration and purpose of home visitors, education and training, and language of the home
visitor.
Information on actual implementation of Learningames will help in the interpretation of impact findings.
For example, if Learningames fails to bring about its expected impacts on providers and children in
Massachusetts, the information collected through the Implementation Study should help us distinguish
among three possible reasons: Learningames was not implemented as planned and was not a true test of
the intervention; contextual factors counteracted the behavioral influence of the demonstration; or the
demonstration was implemented well and in a favorable context, but failed to change behavior in
expected ways. Clearly, if there are no impacts, each of these reasons conveys different policy
information.
Impact Study
The impact study will address the following research questions:


What is the impact of a research-based developmental curriculum (Learningames), designed to
enhance the quality of care and tailored to the needs of family child care provider, on providers’
behavior and interactions with children and on the language and literacy environment of the
home?



What is the impact of the intervention on children’s language and pre-literacy skills?

Family child care homes participating in the evaluation will be randomly assigned to one of two groups.
The treatment group will receive Learningames, in addition to generalized technical assistance provided
by family child care networks. Control homes will receive only the generalized technical assistance. We
will then estimate the impact of Learningames on both providers and children by comparing the
Learningames group and the control group on a set of key outcomes described below. The process of
Abt Associates Inc.

OMB Clearance Request

random assignment ensures that the groups of providers are the same across all measured and unmeasured
characteristics that could be related to the study outcomes. This means that the estimates of the impacts
of Learningames will be unbiased; statistically significant differences on the study outcomes that favor
the Learningames group will provide convincing evidence that the implementation of the curriculum
caused these positive differences.
The logic of the impact study is that Learningames will change what providers do with children, which in
turn will improve children’s developmental outcomes. In addition, it is assumed that there will be
differences in how well caregivers implement the Learningames approach, depending on the caregiver’s
background characteristics. To assess all parts of this logic model, the study will collect three kinds of
data will be collected for the evaluation: observations of provider behavior with children, a brief
interview with the provider about her education and experience, and assessments of children’s
development.
The provider observations will use the QUEST Caregiver Rating Scale, a standardized coding system for
rating the quality of early childhood settings, including either family child care or center care. The
Caregiver Rating Scale is based on the most recent research on instructional practices that are associated
with children’s development and learning. The rating scale focuses on caregiver warmth/responsiveness
and on caregiver support for the child’s development in four critical domains—cognitive development,
especially language development and early literacy; emotional development; social development; and
physical development.
The QUEST describes six main aspects of caregiver interactions in the home:
 Caregiver with Children
o Caring and responding (items 1-10)
o Using positive guidance and discipline (items 11-19)
o Supervision (items 20-23)
o Does no harm (items 24-28)


Supporting Social Emotional Development (items 29-36)



Supporting Play (items 37-40)



Supporting Cognitive Development
o Instructional style (items 41-45)
o Learning activities and opportunities (items 46-56)



Supporting Language Development and Early Literacy (items 57-67)



Television and Computers (items 68-69)

The QUEST is completed based on a minimum of 2.5 to 3 hours of observation in the home. Providers
will be observed up to four times over the two years, before the intervention and three additional times. A
subset of observations will use an additional rating scale – the Family Day Care Rating Scale (FDCRS) –
for the principal purpose of enabling a comparison of the two ratings.
Outcome data on the children will be collected using the Preschool Language Scale-Fourth edition (PLS4). The PLS-4 measures children’s receptive and expressive language. It is appropriate for children from
birth to six years. The norms include children’s total language, auditory comprehension, expressive
Abt Associates Inc.

OMB Clearance Request

communication, standard scores, percentile ranks, and language age equivalents. There are English and
Spanish language versions. The assessment takes between 20 and 45 minutes per child. Outcome data
will be collected at three points in time, at seven, 12 and 24 months after the intervention begins.
After OMB clearance has been obtained, a provider questionnaire will be administered. It will include
questions about the provider’s background, the level and types of education and training obtained,
languages spoken, and motivation for being a child care provider. Information from the provider
questionnaire will be used as covariates for the impact study. A second, shorter questionnaire will be
administered at 24 months of the study, to document additional education and training obtained by
providers, beyond the Learningames intervention, over the two years. Finally, for baseline information on
children’s developmental status, we will use extant data that are currently collected by the home visitors
employed by the family child care networks in the course of their regular work.
b. Illinois
The Illinois experiments, which are being conducted in close cooperation with the Illinois Department of
Human Services (IDHS) and with Action for Children of Illinois (AFC), will examine the effects of
obtaining and losing subsidies on low-income families with employed parents. Exhibit A2.1 shows a
conceptual framework that suggests the hypothesized outcomes from these experiments.
In 2003, 51% of families receiving child care subsidies in Illinois had received TANF within the last five
years (Anderson, Ramsburg, & Rothbaum, 2003). Families with employed parents apply and re-apply for
subsidies within a particular economic and general policy environment. If they are in a treatment group,
they will begin to receive or continue to receive subsidies, which may affect both their child care and
employment decisions. In particular, subsidy receipt may affect the type and nature of the child care they
select. By increasing families’ purchasing power for care, the subsidy may enable families to gain access
to child care that they think is more appropriate for their children. The state’s subsidy payments for this
care may increase the stability of the care arrangement. Similarly, the subsidy may allow parents to stay
at work and the higher income ceiling, may encourage them to take better jobs or work more hours, or
stay employed. In turn, more stable employment may lead to more stable child care arrangements. In
addition, a longer certification period for subsidies may further stabilize both employment and a child
care arrangement if the process of certification is indeed disruptive. Ultimately, child well-being might
be affected if extra employment increases family income and if the child’s experiences in care are more
positive because of the subsidy. Parental well-being might result from extra income, extra employment, a
more stable child care situation, or improved child well-being. Stable employment and income also may
reduce the need to rely on cash assistance and other public benefits.
The research will take place in Cook County, Illinois. Two-thirds of the state’s subsidy funding goes to
this highly urban area; therefore the study results will have strong implications for state policy. The
information may also be useful to other states that have highly urban areas to which the preponderance of
subsidy funding flows. In addition, in order to determine the degree to which the characteristics of the
study population is representative of other families receiving subsidies in the state, the study team will use
data from anonymous statewide subsidy records to compare the study population with 1) families near the
top of the eligibility ceiling in Cook County; and 2) families near the top of the subsidy eligibility ceiling
in non-Cook County areas.

Abt Associates Inc.

OMB Clearance Request

Exhibit A2.1
Conceptual Framework
Community Characteristics (child care market, employment market)
Family and Child Characteristics

Subsidy Policy:
Increased income eligibility at
certification or re-certification

Child care arrangements:
•Type
•Stability

Child’s experience
In care

Subsidy
receipt

Child
Child
Well-being
Well-being
Employment characteristics:
•Stability
•Hours of employment
•Hourly wage

Family Income

Parent
Well-being

The research questions inherent in this conceptual design follow:


For low-income, working families:
–

What is the impact of receiving a child care subsidy on parental employment, family
income and the receipt of public benefits?

–

What is the impact of receiving a subsidy on the type of child care chosen, the stability of
the arrangement, and on parents’ satisfaction with the child care?

–

What is the impact of losing a child care subsidy on parental employment, on family
income and on the receipt of public benefits?

–

What is the impact of losing a subsidy on the type of child care chosen, the stability of
the arrangement, and on parents’ satisfaction with the child care?

–

Does the length of the certification period for subsidies affect how long families receive a
subsidy?

The information collected in this study will provide policymakers in Illinois with much needed
information regarding the effects of raising the income eligibility ceiling for child care subsidies from
50% to 65% of SMI and will inform decisions on whether or not the state should change the income
eligibility ceiling. In addition, since all states must deal with the issue of where to set income eligibility
for child care subsidies, the information produced by this study will help inform state and local policy
decisions across the county.
Abt Associates Inc.

OMB Clearance Request

Study Components
The study includes three components: 1) an Implementation Study which will provide information about
the way in which the initiatives were mounted and helps in the interpretation of the findings from the
impact and benefit cost analyses; 2) an Impact Study which will identify the outcomes affected by
receiving a subsidy; and 3) a Benefit-Cost Analysis which will compare the costs of the initiative (e.g.,
the amount of funding used for subsidies) with the initiative’s benefits.
Implementation Study
When combined with well-designed impact and cost-benefit analyses, a comprehensive Implementation
study is an indispensable evaluation component. In the overall context of the experiments, the research
goals of the implementation analysis are to:





Describe the intervention;
Describe the degree to which the intervention was implemented as planned;
Document relevant contextual factors; and
Help interpret the findings of the impact and cost-benefits studies.

The principal task of the Implementation Study is to describe the intervention. Because observed impacts
are the result of differences in policies, operations, and services actually experienced by treatment and
control group families, those experiences must be documented. Although the interventions in Illinois
appear to be well-designed and well-specified on paper and relatively simple to implement, describing
what actually happens in the field and over time is a critical factor in assessing the faithfulness of the test,
as well as in determining the policy information conveyed by impact findings and their use in guiding
decisions about future policy choices.
Another key research goal of the Implementation Study will be to describe contextual factors that may
affect the outcomes of interest. For example, understanding the structure of the existing child care market
for low-income families, local economic conditions, and local expectations about child care arrangements
among low-income families, helps define the playing field in which impacts may happen. Communitylevel differences in those factors may lead to differences in expected impacts.
Information on actual demonstration program operations, family experiences and attitudes and contextual
factors also may help in the interpretation of impact findings. For example, if the demonstration fails to
bring about its expected impacts in Cook County, the information developed by the Implementation Study
should help us distinguish among three possible reasons: the demonstration was not implemented as
planned and was not a true test of the policy innovation; contextual factors counteracted the behavioral
influence of the demonstration; or the demonstration was implemented well and in a favorable context,
but failed to change behavior in expected ways. Clearly, the three different types of reasons for no
impacts convey different policy information.
Another use of Implementation Study results in the impact analysis is to help understand variations in
impacts over time. By describing and monitoring site changes over time in program implementation and
key contextual factors, the Implementation Study can provide information that can help the study generate
reasonable hypotheses about such impact variations.
These various goals lead to a number of research questions for the Illinois experiments, which are
included in Exhibit A2.2.
Abt Associates Inc.

OMB Clearance Request

Exhibit A2.2
Research Questions for the Implementation Study
1. Describing intervention design, rationale, planning, and start up
 What is the target group for the intervention?
 As designed, what are the treatment conditions and how do they differ from control group conditions?
 What are the intended impacts of the treatment? What is the state agency’s rationale for mounting the
treatment?
 Who was involved in the design process and what were the key design and operational issues?
 What preparations were required and completed before the demonstration could begin? For example:
Did workers require special training, and, if so, how was it conducted?
Were special forms and education materials developed?
Were administrative systems affected?
Were agency staffing or facilities changed in any way to accommodate the demonstration, and, if so,
how?
What other preparations were required?

Describing the operation of the intervention
 How do families apply for subsidies? Has this changed for the demonstration, and, if so, in what ways? Is







it different for different types of care and/or payment types (e.g., center-based vs. family-based; vouchers vs.
contracted slots)?
How do families re-apply for subsidies? Has this changed for the demonstration, and, if so, in what ways?
Is it different for different types of care and/or payment types (e.g., center-based vs. family-based; vouchers
vs. contracted slots)?
How are relevant families informed about the demonstration? How is informed consent solicited? How
many otherwise appropriate families decline the invitation to be in the demonstration?
How are families randomly assigned? How are families informed of RA results? How are RA results
recorded and monitored, including families that have left the subsidy system and return within the
demonstration?
What are the characteristics of the families in the demonstration, including socio-economic and
demographic characteristics, employment background, past child care patterns, and past subsidy use?
Is the demonstration operating as planned, and, if not, in what ways and why?
What changes, if any, were made in design or operations over the course of the demonstration?

Documenting relevant contextual factors for the demonstration
 What is the structure of the existing child care market for low-income families in the demonstration area,
including types of child care used, use of the subsidy system among eligible families, and patterns of child
care subsidy use?
 What are local economic conditions, including proportion of families eligible for subsidized care; local
industries, and types of jobs available to low-income families?
 What are local expectations and knowledge about child care arrangements among low-income families?

Identifying implications of processes for the impact and cost-benefits studies
 Are there aspects of the interventions design and plans for random assignment that have the potential to
affect the impacts of the intervention, and, if so, in what ways?
 Do changes or variations in demonstration operations over time have the potential to affect impacts over
time, and, if so, in what ways?
 Do differences in contextual factors have the potential to affect impacts over time or across sites, and, if so,
in what ways?

Impact Study
The purpose of the Impact Study is to estimate the effects of gaining or losing a subsidy on a number of
outcomes. The random assignment design enables us to identify the net effect of subsidies on these
outcomes by letting us draw comparisons between the treatment and control group.
The types of outcomes we will assess fall into three general categories:
Abt Associates Inc.

OMB Clearance Request



Employment and income outcomes (e.g., months in which employment occurred, average
numbers of weeks worked, stability of employment, average monthly earnings, changes in
earnings levels, average change in total family income)



Child care outcomes (e.g., reliability, flexibility, and stability of child care arrangements,
numbers of child care problems, cost of the child care, average amount parents spend on child
care)



Public assistance outcomes (e.g., percent with any use of TANF assistance, average months of
use of TANF, percent with any use of food stamps, average months use of food stamps)

The research design includes three telephone interviews, so that families are reached at Month 8,5 Month
16, and Month 24 after they enter the study. We assume that at least 25 percent of families will be
interviewed in person, most because they cannot be reached by telephone and a small percentage because
they have more than two children (for which an in-person interview will be needed to collect detailed
information on child care use).
In addition to assessing differences in these outcomes between the treatment and control group, the design
of the impact analysis will also enable us to identify differences within the treatment group of families
who receive subsidies under the enhanced eligibility guidelines with a 6-month certification period and
those that receive subsidies for a 12-month period.
Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with
expenditure information, to determine whether the costs of the experimental programs or policies are
justified, given their impacts.
The benefits and costs of the experiment will be assessed from several perspectives, including families
and children, the government, tax payers, and society.



From the perspective of families and children, we will identify the gains and losses to families
that result from use or loss of subsidies, in terms of parental employment and child care stability.
We also will identify changes in the use of public assistance, families’ child care costs, and total
family income.



From the government perspective, we will tally benefits and costs to federal, state, and local
governments. The benefits of child care subsidies could include increased tax revenue as a result
of increased employment of families that receive subsidies and reduced use of public assistance.
The primary costs to government will be the actual subsidy payments and the costs related to
administering them. The potential costs are increases in the use of public assistance.



From the taxpayers’ perspective, we will tally the benefits and costs to the general public. The
taxpayer will benefit from increased tax revenue since it offers the possibility of greater services
or lower taxes.



The perspective of society as a whole combines the perspectives of families and other taxpayers.
A net gain to society occurs when a gain to one taxpayer is not a loss to other taxpayers. For

The interviews will not commence until OMB clearance has been obtained. Therefore, families who were
recruited into the study in the period of March through May 2005 will have an initial interview that falls
between 12-14 months. This group includes 244 study participants.

Abt Associates Inc.

OMB Clearance Request

example, lower administrative costs resulting from longer certification periods represent a gain to
the government budget that does not come out of the pocket of any taxpayer. By contrast, child
care subsidy payments themselves represent neither a cost nor benefit to society per se because
one taxpayer receives them while other taxpayers pay for them.
One of the challenges of the cost-benefit analysis is that some of the benefits of receiving subsidies are
not valued in dollars. This is particularly the case with outcomes related to parental satisfaction with
child care.
Overview of Data Collection
The impact, implementation, and cost-benefit studies will rely largely on extant data from documents and
administrative records. In addition, a 35-minute interview will be conducted at three points over two
years to study participants. Exhibit A2.3 describes the data collection.
Exhibit A2.3
Summary of Data Collection

Type of Data Collection
Participant interviews
Extant administrative records (i.e., data from state
public assistance and child care data; state
unemployment insurance data)
State memoranda, documents and procedural
manuals
Census and Bureau of Labor Statistics data
Interviews with select key informants
Researcher notes and memoranda from study
development and start-up process

Study Components to be Informed by Data Collection
Cost-Benefit
Impact Study
Study
Implementation
Study
x
x
x
x
x

x
x

x
x
x

c. Washington
Study Components
Exhibit A2.4 shows the alternative co-payment scale and compares it with the existing scale. On the
existing scale – represented by the dark, solid line – families are placed in one of three co-payment tiers.
Tier 1 (income less than 82% of the FPL) is $15 per month, Tier 2 (income between 82 and 137.5% of
FPL) is $50 per month, and Tier 3 (income above 137.5% of FPL) is $50 per month + 44% of each
additional dollar of income above 137.5% of FPL. The alternative fee scale varies by the number of
children in care and is depicted by the other three lines. In Tier 1, the amount of the alternative payment is
also $15. In Tier 2, the co-payment rises gradually to $50, eliminating the notch; in Tier 3, the marginal
co-payment and the overall amount are reduced.

Exhibit A-2.4
Current and Proposed Monthly Alternative Parent Fee Scales

Abt Associates Inc.

OMB Clearance Request

The study includes three components:


An Implementation Study, which will provide information about the way in which the
initiatives were mounted and helps in the interpretation of the findings from the impact and
benefit cost analyses.



An Impact Study, which will estimate the effects of the three alternative co-payment schedules
on child care subsidy recipients; and



A Benefit-Cost Analysis, which will compare the costs of the initiative (e.g., the amount of
funding used for subsidies) with the initiative’s benefits.

The evaluation is guided by a conceptual framework that hypothesizes the specific outcomes that are
likely to be affected by a change in the co-payment schedule (Exhibit A2.5).

Abt Associates Inc.

OMB Clearance Request

Exhibit A2.5
Conceptual Framework
Community Characteristics (child care market, employment market)
Family and Child Characteristics

Subsidy Policy
:
Change in co-payment
scale

Child’s experience
In care

Child care
 Type
arrangements:
 Stability
Amount of
co-payment

Child
Child
Well-being
Well-being
Employment
characteristics:
Stability
 Hours of employment
 Hourly wage

Family
Income
Parent
Well-being

Abt Associates Inc.

OMB Clearance Request

Employed parents with children who need child care apply and re-apply for subsidies within a
particular economic and general policy environment. If they are in the treatment group for this study,
they will be subject to an alternative fee schedule that will likely result in reduced child care costs for
them, either immediately or as their income rises over time. The co-payment represents the cost of
care for subsidized families so it can be considered their “effective price” for child care. An eligible
parent may choose not to receive subsidies but instead select a child care setting with a full price that
is less than the required co-payment amount because this choice is more economically advantageous.
Therefore, a change in the co-payment amount—the effective price– may influence the type and
nature of the child care they select as well as the long-term desirability of subsidies.
In addition, since the alternative co-payment smoothes out the notches in the scale and reduces the
overall marginal rate for families at relatively higher income ranges, the alternative fee scale enables
families to keep more of their earnings and reduces any sharp changes in the amount of their net
earnings as their income rises. These changes may make sustained employment as well as increased
earnings more desirable. Therefore, the alternative fee scale may encourage parents to take better jobs
or work more hours or stay employed. In turn, more stable employment may lead to more stable
child care arrangements, as well as reduced use of public assistance. Ultimately, child well-being
might be affected if extra employment increases family income and if the child’s experiences in care
are more positive because of the subsidy. Parental well-being might result from extra income, extra
employment, a more stable child care situation, or improved child well-being. Stable employment
and income may reduce the need to rely on cash assistance and other public benefits.
Research Questions
The research questions inherent in this conceptual framework are:
For low-income, working families receiving subsidies:


What are the effects of the alternative co-payment scale on parental employment, family
income, length or receipt subsidy receipt and the receipt of other public benefits?



What are the effects of the alternative co-payment scale on the type of child care chosen,
the stability of the arrangement, and on parents’ satisfaction with the child care?



Does the impact of the alternative co-payment differ for families at different income
levels at the beginning of the study (i.e., whether they are in Tiers 1, 2, or 3 under the
current parent fee scale)?

Implementation Study
When combined with well-designed impact and cost-benefit analyses, a comprehensive
implementation study is an indispensable evaluation component. In the overall context of the
experiments, the research goals of the implementation analysis are to:





Abt Associates Inc.

OMB Clearance Request

The principal task of the Implementation Study is to describe the intervention. Because observed
impacts are the result of differences in policies, operations, and services actually experienced by
treatment and control group families, those experiences must be documented. Although the
intervention in Washington appears to be well-designed and well-specified on paper and relatively
simple to implement, describing what actually happens in the field and over time is a critical factor in
assessing the faithfulness of the test, as well as in determining the policy information conveyed by
impact findings and their use in guiding decisions about future policy choices.
Another key research goal of the Implementation Study will be to describe contextual factors that
may affect the outcomes of interest. For example, understanding the structure of the existing child
care market for low-income families, local economic conditions, and local expectations about child
care arrangements among low-income families, helps define the playing field in which impacts may
happen. Community-level differences in those factors may lead to differences in expected impacts.
Information on actual demonstration program operations, family experiences and attitudes and
contextual factors also may help in the interpretation of impact findings. For example, if the
demonstration fails to bring about its expected impacts in Washington, the information developed by
the Implementation Study should help us distinguish among three possible reasons: the
demonstration was not implemented as planned and was not a true test of the policy innovation;
contextual factors counteracted the behavioral influence of the demonstration; or the demonstration
was implemented well and in a favorable context, but failed to change behavior in expected ways.
Clearly, the three different types of reasons for no impacts convey different policy information.
Another use of Implementation Study results in the impact analysis is to help understand variations in
impacts over time. By describing and monitoring site changes over time in program implementation
and key contextual factors, the Implementation Study can provide information that can help the study
generate reasonable hypotheses about such impact variations.
These various goals lead to a number of research questions for the Washington experiment, which are
included in Exhibit A2.6.

Abt Associates Inc.

OMB Clearance Request

Exhibit A2.6
Research Questions for the Implementation Study
1.

Describing intervention design, rationale, planning, and start up
 What is the target group for the intervention?
 As designed, what are the treatment conditions and how do they differ from control group conditions?
 What are the intended impacts of the treatment? What is the state agency’s rationale for mounting the
treatment?
 Who was involved in the design implementation and what were the key design and operational issues?
 What preparations were required and completed before the demonstration could begin? For example:
Did workers require special training, and, if so, how was it conducted?
Were special forms and education materials developed?
Were administrative systems affected?
Were agency staffing or facilities changed in any way to accommodate the demonstration, and, if so,
how?
What other preparations were required?

Describing the operation of the intervention
 How do families apply for subsidies? Has this changed for the demonstration, and, if so, in what ways? Is







it different for different types of care and/or payment types (e.g., center-based vs. family-based; vouchers vs.
contracted slots)?
How do families re-apply for subsidies? Has this changed for the demonstration, and, if so, in what ways?
Is it different for different types of care and/or payment types (e.g., center-based vs. family-based; vouchers
vs. contracted slots)?
How are relevant families informed about the demonstration? How is informed consent solicited? How
many otherwise appropriate families decline the invitation to be in the demonstration?
How are families randomly assigned? How are families informed of RA results? How are RA results
recorded and monitored, including families that have left the subsidy system and return within the
demonstration?
What are the characteristics of the families in the demonstration, including socio-economic and
demographic characteristics, employment background, past child care patterns, and past subsidy use?
Is the demonstration operating as planned, and, if not, in what ways and why? Is it reaching the families
targeted for the intervention?
What changes, if any, were made in design or operations over the course of the demonstration?

Identifying implications of implementation for the impact and cost-benefits studies
 Are there aspects of the interventions design and plans for random assignment that have the potential to
affect the impacts of the intervention, and, if so, in what ways?
 Do changes or variations in demonstration operations over time have the potential to affect impacts over
time, and, if so, in what ways?
 Do differences in contextual factors have the potential to affect impacts over time or across sites, and, if so,
in what ways?

Impact Study
Abt Associates Inc.

OMB Clearance Request

The purpose of the Impact Study is to estimate the effects of reducing parent co-payments for child
care subsidies on a number of outcomes. The random assignment design enables us to identify the
net effect of subsidies on these outcomes by letting us draw comparisons between the treatment and
control group.
The types of outcomes, or dependent variables, we will assess fall into four general categories:


employment and earnings;



family income and public assistance;



child care outcomes; and



subsidy and public assistance expenditures.

The independent variable is the family’s assignment to the treatment or control group. The dependent
variables are the outcomes that will be assessed as a result of the assignment to the treatment or
control condition. Exogenous variables are the characteristics of the family (e.g., family size, racial
composition, geographic location) that are collected before random assignment and therefore are
unaffected by whether they are in the treatment or control group. The exogenous variables are used
to partial out the effects of any pre-existing differences between the treatment and control groups.
The inclusion of such variables increases the power of the evaluation to detect changes that result
from the treatment. Exhibit A2.7 lists all of the variables that will be used in the impact analysis.

Exhibit A2.7
Independent, Exogenous, and Dependent Variables
INDEPENDENT VARIABLE

Assignment to Alternative or Existing Co-Payment Schedule

EXOGENOUS VARIABLES

Family characteristics
Household size
Number of parents in household
Number of adults in household
Ages of parents
Number of children
Ages of children
Immigration status
Race/Ethnicity
Education level(s) of parents
Prior receipt of TANF cash assistance

DEPENDENT VARIABLES

Employment characteristics
Number of employers at point single point in time
Total number of employers
Hourly wage per employer
Weekly earnings
Average number of weekly hours worked
Average monthly earnings
Hourly wage at 24 months
Change in hourly wage from baseline
Change in number of weekly hours worked
Average monthly earnings
Number of breaks in employment
Average length of break

Abt Associates Inc.

OMB Clearance Request

Exhibit A2.7
Independent, Exogenous, and Dependent Variables
Number of work days lost because of child care problems
Number of weeks in which non-traditional schedule occurred
Number of quarters in which any employment occurred
Number of consecutive quarters in which employment did not occur
Number of hours worked per quarter
Number of employers per quarter
Number of hours worked per quarter
Family income and public assistance
Total annual family income
Income from child support
Dependent Care Tax Credit taken
Child care characteristics
Type of child care arrangements for all children in family
Number of child care arrangements per family
Duration of individual child care arrangements
Duration of individual subsidized child care arrangements
Monthly amount paid for child care*
Level of child care flexibility
Level of child care reliability
Number of problems with child care arrangements
Average number of breaks/interruptions in child care arrangements
Average monthly co-payment amount
Cumulative co-payment amount over study period
Subsidy and public assistance expenditures
Any receipt of TANF cash assistance
Average number of months on TANF cash assistance
Average monthly amount of TANF cash assistance
Unit administrative cost of TANF cash assistance
Any receipt of food stamps
Average number of months of food stamp receipt
Average monthly amount of food stamp receipt
Prior receipt of food stamps
Unit administrative cost of food stamps
Per family cost to state of subsidized arrangements
Per child cost to state of subsidized arrangements
Duration of subsidy receipt by arrangement
Number of subsidized arrangements per family
Unit administrative cost of subsidized arrangement

Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with
expenditure information, to determine whether the costs of the experimental programs or policies are
justified, given their positive impacts.
The benefits and costs of each of the experiment will be assessed from several perspectives, including
families and children, the government, tax payers, and society.

Abt Associates Inc.

OMB Clearance Request



From the perspective of families and children, we will identify the gains and losses to
families that result from a decrease in their co-payment, in terms of parental employment
and child care stability. We also will identify changes in the use of public assistance,
families’ child care costs, and total family income.



From the government perspective, we will tally benefits and costs to federal, state, and
local governments. The benefits of reduced co-payments could include increased tax
revenue as a result of increased employment of families that receive subsidies and
reduced use of public assistance. The primary costs to government will be the actual
subsidy payments and the costs related to administering them.



From the taxpayers’ perspective, we will tally the benefits and costs to the general
public. The taxpayer will benefit from increased tax revenue since it offers the
possibility of greater services or lower taxes.



The perspective of society as a whole combines the perspectives of families and other
taxpayers. A net gain to society occurs when a gain to one taxpayer is not a loss to other
taxpayers.

Use of Improved Technology to Reduce Burden

a. Massachusetts
The use of improved technology has been incorporated into the data collection wherever possible to
reduce respondent burden. For instance, information about the number of subsidized children
enrolled in the homes and the duration of their enrollment will be obtained from centralized databases
maintained by each family child care system.
b. Illinois
As summarized in Exhibit A2.3, the information to be collected for this study will come from
documents, existing records and data, in-person or telephone interviews with parents, individual and
small-group interviews with providers and agencies. The use of improved technology has been
incorporated into the data collection wherever possible to reduce respondent burden. When
information is available to the contractor from a centralized, computerized source, such information
has not been included in the data collection instruments. Specifically, automated information will be
used to provide data about the use of public assistance programs, data on the characteristics of the
underlying population will be taken from data collected by the Census Bureau and the Bureau of
Labor Statistics Labor Market Data, and CATI systems will be used in the telephone interviewing.
c. Washington
Much of the information to be collected for this study will come from documents, existing records
and data, and telephone interviews with parents. The use of improved technology has been
incorporated into the data collection wherever possible to reduce respondent burden. When
information is available to the contractor from a centralized, computerized source, such information
has not been included in the data collection instruments. Specifically, automated information will be
utilized to provide data about the use of public assistance programs and CATI systems will be used in
the telephone interviewing.

Abt Associates Inc.

OMB Clearance Request

Efforts to Avoid Duplication

There are no other studies currently underway to examine the effects of Learningames or any other
curriculum for family child care providers in the Commonwealth of Massachusetts. There are no
other studies currently underway to examine the effects of providing subsidies in Illinois to families
who would otherwise be ineligible for them according to current eligibility limits in Illinois. There are
no other studies currently underway to examine the effects of reducing parent co-payments in the
State of Washington.

Efforts to Minimize Burden on Small Businesses or Other Small
Entities

a. Massachusetts
Every attempt has been made to reduce the burden placed on family child care providers participating
in the experiment. To this end, information on child care quality and fidelity of implementation of
Learningames will be collected through direct observation rather than interviews or questionnaires
that address provider behavior.
b. Illinois
No small businesses are involved in this study. However, every attempt has been made to reduce the
burden placed on families participating in the experiments. The evaluator will:


collect information on benefit receipt from standard applications for child care subsidies and
administrative records for TANF, food stamps, and unemployment insurance;



collect information on administrative practices from brief interviews with staff from the
Illinois Department of Human Services and Action for Children of Illinois and select other
local informants; and



collect information on employment and educational characteristics from a brief parent survey.

c. Washington
No small businesses are involved in this study. However, every attempt has been made to reduce the
burden paced on families participating in the experiments. The evaluator will:



collect information on benefit receipt from standard applications for child care subsidies
and administrative records for TANF, food stamps, and unemployment insurance; and



collect information on income, employment and educational characteristics from a brief
parent survey.

Consequences if the Information is Not Collected

a. Massachusetts
Currently, states expend funds on initiatives to improve family child care, but have little guidance on
what constitutes an effective intervention. Compared with center-based care, for which there is an
expanding body of knowledge based on rigorous experimental research, family child care has been
Abt Associates Inc.

OMB Clearance Request

relatively neglected, except for descriptive studies. This study will provide reliable information on the
effectiveness of a research-based intervention that is particularly suited to family child care.
Observational data on caregivers and the home environment is essential in understanding the impact
or lack of impact on children of the intervention, since the caregiver’s behavior mediates those
impacts. If the intervention results in behavior and interactions that promote children’s development,
we might expect to see a positive impact on children’s performance on standardized tests of language
and preliteracy skills. If, on the other hand, the intervention fails to make changes in caregivers that
are greater than those that occur in the control group as a result of conventional assistance, we would
not expect a measurable difference in the outcomes for children in the two groups of homes. We do
not expect to see immediate or substantial change in caregiver behavior. The observations are
repeated over the course of the study so that we can understand the rate at which and the ways that
caregiver behavior changes, with and without an intervention.
Background information on caregivers will be collected once to provide information necessary to
construct covariates for the regression models used in the impact analysis.
Child outcome data will be gathered through administration of standardized assessments at three
points in time. In this case, the repeated measures are necessary to ensure that we capture information
on children who leave the home before the end of the two-year-period of the study. While these data
collections will generally occur at set intervals, like the observations, we will ask providers to let us
know when a child is leaving so that we can schedule an assessment if the regularly scheduled data
collection will miss that child.
b. Illinois
The data for which clearance is being requested for the Illinois experiment will be collected at three
time points over a two-year period for each family. Baseline data on family characteristics will be
taken from state records collected as part of the application or recertification process. Parents will be
interviewed at 8, 16 and 24 months to obtain follow-up data on family characteristics, employment
and educational characteristics, and family income. It will be necessary to speak with them at
intervals of reasonable lengths because we believe that it will be too difficult for participants to recall
work and child care transitions without sufficient detail for a period longer than eight to ten months.
For control group members, we will not be able to augment information on child care use collected
through the interview with extant data from administrative records because they will not be using
child care subsidies unless their incomes decrease and they become eligible for the standard program.
Therefore, without these follow-up interviews it will be impossible to estimate the effects of the
intervention.
c. Washington
The data for the random assignment experiment in Washington for which clearance is being requested
will be collected at three time points over a two-year period. Parents will be interviewed at eight, 16,
and 24 months to obtain follow-up data on family characteristics, employment and educational
characteristics, family income, and child care characteristics. Baseline data on family characteristics,
income, employment, and child care characteristics will be collected from administrative records and
these records will be used for the analysis for the two-year period of the study unless parents deny
researcher access to them. Parent interviews will be necessary to augment administrative data
because the information collected through administrative systems is insufficient to measure fully the
impacts of the changed co-payment policy, especially for families that leave the subsidy system
Abt Associates Inc.

OMB Clearance Request

during the course of the two-year intervention. It also will be necessary to speak with parents at
intervals of reasonable lengths because we believe that it will be too difficult for participants to recall
work and child care transitions without sufficient detail for a period longer than eight-10 months.
Without these follow-up interviews it will be impossible to estimate the effects of reducing parent copayments.

Special Circumstances Requiring Collection of Information in a
Manner Inconsistent with Section 1320.5 (D) (2)

All three proposed data collections are consistent with the guidelines set forth in Section 1320.5 (D)
(2).

Efforts to Consult with Persons Outside the Agency

An announcement of the Administration for Children and Families’ (ACF) intent to seek approval to
collect this information provided an opportunity for public comment on this study. This
announcement was published in the Federal Register, September 21, 2005, Volume 70, Number 182,
pages 55402-55403 and specified a 60-day period for comment ending November 20, 2005. No
comments or suggestions were received in response to this notice. A copy of the relevant Federal
Register announcement is provided in Appendix A.
Several individuals were consulted in developing the design for the studies and identifying the types
of data to be collected. Their feedback was obtained through telephone conversations, on-site
meetings of the full study staff, and meetings with individual consultants. The names and affiliations
of these individuals include:
Bobbie Weber
Coordinator
Oregon Child Care Research Partnership
Oregon State University
236 NW 28th Street
Corvallis, OR 97330-5307
David Edie
State Technical Assistance Specialist
10 S Blackhawk Ave
Madison, WI 53705-3317
Deanna T. Schexnayder
Associate Director
Ray Marshall Center for the Study of Human Resources
LBJ School of Public Affairs
The University of Texas at Austin
3001 Lake Austin Blvd., Suite 3.200
Austin, TX 78703
512-471-2193 (voice)

Abt Associates Inc.

OMB Clearance Request

Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Louise Stoney
NCCIC Technical Assistance Specialist
308 Thais Road
Averill Park, NY 12018
518-674-5635
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA

Payments to Respondents

a. Massachusetts
Family child care homes will be given a $20 gift certificate for children’s educational materials each
time researchers conduct an observation and assess children in their home, to thank them for
accommodating the researchers in their homes. Every effort will be made to minimize the disruption
in their homes caused by the data collection.
b. Illinois
Participants in the two experiments in Illinois will be interviewed three times over the two-year study
period. Three-quarters of these interviews will be conducted by telephone; the remainder will be
completed in-home by field interviewers. All respondents completing interviews will receive a $20
voucher or gift certificate for each interview. Families who return postcards indicating a change of
address also will receive a $5 voucher or gift certificate.
c. Washington
Participants in the Washington experiment will be interviewed three times over the two-year study
period. Three-quarters of these interviews will be conducted by telephone; the remainder will be
completed in-home by field interviewers. All respondents completing interviews will receive a $20
voucher or gift certificate for each interview.

Abt Associates Inc.

OMB Clearance Request

A10 Assurances of Confidentiality Provided to Respondents
Abt Associates is fully committed to protecting the privacy of respondents at all points in the data
collection and analysis Implementation. The following data handling and reporting procedures will
be used to maintain the privacy of all individual respondents:
a. Massachusetts
 Each family child care provider and child who participates in the study will be assigned a
unique identification number which will be used throughout the study. All data collected
will be attached to and stored with this number, rather than to any name or other
identifying information.


Files linking ID numbers to the names of individuals will be kept in a locked file to
which access is restricted. Access to this information will be limited to field
staff/interviewers who need this information to schedule interviews and to Abt senior
staff who will monitor the data collection.



The importance of maintaining privacy will be stressed during data collection training.
All Abt staff will be required to sign a statement that affirms their understanding of the
assurance of privacy and pledges to maintain that privacy.



Coding documents and computer files will refer to respondents by their ID numbers only.
No names or other identifying information will appear on data files. Access to all data
bases will be protected by passwords and restricted to staff involved in the data analysis.



No data will ever be reported by the contractor in any form that can be identified with
individual respondents.

b. Illinois
 Prior to being enrolled in the study, parents sign a consent form that authorizes the
researchers to contact them for the three interviews and indicates all information will be
kept private.


Each family who participates in the study will be assigned a unique identification number
which will be used throughout the study. All data collected will be attached to and stored
with this number, rather than to any name or other identifying information.



The importance of maintaining privacy will be stressed during interviewer training. All
Abt staff will be required to sign a statement that affirms their understanding of the
assurance of privacy and pledges to maintain that privacy.



No data will ever be reported by the contractor in any form that can be identified with
individual respondents.

Abt Associates Inc.

OMB Clearance Request

c. Washington
 Each family who participates in the study will be assigned a unique identification number
which will be used throughout the study. All data collected will be attached to and stored
with this number, rather than to any name or other identifying information.


The state will only release contact information to Moore & Associates, which will use it
for the telephone interview. Abt Associates Inc. and MDRC will only use a study
identification number to track participants. Files linking ID numbers to the names of
individuals will be kept in a locked file to which access is restricted. Access to this
information will be limited to field staff/interviewers who need this information to
schedule interviews and to Moore & Associates senior staff who will monitor the data
collection.



The importance of maintaining privacy will be stressed during interviewer training. All
research staff will be required to sign a statement that affirms their understanding of the
assurance of privacy and pledges to maintain that privacy.



No data will ever be reported by the contractor in any form that can be identified with
individual respondents.

Public and Restricted Use Data
Abt will create and deliver restricted access and public use data files on a CD-ROM for the impact
and cost-benefit analyses. These files must be in ASCII. ACF expects to make restricted access files
available at a secure data center, Research Connections (please see the attached memo discussing the
release of public and restricted use data files on the Research Connections website in Appendix B).
Access to the files will be restricted to researchers whose projects are approved by ACF. These data
files will not contain direct identifiers such as names or social security numbers but will contain the
data used in the impact and cost-benefit analyses. Abt will employ masking and other strategies as
appropriate to ensure the privacy of the sample members is ensured.
The restricted access files must contain sufficient information to allow duplication by other
researchers of analyses performed by Abt unless privacy concerns require aggregating or limited
detailed data. Abt Associates will also prepare and deliver tables describing each variable contained
in the restricted access and public use data files. The tables must include information such as
frequencies and means. Abt will also prepare and deliver documentation to accompany the public use
files that describe the following: 1) each variable on the file; 2) how to use and access the file; and 3)
any editing strategies employed. The CD-ROMS should include all the reports, in Adobe formats.

A11 Justification for Questions of a Sensitive Nature
For all three studies, the interviews that will be conducted do not contain any questions of a sensitive
nature.

Abt Associates Inc.

OMB Clearance Request

A12 Estimates of Respondent Burden
a. Massachusetts
Exhibit A12.1a presents estimates of respondent burden for the Massachusetts surveys. Exhibit
A12.1b presents estimates of burden to family child care network staff for participating in training to
conduct baseline observations and to do record abstraction.

Abt Associates Inc.

OMB Clearance Request

Exhibit A12.1a
Respondent Burden for Massachusetts Interviews
Number of
Family Child
Care
Providers
Year Respondents

Interviews
Per Year

Hours Per
Interview

Hours Total
Burden:
Provider
Interviews
56

2006

350

.16

2007

2008

350

.16

Total

700

Average Hours Hours Total Numbers of
Assessments
Per
Burden:
Home Visitors Interview Per
Per Year
Assessment Assessments Respondents
Year

700

350

700

350

700

350

406

112

2100

1,050

Exhibit A12.1a
Respondent Burden for Training Massachusetts
Network Staff for Baseline Observations
and Administrative Data Abstraction
Number of
Family Child
Care
Network
Staff
Number of
Year Respondents
Efforts

Abt Associates Inc.

Hours Total
Burden: Home Total Burden:
Visitor
Interviews +
Interviews
Assessments

Number of
Children
Respondents

Hours Per
Effort

Hours Total
Burden:

2006

240

2007

2008

Total

320

OMB Clearance Request

Hours Per
Interview
.16

416

1,172

The estimated annual burden for respondents in the Massachusetts study is 497.3 hours (total burden
divided by number of years: 1,172+ 320/3). The total annual response is 2,924 (average annual
respondents multiplied by the average annual response).
b. Illinois
Exhibit A12.2a presents estimates of respondent burden for the Illinois study. Please note it also includes
pre-testing the interview with up to 30 respondents. Exhibit A12.2b presents estimates of respondent
burden for obtaining baseline data and administrative records from the state agency as well as the burden
for the process study interviews that will be undertaken with a maximum of 15 individuals. The estimated
annual burden for respondents is 1,381 hours (total burden hours divided by number of years:
2,627+135)/2). The total annual response is 12,504 (average annual respondents multiplied by the
average annual response).
Exhibit A12.2a
Estimates of Respondent Burden for Illinois Survey
Interview
Wave
Pre-test
1
2

Total

No. of
Respondents
30
1,600
1,500
3,130

Interviews
per year
1
1
1
3

Hours Per
Interview
0.58
0.58
0.58
0.58

Burden (in
hours)
17
928
870
1,815

2007

1,400

0.58

812

Year
2006

Total
Both
Years

4,530

2,627

Exhibit A12.2b
Estimates of Respondent Burden for Illinois Records
Abstraction and Process Study Interviews
Year

No. of State
Staff
Respondents

2006

1
15

Total

2007

Total
Both
Years

Type and # of
Efforts
Records
abstraction
Process Study
Interviews

Records
abstraction

Interviews per
year

Hours Per
Effort

Burden (in
hours)

135

c. Washington
Exhibit A12.3a presents estimates of respondent burden for the interviews for Washington study. Exhibit
A12.3b presents estimates of respondent burden for the administrative records data abstraction. The
estimated annual burden for respondents is 1,815 hours (total burden hours divided by number of years:

Abt Associates Inc.

OMB Clearance Request

3,510+120)/2). The total annual response is 8,128 (average annual respondents multiplied by the average
annual response).
Exhibit A12.3a
Estimates of Respondent Burden for
Interviews
Year

No. of Parent
Respondents

Interviews
per Year

Hours per
Interview

No. State
Interviews Hours /
Respondents per Year
Interview

2006

2000

0.58

0.50

2007

2000

0.58

0.50

Total

4000

TOTAL
BURDEN
1,175
hours
2,335
hours
3,510
hours

Exhibit A12.3b
Estimates of Burden for Data Abstraction
Year

2006
2007
Total

No. of
State Staff
Respondents
2
2
4

Abstractions
per Year

Hours per
Staff

1
2
3

20
20

TOTAL
BURDEN
40
80
120

Across all three studies the total estimated annual burden for respondents is 3,693 hours (sum total of
estimated annual burden hours for respondents for each study: 497 + 1,381 + 1,815), the total annual
number of respondents is 5,280 (974.3 +2,273.5+ 2032), and the total annual response is 23,556 (the sum
of the [annual average respondents multiplied by the average annual responses]: 2,924 + 12,504 + 8,128).
Exhibit A12.4 includes all of the actual and average numbers from which the total annual average
numbers discussed above were calculated.
Exhibit A12.4
Total Number of Respondents and Responses across all three
States

State
MA
IL
WA

Total No. of
Respondents
(Average in
parentheses)
2,924 (974.7)
4,547 (2,273.5)
4,064 (2032)

Total No. of
Responses
(Average in
parentheses)
9 (3)
11 (5.5)
8 (4)

Total No. of
Burden Hours
(Average in
parentheses)
1,492 (497.3)
2,762 (1,381)
3,630 (1,815)

A13 Estimates of the Cost Burden to Respondents
For all three studies, there are no direct monetary costs to respondents other than their time to participate
in the study.

Abt Associates Inc.

OMB Clearance Request

A14 Estimates of the Cost to the Federal Government
The information collection activity and associated forms have been developed in the performance of
DHHS contract number 233-01-0012. The period of performance of the project is from September 30,
2001 through September 30, 2008. The costs associated with the data collection activity for which
clearance is requested are as follows:
a. Massachusetts
Development of Data Collection Instruments
Data Collection

$0
$854,000

b. Illinois
Development of Data Collection Instruments
Data Collection

$32,000
$854,000

c. Washington
Development of Data Collection Instruments
Data Collection

$35,000
$450,000

A15 Reasons for Any Program Changes or Adjustments
All three studies are new projects.

A16 Plans for Tabulation, Statistical Analysis and Publication
a. Massachusetts
Analysis Plan
There are two primary sources of data to be analyzed in this study: (1) observation measures of providers’
behaviors and interactions with children and (2) measures of children’s language development. Questions
about the impact of Learningames on provider behavior and child outcomes will be answered by
estimating the mean difference between the treatment and control groups on the outcome measures at
each post-assessment. One of the primary benefits of a randomized experiment is that it produces
unbiased estimates of program impacts. Because providers have been randomly assigned, on average the
treatment and control groups will be the same across all dimensions except for the presence or absence of
the Learningames curriculum. As a result, any differences in average outcomes can be attributed to the
implementation of Learningames. It should be noted, however, that this estimate could become biased
because of poor implementation of the randomized design. Therefore, it is critical to take steps to ensure
that the fidelity of the design is preserved, both in the execution of the study and analytically. Four
common sources of bias are (a) post-assignment attrition, (b) crossovers of group members from
treatment and control groups, (c) contamination of the control group, and (d) treatment group nonparticipation or under-participation (Shadish, Cook & Campbell, 2002). While the study is designed to
minimize the burden on providers, it is possible that homes could drop out of the study after being
assigned to a condition, particularly as this is a longitudinal study. This would result in both a loss of
statistical power, and, in the case of non-random attrition, would bias estimates. Abt Associates has an
impressive record of minimizing such attrition, and has established strategies for retaining participants
throughout the course of longitudinal studies.
Abt Associates Inc.

OMB Clearance Request

Crossover and contamination, the second and third potential threats, both result in control providers or
children being exposed to the treatment either by switching groups or by being in close proximity to
treatment children or their providers. This seems unlikely in this study, both because providers have little
contact with each other and because the agency staff assigned to visit treatment providers will not visit
any control providers. This will eliminate any temptation to introduce elements of Learningames to
control providers. It is possible that a child could switch from a control provider’s home to a treatment
home during the study. We will identify the children in each home, as well as any previous homes they
attended, so that we can include a measure of crossover in our final analyses.
The final threat to the design is of greater concern. While the goal is for each child in a treatment home to
receive the complete Learningames curriculum, for a variety of reasons (e.g., absences, differences in
provider practices, timing of the child’s enrollment etc.), it is possible that there will be some children
who will not receive the full Learningames treatment. If there are a large number of untreated or undertreated children in our treatment group, we will underestimate the true impact of Learningames.
Therefore, we will examine whether variation in program effects across children is related to differences
in the implementation of the program itself, including child attendance and provider practices as well as
other variables related to the fidelity of the implementation of the curriculum.
Estimating Impacts on Children and Providers. In each home in the sample, there will be a single
provider and a small number of children ranging in age at baseline from 2 to 36 months. We expect the
number of children to range from one to four, and these children will therefore be clustered within homes.
To estimate the impact of Learningames on children while accounting for the fact that they are clustered
within child care homes, we will fit a separate, two-level hierarchical linear model for each child outcome
of interest. The model will include baseline covariates at both the child and provider level, including
demographic characteristics and other characteristics known to be associated with study outcomes. 6
The model is specified as follows:
Level-1 (child):

where:
is the outcome measure (e.g., PLS-4) for child i, in home j,
is the mean of the outcome in home j,
BLij is the measure of the outcome at baseline for child i, in home j,
is the coefficient associated with the baseline measure,
are a set of child-level covariates (1-h) for child i, in home j,
6

Although random assignment means that in expectation the characteristics of the children and homes in the
treatment and control groups will be the same, in our particular sample of children and homes there may in fact
be differences between the two groups on observable characteristics. Therefore, we will include demographic
and other child and provider characteristics in our models to control for any observable differences between the
two groups.

Abt Associates Inc.

OMB Clearance Request

are the coefficients associated with those covariates, and
is the unique error term associated with child i, in home j.

Level-2 (provider/home):

where:
is the mean of the outcome (e.g., PLS-4) in control homes,
LGj is a school-level group indicator variable equaling 1 for Learningames homes and 0 for control
homes,
is the mean difference between the Learningames and control homes on the outcome measure,
are a set of provider level covariates (1-h) for home j,
are the coefficients associated with those covariates, and
is the unique error term associated with home j.

The coefficient associated with the Learningames indicator (

) can then be directly interpreted as the

impact of Learningames on the outcome measure. If this coefficient is positive and significant, we will
conclude that Learningames has had a positive impact on the children in family child care homes.
To estimate the impact of Learningames on providers, we will fit separate OLS regression models for
each provider outcome. These models will also control for baseline measures as well as provider
demographics and other characteristics known to be related to study outcomes, such as the size of the
child care home.
The model is specified by the following equation:

where:
is the outcome measure for provider i,
is the mean of the outcome for control providers,
LGi is an indicator variable equaling 1 for providers in the Learningames group and 0 for those in the
control group,
Abt Associates Inc.

OMB Clearance Request

is the difference between the Learningames and control group on the outcome measure,
BLj is the measure of the outcome at baseline for provider i,
is the coefficient associated with the baseline measure,
are a set of provider-level covariates (1-h) for provider i,
are the coefficients associated with those covariates, and
is the unique error term associated with provider i.

The coefficient

, can then be directly interpreted as the impact of Learningames on the outcome

measure, controlling for a set of provider characteristics known to be related to the study outcomes. We
will then conduct a hypothesis test to assess whether this estimate is statistically different from zero in
favor of the Learningames group (a one-tailed hypothesis test).
Study Schedule
The planned time schedule for the study is as follows:
Expected OMB approval
Recruitment and random assignment begins
Random assignment ends
First data collection: provider observations
Second data collection provider observations +
Child assessments
Final data collection
Final Report

May 2006
May 2005
July 2007
June 2006
May/June 2007
January 2008
December 2008

b. Illinois
Analysis Plan
There are two major sets of statistical analyses that will be completed as part of the evaluation: impact
analyses and cost-benefit analyses. The analysis plan for each is provided below.
Impact Analysis
The purpose of the impact analysis is to estimate the effect of the treatments on outcomes such as
employment, child care quality, and child care satisfaction relative to the control group. Random
assignment ensures the internal validity of the analysis based on this comparison: we can assume that the
three groups are equivalent, and that the outcomes observed for the control group in each experiment
represent on average the outcomes that would have been experienced by members of the treatment group,
absent the experiment.
Although simple comparisons of means and proportions would yield consistent impact estimates, more
precision can be obtained by use of multivariate methods. Within each group, outcomes will vary
because of other underlying factors, such as mother’s employment opportunities, children’s ages, and so
on. It is easier to detect differences in central tendencies between two groups if each group is more
Abt Associates Inc.

OMB Clearance Request

homogeneous, i.e., if their outcomes are more tightly clustered. Regression-adjusted outcomes are indeed
more tightly clustered to the extent that the covariates have explanatory power. For example, a partial R 2
of 0.25 due to the covariates implies that the variance of the regression-adjusted mean is only threequarters as large as the variance of the raw mean, and this reduction correspondingly increases the
statistical power of the comparison.
Regression adjustment will therefore be used for each outcome, with a technique appropriate to the form
of the outcome. The goal of these models is not to “explain” the outcomes; these will be reduced-form
models and the coefficients, other than those on the treatment indicators, will in general be
uninterpretable. Baseline data will include information such as the ages and relationships of family
members, work and training activities of other parents or stepparents in the household, ethnicity and
citizenship of children for whom subsidies are sought, family participation in TANF and other federal
cash income programs, and child support received.
The form of the regression equation for a continuous outcome will therefore be:
yi = b0 + (b1 × T6i )+ (b2 × T12i)+ Σj (b3j × Xij) + εi,
where yi = value of outcome for family i,
T6i = indicator that family i is in the treatment group, with a 6-month recertification period,
T12i = indicator that family i is in the treatment group, with a 12-month recertification period,
Xij = value of baseline variable j for family i, and
εi = residual.
The parameters b1 and b2 are estimates of the impact of the treatment with a 6- and 12-month
recertification period, respectively. The average of b1 and b2 provides an estimate of the average effect of
the two versions of the treatment, while the difference between them is an estimate of the added effect of
extending certification to 12 months. Results will be presented in tables as in Exhibit A16.1 (which
would show the average effect of the full treatment group compared to the control group) and Table
A.16.1a (which would show the effect of the treatment with the two different eligibility periods and the
effect of the extended eligibility period as measured by the difference in outcomes between the 12-month
and 6-month groups). In addition, key results will be presented graphically, in bar and line charts as in
Exhibits A16.2 and A 16.3.

Exhibit A16.1
Sample Table Shell: Regression-Adjusted Estimates of Total Impact of Intervention on Employment
and Earnings
Treatment
Group

Control Group

Estimated Impact
(Difference)

Employment (%)
Any covered employment,
Q1-Q8a
Any employment, months
1-24b
(etc.)

Abt Associates Inc.

OMB Clearance Request

Earnings ($)
Total UI earnings, Q1-Q8a
Total reported earnings,
months 1-24b
(etc.)
Sample size
UI records
Follow-up survey

Abt Associates Inc.

OMB Clearance Request

Exhibit A16.1a
Sample Table Shell: Regression-Adjusted Estimates of Impacts of Experiment 1 on Employment and
Earnings
Treatment Group:
6-Month Recertification
Control
Group
Mean

RegressionAdjusted
Mean

Estimated
Impact

Treatment Group:
12-Month Recertification
RegressionAdjusted
Mean

Estimated
Impact

Estimated
Impact of
Longer
Eligibility

Employment (%)
Any covered
employment, Q1Q8a
Any employment, months 124b
(etc.)
Earnings ($)
Total UI earnings,
Q1-Q8a
Total reported
earnings, months
1-24b
(etc.)
Sample size
UI records
Follow-up survey
***Statistically significant difference from control group, p < 0.01
** Statistically significant difference from control group, p < 0.05
* Statistically significant difference from control group, p < 0.10
Sources: a: UI records
b: Follow-up survey

Abt Associates Inc.

OMB Clearance Request

Exhibit A16.2
Sample Exhibit: Covered Employment, Q1-Q8, by Treatment Group

Exhibit A16.3
Sample Exhibit: Covered Employment, by Quarter and Treatment Group

Subgroup Analyses
The population participating in this experiment will be low-income families in Cook County. There may
be measurable differences in impacts among subgroups. We believe the most important of these will be
between new applicants and those at recertification, but there might also be differences by age of
youngest child, or by race and ethnicity, which are two important determinants of parental decisions with
regard to both child care modal choice and maternal employment. We note that the two standard
approaches to subgroup analysis—including interaction terms in the regression model, and estimating
separate models for the groups—test different hypotheses. Interaction terms find effects for members of a
subgroup holding other factors constant—e.g., the effect of ethnicity on outcomes, conditional on receipt
of cash assistance. Estimating separate models finds effects for subgroups given their different
characteristics—e.g., the effects of race and ethnicity on outcomes, given that some groups are more

Abt Associates Inc.

OMB Clearance Request

likely to receive cash assistance than others. We expect that the latter approach would be of more interest
to policymakers.
Causal Pathways
If significant differences in outcomes are found, it will also be of interest to determine the paths through
which these impacts occurred. Although the pathways cannot be traced with as much confidence as the
assessment of impacts, joint examination of proximal and distal outcomes may yield suggestive results.
Suppose, for example, it is hypothesized that the experiment improved a distal outcome (reliability of
child care arrangements) by altering a proximal outcome (mode of care). Evidence in support of this
hypothesis would be that:


Children’s care was more reliable on average in the treatment group;



Children in the treatment group were more likely to be in more formal modes of care; and



Within the treatment and control groups, children in more formal modes of care had more
reliable care.

This sort of evidence would suggest that the treatment had changed the allocation of children among
modes of care, from those that were less reliable to those that were more reliable.
Implementation Study
To meet its objectives within the context of the overall evaluation of the demonstration, the
Implementation Study includes a variety of analytic approaches. In this section, we summarize our
approach to analysis for the Implementation Study of the Illinois demonstration.
Describing the Demonstration and Contextual Factors
These study goals are addressed mainly through two types of descriptive strategies: narrative description
and descriptive statistics. The narrative description consists in weaving together a coherent “story of
the demonstration project” from the information gathered in interviews, focus groups, and through
documentary materials. For this study, the “story” should include the following elements:


Initial implementation of demonstration policies—key actors in implementing demonstration;
reasons for participating in the demonstration; startup experiences and problems; solutions to
problems; time taken to reach stable operations.



Ongoing administration of demonstration policies for both treatment and control group
parents—demonstration operations, including methods of informing demonstration parents
and providers; subsidy eligibility determination and recertification processes; random
assignment; differences in administration of treatment and control group parents and
providers; methods of maintaining research group status.

A first step in building the “story of the program” is synthesizing the varied accounts of the program
collected through a select number of interviews with key staff, and through the research staff’s own
documentation of the process. Because the demonstration is fairly basic (i.e., providing subsidies to
families over the current eligibility income) and its administration largely relies on the existing
administrative processes, which are well-documented, we believe that the Implementation Study will be
straightforward.

Abt Associates Inc.

OMB Clearance Request

The analysis and presentation of a description of the contextual factors in Cook County will use
descriptive statistics, which are simply the measurement of some set of variables across a population or
groups within a population. We will be using descriptive statistics to document demonstration participant
characteristics and participation patterns, as well as many contextual factors.
Assessing the Faithfulness of the Test
The study will assess the faithfulness of the test in two ways. First, we will describe relevant
demonstration operations, including informing families of potential eligibility for the demonstration;
administering consent forms; random assignment; application of appropriate policies to treatment and
control group families; and maintaining group assignments over the life of the demonstration. Second,
we will analyze IDHS administrative data to verify differential treatment of treatment and control group
members.
Cost-Benefit Analysis
The cost-benefit analysis will combine results from the impact and implementation analyses with
expenditure information, to determine whether the costs of the experimental programs are justified given
their impacts. The foundation for the analysis will be the framework shown in Exhibit A16.4 that
indicates the components we plan to include in the analysis (the rows) and the perspectives from which
they will be assessed (the columns). The pluses and minuses following the name of each component
indicate whether that item is expected to increase or decrease; the pluses and minuses in the body of the
table indicate whether that change represents a benefit or a cost from each perspective. When the analysis
is performed, the table will be filled in with incremental measures, i.e., treatment-control group
differences. Separate analyses will be performed for each of the two variants of each experiment.
The families and children perspective will identify gains and losses to families and children in the
treatment group relative to the control group. These can occur through changes in:


Parental employment, earnings, and fringe benefits;



Receipt of means-tested benefits such as TANF, food stamps, public housing, and Medicaid;



Amount paid out of pocket for child care; and



Quality of child care, along such dimensions as flexibility, reliability, and stability.

The government and society perspectives identify gains and losses to federal, state, and local
government budgets associated with the treatment, and also non-monetary benefits due to taxpayers’
preferences that funds be spent on programs (such as child care subsidies) that support work, rather than
on cash welfare.
We discuss measurement of each group of costs and benefits below.

Abt Associates Inc.

OMB Clearance Request

Exhibit A16.4
Cost-Benefit Framework

Expected Impacts
Employment and Earnings
Hours worked (+) / foregone leisure & time
for family responsibilities ()
Earnings after taxes (+)
Fringe benefits (+)

Anticipated Effects on:
Government and the
Families and Children
Taxpayers


+
+

Means-tested income
TANF benefits ()
Public housing subsidies ()
Other programmatic income ()





+
+
+

Taxes/Transfers
Income and sales taxes (+)
EITC payments (+)


+

+


Child care costs
Out-of-pocket child care costs ()
Subsidy payments (+)
Program administrative costs (+)

+



Quality of child care
Flexibility (+)
Reliability (+)
Stability (+)

+
+
+

Net cost/benefit

Employment and Earnings
Adults in the treatment group families are expected to work more hours. This represents a loss to them in
terms of forgone leisure that could have been spent on other activities. Taxpayers are assumed to
consider the increased employment of participating families to be a benefit. The increased earnings and
fringe benefits associated with the increased employment are benefits to families.
Although earnings will be measured in dollars, the other components in this category will be in natural
units: hours per week, likelihood of paid vacation and holidays, etc.
Means-Tested Income
Families are expected to reduce their reliance on TANF and other means-tested programs. These changes
represent a loss to them and a gain to taxpayer budgets. TANF effects can be measured in dollars.
Taxes and Transfers
The higher income received by families will lead to greater tax payments by them—a loss to them, a gain
to other taxpayers. This will probably be countered by an increase in EITC payments (although the rules
Abt Associates Inc.

OMB Clearance Request

are such that if a family achieves more than some level of earnings, its EITC payments will start to
decline). These costs and benefits are all measured in dollars.
Child Care Costs
Families’ out-of-pocket expenditures will probably, but not certainly, decline. It is expected that the
subsidy will substitute for part of what families pay. It is possible, nonetheless, that the availability of a
subsidy will induce some families to move from “free” care provided by household members or other
relatives to care that requires some co-payment, so that their out-of-pocket costs will increase.
Furthermore, if loss or absence of a subsidy keeps a control group mother from working, she will incur no
out-of-pocket expenditures while her treatment group counterpart is making copayments.
The subsidy payments themselves, of course, represent a cost to taxpayers that will be greater for the
treatment than the control group. In addition, taxpayers must bear the incremental cost of administering
the expanded subsidy program. The costs and benefits in this category are all measured in dollars.
Quality of Child Care
Finally, it is to be hoped that the families in the treatment group obtain better quality child care. Three
dimensions that will be measured on the follow-up survey are flexibility, reliability, and stability. These
will be measured in natural units (e.g., percent of respondents that rate their child care arrangements as
very/somewhat flexible, very/somewhat reliable; percent of focus children that were in the same primary
care arrangement for an entire school year).
Net Cost/Benefit
The monetary costs and benefits will be summed for both families and taxpayers, and the non-monetary
effects borne in mind. It is anticipated that the net effect of the intervention for families is positive; they
are being offered benefits for which they would otherwise not be eligible. Presumably they can therefore
be no worse off.
Taxpayers will almost certainly be worse off, in the narrow sense that expanded child care subsidies, with
their associated administrative costs, will not pay for themselves. There may be small offsets to these
costs in terms of increased tax receipts and lower TANF payments, but even these may be reduced by
higher EITC payments. Nonetheless, taxpayers may consider themselves better off in that it is preferable
to spend tax dollars on work supports than on TANF.
Study Schedule
The planned time schedule for the study is as follows:
Recruitment and random assignment begins
First Implementation Study visit
Random assignment ends
Expected OMB approval
Survey at 8 months
Survey at 16 months
Second Implementation Study visit
Survey at 24 months
7

March 2005
March 2006
March 2006
May 2006
May 2006 –February 20077
July 2006 – June 2007
August 2006
March 2007 – February 2008

The interviews will not commence until OMB clearance has been obtained, which is expected in March.
Therefore, families who were recruited into the study in the period of March through May 2005 will have an
initial interview that falls between 10-12 months. This group includes 244 study participants

Abt Associates Inc.

OMB Clearance Request

Last subsidy payment is made under
enhanced income eligibility ceiling

February 2008

Final report

August 2008

c. Washington
Analysis Plan
There are two major sets of statistical analyses that will be completed as part of the evaluation: impact
analyses and cost-benefit analyses. The analysis plan for each is provided below.
Impact Analysis
The purpose of the impact analysis is to estimate the effects of the alternative co-payment structure on a
number of outcomes. The random assignment design enables us to identify the net effect of subsidies on
these outcomes by letting us draw comparisons between the treatment and control group.
The types of outcomes we will assess fall into three general categories:




Child care outcomes (e.g., reliability, flexibility, and stability of child care arrangements,
type of care, hours in care, numbers of child care problems, cost of the child care, average
amount parents spend on child care)



Public assistance outcomes (e.g., percent with any use of TANF assistance, average months
of use of TANF, percent with any use of food stamps, average months use of food stamps)
Child care subsidy outcomes (e.g., average length of time on subsidy program; average
amount of state subsidy payments per child and per family)



Random assignment ensures the internal validity of the analysis based on this comparison: we can
assume that the two groups are equivalent, and that the outcomes observed for the control group in each
experiment represent on average the outcomes that would have been experienced by members of the
treatment group, absent the experiment.
Although simple comparisons of means and proportions would yield consistent impact estimates, more
precision can be obtained by use of multivariate methods. Within each group, outcomes will vary
because of other underlying factors, such as mother’s employment opportunities, children’s ages, and so
on. It is easier to detect differences in central tendencies between two groups if each group is more
homogeneous, i.e., if their outcomes are more tightly clustered. Regression-adjusted outcomes are indeed
more tightly clustered to the extent that the covariates have explanatory power. For example, a partial R 2
of 0.25 due to the covariates implies that the variance of the regression-adjusted mean is only threequarters as large as the variance of the raw mean, and this reduction correspondingly increases the
statistical power of the comparison.
Regression adjustment will therefore be used for each outcome, with a technique appropriate to the form
of the outcome: linear regression for continuous outcomes such as quarterly earnings, logistic regression
for dichotomous outcomes such as an indicator that the family received any TANF income. The goal of
Abt Associates Inc.

OMB Clearance Request

these models is not to “explain” the outcomes; these will be reduced-form models and the coefficients,
other than those on the treatment indicators, will in general be uninterpretable. Baseline data will include
information on the ages and relationships of family members, work and training activities of other parents
or stepparents in the household, ethnicity and citizenship of children for whom subsidies are sought,
family participation in TANF and other federal cash income programs, and child support received.
The form of the regression equation for a continuous outcome will therefore be:
yi = b0 + (b1 × Ti ) + Σj (b3j × Xij) + εi,
where yi = value of outcome for family i,
Ti = indicator that family i is in the treatment group,
Xij = value of baseline variable j for family i, and
εi = residual.
The parameter b1 is the effect of the treatment. Results will be presented in tables as in Exhibit A16.5.
Exhibit A16.5
Treatment Group
Control Group
Mean

RegressionAdjusted Mean

Estimated Impact

Employment (%)
Any covered employment,
Q1-Q8a
Any employment, months
1-24b
(etc.)
Earnings ($)
Total UI earnings, Q1-Q8a
Total reported earnings,
months 1-24b
(etc.)
Sample size
UI records
Follow-up survey

Subgroup Analyses
We expect that the alternative co-payment scale may result in differential impacts according to subgroups,
including the income level of the family as well as whether the family is newly receiving subsidies or is a
continuous subsidy user. We note that the two standard approaches to subgroup analysis—including
interaction terms in the regression model, and estimating separate models for the groups—test different
hypotheses. Interaction terms find effects for members of a subgroup holding other factors constant—
e.g., the effect of ethnicity on outcomes, conditional on receipt of cash assistance. Estimating separate
models finds effects for subgroups given their different characteristics—e.g., the effects of ethnicity on
outcomes, given that some groups are more likely to receive cash assistance than others. We expect that
the latter approach would be of more interest to policymakers.
Abt Associates Inc.

OMB Clearance Request

Causal Pathways
If significant differences in outcomes are found, it will also be of interest to determine the paths through
which these impacts occurred. Although the pathways cannot be traced with as much confidence as the
assessment of impacts, joint examination of proximal and distal outcomes may yield suggestive results.
Suppose, for example, it is hypothesized that the experiment improved a distal outcome (stability of child
care arrangements) by altering a proximal outcome (mode of care). Evidence in support of this
hypothesis would be that:


Children’s child care arrangements were more stable on average in the treatment group;



Children in the treatment group were more likely to be in more formal modes of care; and



Within the treatment and control groups, children in more formal modes of care had more
reliable care.

This sort of evidence would suggest that the treatment had changed the allocation of children among
modes of care, from those that were less reliable to those that were more reliable.
Implementation Study
To meet its objectives within the context of the overall evaluation of the demonstration, the
Implementation Study includes a variety of analytic approaches. In this section, we summarize our
approach to analysis for the Implementation Study of the Washington demonstration.
Describing the Demonstration and Contextual Factors
These study goals are addressed mainly through two types of descriptive strategies: narrative description
and descriptive statistics. The narrative description consists in weaving together a coherent “story of
the demonstration project” from the information gathered in interviews, focus groups, and through
documentary materials. For this study, the “story” should include the following elements:




A first step in building the “story of the program” is synthesizing the varied accounts of the program
collected through a select number of interviews with key staff, and through the research staff’s own
documentation of the process. Because the demonstration is fairly basic (i.e., using an alternative
schedule to calculate co-payments) and its administration largely relies on the existing administrative
processes, which are well-documented, we believe that the Implementation Study will be straightforward.
The analysis and presentation of a description of the contextual factors in Washington will use
descriptive statistics, which are simply the measurement of some set of variables across a population or

Abt Associates Inc.

OMB Clearance Request

groups within a population. We will be using descriptive statistics to document demonstration participant
characteristics and participation patterns, as well as many contextual factors.
Assessing the Faithfulness of the Test
The study will assess the faithfulness of the test in two ways. First, we will describe relevant
demonstration operations, including informing families of potential eligibility for the demonstration;
administering consent forms; random assignment; application of appropriate policies to treatment and
control group families; and maintaining group assignments over the life of the demonstration. Second,
we will analyze state administrative data to verify differential treatment of treatment and control group
members.
Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with
expenditure information, to determine whether the costs of the experimental programs or policies are
justified, given their positive impacts.
The benefits and costs of each of the experiment will be assessed from several perspectives, including
families and children, the government, tax payers, and society.


From the perspective of families and children, we will identify the gains and losses to
families that result from co-payment amounts and use or loss of subsidies, in terms of
parental employment and child care stability. We also will identify changes in the use of
public assistance, families’ child care costs, and total family income. Because the study will
provide parents with lower (or the same) co-payment amount and because all parental
changes in employment, child care choices, and other behavior will be completely voluntary,
the intervention should be beneficial to families overall and should not harm any families.
However, the benefit to families might be partially offset by costs such as increased taxes and
reduced public assistance and leisure time.



From the government perspective, we will tally benefits and costs to federal, state, and local
governments. The benefits of child care subsidies could include increased tax revenue as a
result of increased employment of families that receive subsidies and reduced use of public
assistance. The primary costs to government will be the actual subsidy payments for families
who use subsidies longer than they would with a higher co-payment, and the higher state
payment made to compensate for the lower co-payment amount in the alternative co-payment
scale.



From the taxpayers’ perspective, we will tally the benefits and costs to the general public.
The taxpayer will benefit from increased tax revenue since it offers the possibility of greater
services or lower taxes.



The perspective of society as a whole combines the perspectives of families and other
taxpayers. A net gain to society occurs when a gain to one taxpayer is not a loss to other
taxpayers.

The foundation for the analysis will be the framework shown in Exhibit A16.6 that indicates the
components we plan to include in the analysis (the rows) and the perspectives from which they will be
assessed (the columns). The pluses and minuses following the name of each component indicate whether
that item is expected to increase or decrease; the pluses and minuses in the body of the table indicate
Abt Associates Inc.

OMB Clearance Request

whether that change represents a benefit or a cost from each perspective. When the analysis is performed,
the table will be filled in with incremental measures, i.e., treatment-control group differences. Separate
analyses will be performed for each of the two variants of each experiment.
The families and children perspective will identify gains and losses to families and children in the
treatment group relative to the control group. These can occur through changes in:


Parental employment, earnings, and fringe benefits;



Receipt of means-tested benefits such as TANF, food stamps, and public housing;



Amount paid out of pocket for child care; and



Quality of child care, along such dimensions as flexibility, reliability, and stability.

The taxpayer and government perspectives identify gains and losses to federal, state, and local
government budgets associated with the treatment, and also non-monetary benefits due to taxpayers’
preferences that funds be spent on programs (such as child care subsidies) that support work, rather than
on cash welfare.
We discuss measurement of each group of costs and benefits below.

Exhibit A16.6
Cost-Benefit Framework

Expected Impacts
Child care costs
Out-of-pocket child care costs ()
Subsidy payments (+)
Program administrative costs (+)

Anticipated Effects on:
Taxpayers and
Families and Children
Government
+



Child Care Quality
Flexibility (+)
Reliability (+)
Stability (+)

+
+
+

Employment and Earnings
Hours worked (+) / foregone leisure ()
Earnings after taxes (+)
Fringe benefits (+)


+
+

Means-tested income
TANF benefits ()
Public housing subsidies ()
Other programmatic income ()





+
+
+

Taxes/Transfers
Income and sales taxes (+)
EITC payments (+)


+

+


Abt Associates Inc.

OMB Clearance Request

Exhibit A16.6
Cost-Benefit Framework
Net cost/benefit

Child Care Costs. Families’ out-of-pocket expenditures will decline or stay the same at any given
income level. However, if treatment group families work more than their control group counterparts, they
might incur greater out-of-pocket costs for child care if the increased work hours result in them moving
from part-time to full-time care. However, the alternative co-payment schedule has been designed so that
the after-tax income of families in the treatment group will increase if they choose to work more.
If subsidy payments are increased because co-payments are reduced, this represents a cost to taxpayers. If
the alternative co-payment schedule encourages parents to work more hours and earn more, this cost
might be reduced or reversed. The costs and benefits in this category are all measured in dollars.
Quality of Child Care. Reducing co-payments might encourage families in the treatment group to use
better quality child care. Three dimensions that will be measured on the follow-up survey are flexibility,
reliability, and stability. These will be measured in natural units (e.g., percent of respondents that rate
their child care arrangements as very/somewhat flexible, very/somewhat reliable; percent of focus
children that were in the same primary care arrangement for an entire school year).
Employment and Earnings. Reduced co-payments might encourage adults in treatment group families
to work more hours. This represents a gain to them in earnings and fringe benefits although the gain may
be offset somewhat by a loss of forgone leisure. Taxpayers are assumed to consider the increased
employment of participating families to be a benefit.
Although earnings will be measured in dollars, the other components in this category will be in natural
units: hours per week, likelihood of paid vacation and holidays, etc.
Means-Tested Income. If the intervention encourages parents to work more, it will also reduce their
reliance on TANF and other forms of public assistance. These changes represent a loss to them and a
gain to taxpayer budgets. (It is important to note that the alternative co-payment schedule was set so that
family’s income will increase as their hours of work increase, even if their public assistance benefits are
reduced). Public assistance amounts will be measured in dollars.
Taxes and Transfers. If the intervention encourages parents to work more, they will also pay more in
payroll taxes and might pay more in income taxes, although they might receive greater refunds from the
federal Earned Income Tax Credit. An increase in tax payments represents a loss to families and a gain to
other taxpayers, while an increase in EITC payments represents a gain to participating families and a loss
to other taxpayers (assuming that taxpayers do not value the redistribution of income). (It is important to
note that the alternative co-payment schedule was set so that families’ income will increase as their hours
of work increase, even if they pay more in taxes). These costs and benefits are all measured in dollars.
Net Cost/Benefit. The monetary costs and benefits will be summed for both families and taxpayers, and
the non-monetary effects borne in mind. The net effect of the intervention for families is expected to be

Abt Associates Inc.

OMB Clearance Request

positive since all parental decisions are completely voluntary. Presumably they can therefore be no worse
off.
Taxpayers might be worse off or better off. Expanded child care subsidies, with their associated
administrative costs, will represent a cost to taxpayers. These might be offset to some extent by increased
tax receipts and lower public assistance benefits. Regardless, taxpayers might consider themselves better
off in that it is preferable to spend tax dollars on work supports than on TANF.
Study Schedule
The planned time schedule for the study is as follows:
Recruitment and random assignment period
Expected OMB approval
Survey at 8 months
Survey at 16 months
Survey at 24 months
Last co-payment is made under reduced co-payment schedule
Final Report

October 2005
May 2006
June 2006
March 2007
September 2007
September 2007
March 2008

A17 Display of Expiration for OMB Approval
A space for the OMB approval number and expiration date is indicated at the top of the cover page for
each instrument submitted.

A18 Exception to the Certification Statement Identified in Item 19.0 of Form
OMB 83-I
None.

Abt Associates Inc.

OMB Clearance Request

Part B
Collection of Information Using Statistical Methods
B1

Sample Universe, Sampling Method and Expected Response Rates

a. Massachusetts
Sample Universe
The target population includes family child care providers who are licensed, part of a network, and stable
(i.e., in business for at least two years). The study is being conducted with a sample of family child care
providers from family child care networks in the state who have indicated interest in and the capacity to
implement Learningames. A statewide sample of such family child care homes is desired in order to
obtain results that are applicable to the state as a whole.8 Choosing a sample from only part of the state
would yield results that are representative of those parts of the state, but substantial differences in
economic and personal circumstances of family child care providers and families in different parts of the
state would mean the results would be of less use to the State. However, in order to increase efficiency
and reduce costs we will try to cluster the sample of providers within a relatively small number of regions
of the state. The study will include children in family child care homes enrolled in the home who, at the
start of the study, are 36 months of age or younger.
Sampling Method
Within each region included in the sample, we will recruit family child care networks that can contribute
at least 10 homes to the study (e.g., they have 10 homes that will volunteer to participate, and each has
been in business for two years and cares for two children under 36 months of age). The number of
networks participating in the study will vary by region. Randomization will occur within each family
child care network so that all participating agencies are guaranteed to have half of their homes be in the
Learningames group.
Within these providers’ homes, we will study the development of preschool children. Since the study is
longitudinal, following the same providers over two years, we will include in the study children in the
homes who are 36 months old or younger who either (a) are in the homes at the outset of the evaluation or
(b) who enroll in the homes during the first 18 months of the study. This “rolling” sampling strategy will
help increase our chances of having an adequate sample of children for the impact analyses on child
outcomes. We will close study enrollment to new children six months before the end of the evaluation
period so that all children evaluated at the final assessment point will have been in the home at least six
months.
The children will be in the family child care homes for differing amounts of time. Some children will
enter the home during the study period and others will leave. At the end of the two years of the
evaluation, the analyses of child impacts will first analyze the average-age standardized score on the
8

Not all family child care providers in the state are associated with networks. This limits the generalizability of
the findings to providers who are linked to networks and receive the support and monitoring provided by
network staff. However, this subset of providers, who receive child care subsidies, is of particular policy
interest to the state. We will use additional extant data from the networks, and from Abt’s Cost-Quality Study
of Family Child Care in Massachusetts, to investigate the differences between the study sample of providers and
the wider universe of providers in Massachusetts.

Abt Associates Inc.

OMB Clearance Request

measure of language development across all children clustered within the home. Second, we will
examine the impacts for different age groups of children, assuming the final sample includes sufficient
numbers of children in the relevant age categories. We propose to divide the sample into four age groups,
based on age of child at the completion of the study or at the last testing point before the child leaves the
home:





under 12 months,
12-23 months,
24-35 months, and
36 – 60 months.

Sample size is determined by our desire to measure child outcomes as well as provider behavior. We will
net approximately 350 providers, 175 treatment and 175 control. We assume that each provider will have
at least two children in the sample. This sample size allows us to detect effects on children and on
providers of 0.23 standard deviations.9
b. Illinois
Sample Universe
The sample universe for the experiment includes all families with income between 50 and 65% of SMI
who apply for child care subsidies in Cook County, Illinois either as new applicants or at recertification.
In addition, families must meet a number of other conditions to be eligible to participate in the study.
They must comply with all other requirements for receiving the subsidy, such as using a provider who has
been approved by the state to receive subsidies. In addition, they must (1) provide a valid Social Security
number (SSN) so that we can retrieve administrative records on other forms of public assistance and
earnings reported to the unemployment insurance (UI) system; (2) provide informed consent; (3) live in
Cook County; and (4) not work for the subsidy administering agency (Action for Children, AFC).
Finally, the applicant must not be a member of a group that is currently certified for subsidies for less than
six months: the self-employed, those paid in cash, teachers or students, and employees of temporary
agencies.
Sampling Method
Exhibit B1.1 presents the expected samples sizes for the experiments, by subgroup. A total of 2,000
families will be randomly assigned—1,000 to the treatment group and 1,000 to the control group. Within
the treatment group, half of the sample will be a 6-month authorization period (at which point they need
to reapply or be “redetermined”). The other half will be assigned a 12-month authorization period. The
size of the sample will allow us to be able to detect an effect of 0.11 standard deviations for impacts
measured using administrative records—for example, a difference in employment from 50 percent in the
treatment group to 55 percent of the combined treatment group—with power of 0.80. It will also allow us
to detect effects of .125 standard deviations for outcomes measured using the follow-up surveys assuming
80 percent of participants respond to the survey. This assumes no regression adjustment in the impact
calculations, and also does not correct for making multiple comparisons.

We are purposefully looking for whether or not Learningames produces positive changes in provider and child
outcomes. Therefore, we will conduct a one-tailed test using α1=0.05. We have also assumed that the analyses
will include baseline measures that explain 25% of the variation in study outcomes.

Abt Associates Inc.

OMB Clearance Request

Exhibit B1.1
Number of Families to Be Included in Each Experiment, by Treatment Status

Control group
Treatment group
12-month certification period
6-month certification period

Families At Application
500

Families At Reapplication
500

250
250

Recent data indicate that about 200 applicant families per month are being recruited into the sample.
AFC, the local child care resource and referral agency that is administering the subsidies, is recruiting the
sample for the experiment.
The sample is being collected in Cook County, Illinois, which includes the Chicago and its suburban
areas. We will obtain de-identified subsidy records for the study period for Cook County as well as the
entire state. Using the records of families between 45-50% of SMI (i.e., families just below the state’s
eligibility ceiling for subsidies) we will compare the participants in the treatment group to those in the rest
of the state in terms of family composition, child care selection, and employment characteristics. This
will help us to ascertain roughly the similarities and differences between (1) the study group and similar
families whose incomes make them eligible in Cook County; and (2) the study group and such Illinois
families who do not reside in Cook County.

c. Washington
Sample Universe
The study identified all households with adult parents (i.e., those over 18 years of age) statewide who
were approved to receive child care subsidies for a three-week period in October and November 2005.
This period was sufficient to build up the desired sample of 2,000 families in the treatment group. (In
order to have equal numbers of families in the treatment groups at each of the three tiers of the current
eligibility scale, there are 3,142 families in the control group, since families at the highest income tier do
not apply for subsidies at the same rate as those at the lowest tier.) The study has excluded the following
groups: (1) those whose youngest child is age 11, as they will not be eligible for a subsidy after their child
reaches age 13, and therefore will not be eligible for the subsidy; and (2) households headed by nonneedy adults, as they will always be assigned the minimal co-payment. A statewide sample was desired in
order to obtain results that are applicable to the state as a whole. Choosing a sample from only part of the
state would have yielded results that are representative of those parts of the state, but substantial
differences in economic and personal circumstances of subsidy recipients in different parts of the state
would mean the results would be of less use to the State.
Sampling Method
Exhibit B1.2 describes the following recruitment process and information provided to study participants.
In summary:


5,142 families were assigned either to the alternative or standard co-payment.

Abt Associates Inc.

OMB Clearance Request



All families were sent a notice describing the study and providing them with an opportunity
to withhold contact information. The State will process and link administrative records for
these families and provide them to the research team without identifying information. All
identifiers indicated in 45 CFR 164.514 will be removed from the data for these families
before they are shared with the Abt research team.



The contact information (name and address) for a group of approximately 2,500 of the
remaining families will be sent to Moore and Associates, the survey firm. These families
(divided equally between the treatment and control groups) will be sent a letter by researchers
reminding them that they will be contacted. The families not contacted will not be asked for
consent for release of their administrative records. Administrative data about families not
contacted will also be stripped of identifiers and used for analysis.



If consent is given, families will be interviewed at three points in time. The administrative
records for those participants that have been contacted and agree to have them released will
be linked to the survey information by Abt and used for analysis by Abt and MDRC. The
State will not have access to survey information with identifying information. We estimate
that 1,000 families from the treatment group and 1,000 families from the control group will
be in this category.

It is important to be clear about what study participants can and cannot opt out of.


Study participants cannot opt out of being assigned to the alternative co-payment. Making copayments is one of the requirements of receiving a child care subsidy and Washington
Department of Child Care and Early Learning (DCCEL) determine their level. Study
participants can implicitly opt out of the alternative, lower co-payment schedule in one of two
ways. First, they could decide not to use the subsidy for which they were approved. Second,
they could voluntarily pay the higher amounts of the standard schedule. Although this
violates DCCEL regulations, in practice there would be no way to know whether this
happened.



Study participants cannot opt out of having DCCEL collect and analyze data on subsidy use
and receipt of TANF and food stamps. DCCEL collects and analyzes these data as part of its
normal operations. In addition, DCCEL can collect and analyze data from the unemployment
insurance system in accord with whatever agreements are reached between DCCEL and the
Department of Employment Security.



Study participants who have been contacted and withheld consent can opt out of having these
data shared with Abt Associates and MDRC. However, the research design includes analysis
by Abt and MDRC of administrative data records only for the 3,000 families who are not
contacted and asked for the release of records. These records will be stripped of personal
identifying information.



Study participants can opt out of having contact information sent to the study team. They can
opt out of the survey and can refuse to answer any questions on the survey. As described
above, if they are contacted and ask, they can also opt out of having their administrative
records shared with the evaluators.

Abt Associates Inc.

OMB Clearance Request

Exhibit B1.2

Proposed Sample Size and Power of Planned Statistical Tests
Using random assignment allows us to estimate the effects of lower co-payments by comparing average
outcomes for families in the treatment and control groups. With 2,000 families in the treatment group and
3,142 families in the control group, such comparisons will give us an 80 percent chance of detecting
impacts of .07 standard deviation using one-tailed t-tests at the 0.05 significance level for outcomes based
on administrative records (for which we will have information on all families in the study, stripped of
identifiers in those cases where we did not receive approval for the release of records). We might be able
to boost the power of these tests by adjusting for the families’ baseline characteristics. An effect of .08
standard deviations is a 3 percentage point change in employment or subsidy receipt if we expect the
typical control group family to work or receive subsidies three-quarters of the time during the two years
following random assignment.
For measuring the effects of the intervention for families in different income tiers, the sampling strategy
would allow us to detect effects of .11 standard deviations for families in income Tier 1, .12 for families
in Tier 2, and .14 for families in Tier 3. For example, this would allow us to detect an increase in
employment or subsidy use from 75% of the control group to 80-81% of the treatment group for each
subgroup.
As described above, the study will also conduct follow-up surveys with 1,000 treatment group families
and 1,000 control group families. For outcomes based on these follow-up surveys, the minimum
detectable effects will be 41% larger than the minimum detectable effects described above for outcomes
based on administrative records. In no case is the minimum detectable sample size larger than .20.

Exhibit B.1.3 shows sample sizes and minimum detectable (mde)effects for each of the three income tiers
in Washington as well as the total. Recall that Tier 1 contains families below 82 percent of the federal
poverty level (in which all families have a copayment of $15), Tier 2 contains families with income
between 82 percent and 137.5 percent of the federal poverty level, and Tier 3 contains families with
higher income. The top half of the exhibit shows mdes for the full sample, which would be used in
estimating effects using administrative data, and the bottom half of the exhibit shows mdes for the survey
sample. For purposes of this exhibit, we assume the survey sample would be drawn equally from the three
tiers.
Exhibit B.1.3
Sample Size and Minimum Detectable Effects for Income Tiers in Washington Child Care Study

Tier 1

Tier 2

Tier 3

Total

Full sample
Abt Associates Inc.

OMB Clearance Request

Control group
Treatment group
Minimum detectable effect size
(standard deviations)
Minimum detectable effect on
employment (percentage points)
Survey sample
Control group
Treatment group
Minimum detectable effect size
(standard deviations)
Minimum detectable effect on
employment (percentage points)

1,292
524

1,073
710

777
766

3,142
2,000

0.13

0.12

0.13

0.07

6.4

6.0

6.3

3.6

333
333

334
334

1,000
1,000

0.19

0.11

9.6

5.6

Notes: Power calculations were based on one-tailed t-tests at the 5 percent significance level and assumes that baseline covariates
will not increase the precision of estimated effects. The calculations assumed a 50 percent employment rate for the control group.

Data Collection Strategy

a. Massachusetts
Four kinds of measures will be collected for the evaluation: systematic observations of provider
behavior; standardized assessments of children’s development; a provider questionnaire; and a home
visitor questionnaire.
Exhibit B2.1 shows the categories of data to be collected, data sources, time-period for collection and
analyses in which they will be used.
Exhibit B2.1
Overview of Data Needs and Data Sources
Data Needs

Sources of Data

Time Collected

Analyses for Which Data
Are Used

Child characteristics
Age, gender, home language,
length of time in care setting

 Provider records

 June 2006 and as children
enter the home

 Impact analysis

Provider characteristics Age,
ethnicity, education, training,
experience, job motivation

 Provider questionnaire

 June 2006
 January 2008

 Implementation study
 Impact analysis

Home environment
Health and safety, support for
cognitive, language, socialemotional development,
equipment materials

 QUEST Environment
checklist
 Caregiver rating

 Baseline (July-September
2005)

 Implementation study
 Impact analysis

 Fidelity observation
 Provider log

 June 2006
 January 2007
 January 2008

 Implementation study
 Impact analysis

QUEST caregiver rating

 June 2006
 January 2007
 January 2008

 Impact analysis

Provider behaviors and
interactions
 Level of implementation of
Learningames (treatment
group only)
 Behaviors and interactions

Abt Associates Inc.

OMB Clearance Request

Exhibit B2.1
Overview of Data Needs and Data Sources
Data Needs
Child outcomes
 Child development outcomes
 Child languages and pre
literary skills

Home visitor characteristics
Education, training, experience,
caseload size, frequency and
duration of home visits, job
responsibilities

Sources of Data

Time Collected

 Ages and stages
(extant data)

 Baseline (June-September
2005

 PLS4

 June 2006
 January 2007
 January 2008

 Home Visitor
Questionnaire

 June 2006

Analyses for Which Data
Are Used
 Impact analysis

 Implementation study
 Impact analysis

Observations of Providers
Baseline data will be collected by the staff of the family child care networks using the QUEST form and
trained by Abt staff. Study staff, hired by Abt Associates, will collect similar observation data using
QUEST and FDCRS six months after the intervention begins, and again at 12 and 24 months. At each
observation point, providers will be for approximately 2.5-3.0 hours. The observations will use a
standardized rating system. All observers trained to reliability by the Abt staff.
Child Assessments
Baseline information on the developmental status of children in the study will be drawn from extant data
collected by the participating family child care systems for children who are in the homes at the outset of
the study, and for children who enter the homes at a later date and up to six months before the study ends.
Similar data will be obtained for children who are under 36 months and enroll in the family child care
home after the study begins and up to six months before the end of the study. The evaluation team will
collect assessment data at three points over the two years, on the same schedule as for the observation
data. These assessments will use a standardized measure, the PLS-4 described in an earlier section. For
those children 3 years and older, the test will be administered individually to the children by study staff, at
the family child care homes.
Provider Questionnaire
A provider questionnaire will be administered by Abt study staff in June 2006 and January 2008. The
initial questionnaire will obtain information on the background and educational and training experience,
and motivation of the providers. The second questionnaire will focus specifically on additional education
and training obtained over the two years, beyond that offered by Learningames.
Home Visitor Questionnaire
A questionnaire for home visitors will be distributed by Abt study staff in June 2006. The questionnaire
will collect data on education and training, caseload size, job responsibilities, frequency and duration of
home visits.

b. Illinois
Abt Associates Inc.

OMB Clearance Request

Exhibit B2.2 presents a summary of our data collection strategy. Our main sources of data are extant
administrative data and documents, the parent interview as described in sections above, and unstructured
interviews with state officials and child care experts.
Exhibit B2.2
Overview of Data Needs and Data Sources
Data Needs
Family and household
characteristics (e.g., family
size, number of parents,
number and ages of children)

Employment and educational
characteristics (e.g., number of
employers, employment hours
and schedules, earnings, school
attendance)

Family income (e.g., total
household income, child
support received, household
income from employment)

Public assistance use and costs
(e.g., use of TANF cash
assistance, use of food stamps,
administrative costs of subsidy
receipt)

Sources of Data

Time Collected



Standard application
for child care
subsidies



Parent survey



Standard application
for child care
subsidies



Parent survey



Unemployment
Insurance records

 Quarterly, Months 024



Standard application
for child care
subsidies

 Baseline



Parent survey



Administrative
records for TANF
and food stamps
State and agency
budget documents



 Baseline

 Months 8, 16, & 24
 Baseline

 Months 8, 16, & 24

 Months 8, 16 & 24

 Ongoing



Interviews with
IDHS and DCACI
staff



4 months after
random
assignment

Child care characteristics (e.g.,
number of children receiving
child care, type of subsidized
arrangements, schedule of
arrangement, child care subsidy
costs, administrative costs,
family costs)



Standard application
for child care subsidies



Baseline



Parent survey


Months 8, 16 & 24



Administrative records
from child care subsidy
system



Ongoing

Planning and start up (e.g.,
demonstration design, rationale,
target groups, intended impacts,
planning Implementation, startup experiences, etc.)



Unstructured
interviews with
informants from IDHSI
and ACI



3 months prior to
random assignment
through 1 month
into random
assignment



Demonstration design

Abt Associates Inc.

Analyses for Which
Data Are Used
 Impact analysis
 Cost benefit
analysis
 Implementation
study (baseline
only)
 Impact analysis
 Cost benefit
analysis
 Implementation
study (baseline
only)

 Impact analysis
 Cost benefit
analysis
 Implementation
study (baseline
only)
 Impact analysis
 Cost benefit
analysis
 Implementation
study




Impact analysis
Cost benefit
analysis

Implementation
study (baseline
only)



Implementation
study

OMB Clearance Request

Exhibit B2.2
Overview of Data Needs and Data Sources
Data Needs

Sources of Data

Time Collected

Analyses for Which
Data Are Used

plans

Demonstration operations
(e.g., client flow through
random assignment, levels and
patterns of participation)



Memo of
Understanding



Meeting minutes



Unstructured
interviews with
informants from IDHSI
and DCACI
Administrative records
from subsidy intake
unit



Site-related contextual factors
(e.g., local child care market
conditions, local economic
conditions, expectations about
subsidy use among low-income
families)






Unstructured
interviews with
informants from
IDHSI, DCACI, local
child care and public
interest groups;
families using the
subsidy system
Local research reports
and public interest
documents
Bureau of Labor
Statistics area
employment and
earnings data



Throughout the
period of random
assignment



Implementation
study

Throughout the
period of random
assignment



Implementation
study

Data Collection for the Implementation Study
The Implementation Study will rely on information from the baseline, administrative, and survey data
(Exhibit B2.2). In addition, this part of the subsidy evaluation will rely on a range of open-ended
interviews and document reviews. These are described briefly below.
On-Site Data Collection
Open-ended interviews, as well as the collection of various documents, will take place on-site through
two field visits over the course of the demonstration. During the visits, researchers will conduct
individual and small-group interviews with State DCCEL and DSHS management and staff, and local
DSHS management and staff. Researchers will also use both visits to observe demonstration operations.

Exhibit B2.4
Data Collection Strategies for the Implementation Study
Data Source

Collection Strategy

Demonstration providers

Small group open-ended interviews

Abt Associates Inc.

OMB Clearance Request

Parents

Follow-up surveys (as part of the impact analysis)

State DSHS staff

Individual and small group open-ended interviews

Local DSHS staff

Individual and small group open-ended interviews

DSHS administrative data

Periodic files provided by DCCEL (as part of impact
analysis)

DSHS statistical reports

Periodic requests to DCCEL

Demonstration plans and design

Requests for DCCEL planning documents; MOAs
between Abt Associates and DCCEL

Subsidy system policy manuals and eligibility forms

Requests to DCCEL

Census information

U.S. Census

BLS labor market data

BLS publications (hard copy and online)

Individual and Small Group Open-Ended Interviews
Much of the descriptive information about intervention design, planning, and implementation, as well as
about the context in which the demonstration will operate, will come from individual and small-group
interviews with key informants during the first site visit to Washington. The Implementation Study will
include open-ended interviews with the following informant state and local subsidy and public assistance
agency staff.
Researchers will use interview guides that will be developed for each type of informant. The open-ended
interviews will be conducted individually or in small groups of up to three informants. An advantage of
small-group interviews (where possible) is that although one respondent may forget one or more details,
or may answer incorrectly, informants in small groups usually correct one another and can fill in details
others may leave out. Because we are primarily interested in "getting the story right," we will try to
organize small-group interviews, where possible.
The interview guides will be organized by topic area for each type of informant. Within each topic area,
the guides will include basic questions and probes designed to stimulate discussion and more complete
information for each topic area. The use of detailed interview guides ensures some level of uniformity
across researchers and informants. Also, the guides as annotated by interview notes provide a structure to
data collection that readily organizes field notes for analysis and reporting.
Another useful practice in conducting open-ended interviews is to ask respondents the reasons and/or
evidence for their judgmental answers. First, this may force informants to think more carefully about
their responses and qualify them in the light of their grounds for holding their opinions. Second, it allows
the researcher to weigh the informant's opinion against the strength of the evidence used to support it.
Subsidy Agency Statistical Reports
Extant subsidy agency statistical reports will be used to help characterize the child care subsidy market in
the demonstration sites. We expect such reports to provide basic information about: subsidy use,
including numbers of families, children, and providers; mean subsidy amounts; types of care used.
Subsidy System Policy Manuals and Eligibility Forms
We will collect demonstration site subsidy system manuals and eligibility forms as our primary source of
information about subsidy eligibility criteria, subsidy levels, and co-payment amounts and collection
Abt Associates Inc.

OMB Clearance Request

processes. The manuals and eligibility forms will also allow some insight into the initial eligibility and
recertification processes, although information about those operations will also be collected in the openended interviews at demonstration sites.
Census Information
Census Bureau information will be used as a primary data source for information about site demographic
and socio-economic characteristics. Using data from census tracts that most closely overlap with the
demonstration sites, the Implementation Study will summarize information about demonstration site
ethnicity, household number and composition, number of families with children, distribution of children
by age, and other relevant contextual factors.
c. Washington
Exhibit B2.3 aligns the categories of data with our data sources and provides the time period during which
they will be collected.
Exhibit B2.3
Overview of Data Needs and Data Sources
Data Needs

Sources of Data

Family and household
characteristics (e.g., family
size, number of parents,
number and ages of children)

 Standard application for
child care subsidies

 Baseline

 Parent survey

 Months 8, 16 & 24

Employment and educational
characteristics (e.g., number of
employers, employment hours
and schedules, earnings, school
attendance)

 Standard application for
child care subsidies

 Baseline

 Parent survey

 Months 8, 16 & 24

 Unemployment
Insurance records

 Quarterly, Months 0-24

Family income (e.g., total
household income, child
support received, household
income from employment)

 Standard application for
child care subsidies

 Baseline

 Parent survey

 Months 8, 16, & 24

Public assistance use and costs
(e.g., use of TANF cash
assistance, use of food stamps,
administrative costs of subsidy
receipt)

 Administrative records
for TANF and food
stamps
 State and agency budget
documents
 Interviews with state
staff

 Ongoing

Abt Associates Inc.

Time Collected

Analyses for Which Data
Are Used
 Impact analysis
 Cost benefit analysis
 Implementation study
(baseline only)
 Impact analysis
 Cost benefit analysis
 Implementation study
(baseline only)

 Impact analysis
 Cost benefit analysis
 Implementation study
(baseline only)
 Impact analysis
 Cost benefit analysis
 Implementation study
(baseline only)

OMB Clearance Request

Exhibit B2.3
Overview of Data Needs and Data Sources
Data Needs

Sources of Data

Time Collected

Child care characteristics (e.g.,
number of children receiving
child care, type of subsidized
arrangements, schedule of
arrangement, child care subsidy
costs, administrative costs,
family costs)

 Standard application for
child care subsidies

 Baseline

 Parent survey

 Months 8, 16, & 24

 Administrative records
from child care subsidy
system

 Ongoing

Analyses for Which Data
Are Used
 Impact analysis
 Cost benefit analysis
 Implementation study
(baseline only)

Data Collection Strategies
The major sources of data include the parent interview, administrative data and other extant information,
and interviews with staff at Washington Department of Social and Human Services (DSHS) and the
State’s regional offices. Each of these is discussed briefly below.
Parent Interviews
As stated earlier, parent interviews will be conducted by telephone at 8, 16, and 24 months after random
assignment. We do not plan to interview people in their homes. We will attempt to interview about 2,500
families and expect to interview 2,000 families (for an 80 percent response rate). Interviews will be
divided about equally between the treatment and control groups. The interview will be a vital source of
information for the impact and benefit-cost analyses and parts of it may also be used for the
Implementation Study. The interview will provide us with more detailed information about family
characteristics than is available in the baseline data, as well as changes that have occurred in some of
these characteristics (e.g., the birth of a child, an additional adult moving into the household) since
random assignment began. The survey will also be the study’s primary source of information about child
care and employment characteristics over the course of the two-year period.
We believe that we can capture changes in employment and child care with sufficient accuracy through a
telephone interview. If, at the completion of the first interview at 8 months, it becomes clear that a higher
percentage of in-person interviews will be necessary, we would be able to adjust our data collection plan.
Administrative Data
Records from various public assistance programs will be used for the impact and benefit-cost analyses. In
general, these records will be used to quantify participants’ use of various forms of public assistance. In
addition, information on employment and earnings from Unemployment Insurance records will augment
data from parent surveys.
For each automated system, data will be provided one year (understanding data system limitations) prior
to and two years following random assignment. If additional funding is obtained, we may seek additional
follow-up data. In that case, we would ask for consent for release of identifying information when
families are surveyed at the 24-month point. The automated systems include the following:




Child care subsidy amounts and provider information;
TANF authorized grant amounts and dates;
Food Stamps authorized amounts and dates; and

Abt Associates Inc.

OMB Clearance Request



Unemployment Insurance (UI), quarterly wages (earnings), and employer ID numbers.

Data Collection for the Implementation Study
The Implementation Study will rely on information from the baseline, administrative, and survey data
(Exhibit B2.4). In addition, this part of the subsidy evaluation will rely on a range of open-ended
interviews and document reviews. These are described briefly below.
On-Site Data Collection
Open-ended interviews, as well as the collection of various documents, will take place on-site through
two field visits over the course of the demonstration. During the visits, researchers will conduct
individual and small-group interviews with State DCCEL and DSHS management and staff, and local
DSHS management and staff. Researchers will also use both visits to observe demonstration operations.

Exhibit B2.4
Data Collection Strategies for the Implementation Study
Data Source

Collection Strategy

Demonstration providers

Small group open-ended interviews

Parents

Follow-up surveys (as part of the impact analysis)

State DSHS staff

Individual and small group open-ended interviews

Local DSHS staff

Individual and small group open-ended interviews

DSHS administrative data

Periodic files provided by DCCEL (as part of impact
analysis)

DSHS statistical reports

Periodic requests to DCCEL

Demonstration plans and design

Requests for DCCEL planning documents; MOAs
between Abt Associates and DCCEL

Subsidy system policy manuals and eligibility forms

Requests to DCCEL

Census information

U.S. Census

BLS labor market data

BLS publications (hard copy and online)

OMB Clearance Request

The interview guides will be organized by topic area for each type of informant. Within each topic area,
the guides will include basic questions and probes designed to stimulate discussion and more complete
information for each topic area. The use of detailed interview guides insures some level of uniformity
across researchers and informants. Also, the guides as annotated by interview notes provide a structure to
data collection that readily organizes field notes for analysis and reporting.
Another useful practice in conducting open-ended interviews is to ask respondents the reasons and/or
evidence for their judgmental answers. First, this may force informants to think more carefully about
their responses and qualify them in the light of their grounds for holding their opinions. Second, it allows
the researcher to weigh the informant's opinion against the strength of the evidence used to support it.
Subsidy Agency Statistical Reports
Extant subsidy agency statistical reports will be used to help characterize the child care subsidy market in
the demonstration sites. We expect such reports to provide basic information about: subsidy use,
including numbers of families, children, and providers; mean subsidy amounts; types of care used.
Subsidy System Policy Manuals and Eligibility Forms
We will collect demonstration site subsidy system manuals and eligibility forms as our primary source of
information about subsidy eligibility criteria, subsidy levels, and co-payment amounts and collection
processes. The manuals and eligibility forms will also allow some insight into the initial eligibility and
recertification processes, although information about those operations will also be collected in the openended interviews at demonstration sites.
Census Information
Census Bureau information will be used as a primary data source for information about site demographic
and socio-economic characteristics. Using data from census tracts that most closely overlap with the
demonstration sites, the Implementation Study will summarize information about demonstration site
ethnicity, household number and composition, number of families with children, distribution of children
by age, and other relevant contextual factors.
Bureau of Labor Statistics Labor Market Data
The BLS is an important source of data about local area labor markets, wage rates, industrial mix,
employment/population ratios, unemployment data, and other labor market factors. The BLS data are
organized by major metropolitan areas and the larger standard statistical metropolitan areas (SMSA). The
BLS data will be important in characterizing the low-income labor market facing many subsidy families.

Methods to Maximize Response Rates

a. Massachusetts
The data collection strategies planned for the study involve observations in the family child care home
and direct assessment of children. Early in the study, providers will be asked to complete a brief
questionnaire about their educational background, experience and motivation. Since the response burden
for providers is very little (7-10 minutes), and since home visitors will assist Abt staff in collecting any
missing questionnaires, we expect a response rate for the questionnaire of better than 90%. There is,
however, a burden imposed by the presence of observers and assessors; if not addressed with sensitivity,
this could, over time, affect provider willingness to allow data collection in their home.
Abt Associates Inc.

OMB Clearance Request

Using past experience as a guide, we propose several strategies to address this issue. First, in scheduling
visits to the home, we will emphasize that the visit will occur on a morning that is convenient for the
provider, and that their schedule and preferences will be decisive in scheduling a visit. The date and
length of the visit will be confirmed in a letter, which will also set out expectations for what will happen
during the visit. Data collection staff will telephone providers the day before the visit to confirm the
schedule since, in any child care setting, unscheduled events can throw off the provider’s schedule. If this
occurs, we will reschedule the visit at a time that is convenient for the provider.
Second, at the end of each visit, we will give each provider a $20 gift certificate to compensate her for the
disruption in her schedule occasioned by the data collection.
Finally, as part of our validation efforts, we will telephone a sample of providers visited by each data
collector to ensure that the visit went as planned, that the data collector explained what she was doing,
answered questions, and was respectful and unobtrusive. For all other providers, we will send a thank you
card with a toll-free number they can call if they have any concerns about the data collection.
In addition to these strategies, early in the study, each provider will receive a library of 12 children’s
books. We will maintain contact with providers through holiday cards and newsletters.
We expect that these efforts will be successful in maintaining providers’ cooperation. However, there are
many reasons why we might experience attrition from the study that have to do with providers’ own lives.
Providers may leave the study because they have decided not to continue providing care, because of a
family or personal emergency or for reasons beyond our control. If their reasons for leaving the study
have to do with the demands of the study, we will work with home visitors and system staff to negotiate a
solution. We have planned for approximately 15% attrition. If attrition increases beyond this rate, we
plan to refresh the sample by adding new providers. We would randomly assign these providers within
systems to either Learningames or the control group, following the same procedures as those initially
used.
We expect children to leave the child care home in the course of the study and will replace these children
with new entrants under three years of age. While we hope to obtain two assessments on each child, the
design does not call for a longitudinal study of specific children. We will continue to recruit age-eligible
children into the study until six months before the study ends. Our plan is to have essentially continuous
data collection and to have providers notify us if a child is leaving the home. This will allow us
maximum flexibility in assessing children and reduce non-response because of brief absence or
permanent attrition. At the same time, we expect to have no more than three measurements of each child,
for the purposes of calculating burden.
b. Illinois
Survey data will be collected at three points in time. All families in the treatment and control group (a
total of 2,000) families will be contacted to be interviewed. Our goal is to achieve an 80% response rate
at the first survey wave, conducted approximately 8 months after random assignment (1,600 respondents),
75% of the sample at Wave 2 at 18 months (1,500 respondents) and 70% at Wave 3 at 24 months (1,400
respondents). For each wave, we will attempt to reach the entire study sample, excluding those who ask
not to be contacted further. For example, for Wave 2, we will not exclusively attempt to contact the 1,600
respondents who participated in the Wave 1 interview but will use the total study group of 2,000, with the
exception of those who refused to be contacted further. While we are estimating a response rate for each
Abt Associates Inc.

OMB Clearance Request

wave of the study, we estimate that the overall response rate will be close to 80%; that is, 80% of the
sample will respond to at least one of the three survey waves.
We believe, based on our past experience, that an 80% response rate is an achievable goal. Exhibit B.3.1
provides information on response rates from similar data collection efforts.
Exhibit B.3.1 Response rates for similar data collection efforts.
Study
Moving to Opportunity
New York’s State Child
Assistance Program
Summer Career Exploration

Mode
Mail, telephone, in-field
surveys tracking household
members over six years
Longitudinal telephone,
with AFDC recipients

Sample Size/Sites
4,608 household heads in 5
cities; 3 waves

Telephone interviews: two
follow-up waves of
economically
disadvantaged high school
students

1,708 high school students
in three cities.

4,300 welfare recipients in
3 counties of New York

Response Rate and
Follow-up Period
92% (1-2 years)
83% (2-6 years)
81% (2-6 years)
90% (1 year)
87% (2 years)
85% (5 years)
93% (8 weeks)
89% (8 months)

In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact
information from study participants is accurate and of high quality. The contact information provided by
the study participants will include their own address and telephone numbers as well as similar contact
information of relatives and friends who are likely to know the participant's whereabouts and do not
cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information
that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over
the course of the study period. Contact information will be entered into a centralized sample database that
will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a
number of interim tracking methods to ensure that we continue to have up-to-date information. The
evaluation team will provide the study participants with a toll-free number to call should they move or get
a new phone number. To ensure that the number is on hand we will print it on both a refrigerator magnet
and a coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they
may send with any updated address or telephone information. Finally, sample members will be mailed
"tracking" letters at points prior to their interviews times. Families who return these postcards will receive
a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of
efforts to locate the proper address and telephone number. All respondents who complete an interview
will receive a $20 voucher or gift certificate.
Using data from UI wage records and other public records, we will be able to gather basic information
about the non-respondents. If necessary, we will be able to construct weights to address non-response.
We do not expect that there will be differential response rates between the treatment and control groups.
c. Washington
Survey data will be collected at three points in time. All 2,500 families who are selected to be in the
interview sample (drawn evenly from the treatment and control groups) will be contacted to be
interviewed. Our goal is to achieve an 80% response rate at the first survey wave, conducted
Abt Associates Inc.

OMB Clearance Request

approximately 8 months after random assignment (2,000 respondents), 75% of the sample at Wave 2 at 18
months (1,875 respondents) and 70% at Wave 3 at 24 months (1,750 respondents). For each wave, we
will attempt to reach the entire interview sample, excluding those who ask not to be contacted further.
For example, for Wave 2, we will not exclusively attempt to contact the 2,000 respondents who
participated in the Wave 1 interview but will use the total interview group of 2,500, with the exception of
those who refused to be contacted further. While we are estimating a response rate for each wave of the
study, we estimate that the overall response rate will be close to 80%; that is, 80% of the sample will
respond to at least one of the three survey waves. Again, based in prior experience, we believe that this
response rate is reasonable. (See Exhibit 2.3 for Abt’s relevant data collections.)
In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact
information from study participants is accurate and of high quality. The contact information provided by
the study participants will include their own address and telephone numbers as well as similar contact
information of relatives and friends who are likely to know the participant's whereabouts and do not
cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information
that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over
the course of the study period. Contact information will be entered into a centralized sample database that
will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a
number of interim tracking methods to ensure that we continue to have up-to-date information. The
evaluation team will provide the study participants with a toll-free number to call should they move or get
a new phone number. To ensure that the number is on hand we will print it on a refrigerator magnet and a
coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they may
send with any updated address or telephone information. Finally, sample members will be mailed
"tracking" letters at points prior to their interviews times. Families who return these postcards will receive
a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of
efforts to locate the proper address and telephone number. All respondents who complete the first
interview will receive a $10 voucher or gift certificate; $15 for completion of the second interview; and
$20 for completion of the third interview. The incentive will not affect participants’ potential benefits for
public benefits.
Using data from UI wage records and other public records, we will be able to gather basic information
about the non-respondents. If necessary, we will be able to construct weights to address non-response.
We do not expect that there will be differential response rates between the treatment and control groups.

Tests of Procedures

a. Massachusetts
The observation measures and provider questionnaire have all been tested and used in other large-scale
studies with similar populations and so do not require pretesting. The same is true for the standardized
child assessments. However, to ensure that our plan for collecting the data is realistic and does not
impose undue burden on the provider, we will pretest the data collection procedures in nine family child
care homes early in 2006. The results of the pretest will be sent to OMB, with a description of any
recommended changes in procedures.
For the children in the study, we want to obtain permission from the maximum number of parents to
allow their child to participate in the standardized assessments. We will work closely with the providers
Abt Associates Inc.

OMB Clearance Request

to have them help us contact and convince parents of the importance of the study and the low risk of
negative consequences for their child. We have a hotline that parents and providers will be able to use to
call with questions or concerns at any time during the study.

b. Illinois
We will pre-test the parent telephone interview survey with up to 30 respondents. The results of the
pretest will be sent to OMB, with a description of any recommended changes in wording or
administration of the survey. (See burden estimate in Exhibit A.12.2a).
c. Washington
The parent telephone interview survey used in the Illinois study will also be used in the Washington
study. The results of the pre-test conducted in the Illinois study apply to the Washington study as well.

Individuals Consulted on the Statistical Aspects of the Design

The information for all three studies is being collected by Abt Associates Inc. and its subcontractor,
Moore & Associates, on behalf of the Administration for Children and Families (ACF), U.S. Department
of Health and Human Services. With ACF oversight, Abt Associates is responsible for study design, data
collection, analysis, and report preparation.
a. Massachusetts
The project staff responsible for the design include the project director (Jean Layzer) the deputy project
director (Ann Collins), and the director of analysis (Barbara Goodson).
b. Illinois
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the
deputy project director, Ann Collins (Abt Associates), and the co-leads for analysis, Nancy Burstein (Abt
Associates) and Charles Michalopoulos (MDRC).
c. Washington
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the
deputy project director, Ann Collins (Abt Associates) and the director of analysis, Charles Michalopoulos
(MDRC).
For all three studies, outside consultants reviewed the statistical aspects of the design. These include:
Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Abt Associates Inc.

OMB Clearance Request

Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA

Abt Associates Inc.

OMB Clearance Request

References
Anderson, S., Ramsburg, D., & Rothbaum, B. (2003). Illinois study of license-exempt child care: Interim
report. Report to the Child Care Bureau, U.S. Department of Health and Human Services.
Urbana, IL: School of Social Work, University of Illinois at Urbana-Champaign.
Blau, D. M., “The Economics of Means-Tested Child Care Subsidies,” paper prepared for an
NBER volume on Means-Tested Transfers, February, 2000.
Burstein, N., Layzer, J., and Cahill, K. (2001). National Study of Child Care for Low-Income
Families. Patterns of Child Care Use Among Low-Income Families. Cambridge, MA: Abt
Associates, Inc.
Capizzano, J., Adams, G., and Sonenstein, F. (2000). Child Care Arrangements for Children Under
Five: Variations Across States. Washington DC: Urban Institute.
Casper, L. What Does It Cost to Mind Our Preschoolers? Census Bureau, Current Population
Reports, Household Economic Studies, P70-52.
Chaplin, D., Robins, P.K., Hofferth, S., Wissoker, D., and Fronstin P., “The Price Sensitivity of Child
Care Demand: A Sensitivity Analysis,” unpublished manuscript, January 1999.
Collins, A., Layzer, J., Kreader, J., Werner, A., and Glantz, F. (2000). National Study of Child Care for
Low Income Families: State and Community Substudy Interim Report. Cambridge, MA: Abt
Associates Inc.
Council of Economic Advisors. (December, 1997). The Economics of Child Care. White Paper.
Washington, DC: Author.
Hart, B. & Risley, T.R. (1995) Meaningful differences in the everyday experience of young American
children. Baltimore, MD: Paul H. Brookes Publishing Co.
Marshall, N. L., Creps, C.L., Burstein, N. R., Glantz, F. B., Robeson, W. W., and Barnett S. (2001). The
Cost and Quality of Full Day, Year-round Early Care and Education in Massachusetts:
Preschool Classrooms. Wellesley Center for Women and Abt Associates Inc.
Marshall, N. L., Creps, C.L., Burstein, N. R., Cahill, K., Robeson, W. W., Wang, S. Y., Schimmenti, J.,
and. Glantz, F. B (2003). Family Child Care Today. Wellesley Center for Women and Abt
Associates Inc.
Meyers, M., Peck, L., Davis, E., Collins, A., Kreader, J.L., George, A., Schexnayder, D., Schroeder, D.,
Olson, J. (2002). The Dynamics of Child Care Subsidy Use. A Collaborative Study of Five States.
New York, NY: The National Center for Children in Poverty, Mailman School of Public Health,
Columbia University.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for
Generalized Causal Inference. Boston: Houghton-Mifflin.

Abt Associates Inc.

OMB Clearance Request

Siegel, G.L., and Loman, L.A. (1991). Child Care and AFDC Recipients in Illinois: Patterns,
Problems, and Needs. St. Louis, MO: Institute of Applied Research.

Abt Associates Inc.

OMB Clearance Request

Appendix A: First Federal Register Notice
[Federal Register: September 21, 2005 (Volume 70, Number 182)]
[Notices]
[Page 55402-55403]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr21se05-89]
----------------------------------------------------------------------DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Proposed Information Collection Activity; Comment Request
Proposed Projects:
Title: Evaluation of Child Care Subsidy Strategies.
OMB No.: New Collection.
Description: To conduct four experiments to test aspects of the
child care subsidy system. Two simultaneous experiments will occur in
Cook County, Illinois; one will occur in Washington State; and one will
occur in Massachusetts.
Illinois. The State of Illinois has agreed to conduct two
simultaneous experiments, which will occur in Cook County. The first
will test the impact of receiving a child care subsidy on parental
employment and income, and on the stability of child care arrangements;
the second experiment will test the impact of losing a subsidy on the
same set of outcomes. For the first experiment, families with incomes
above the current income eligibility ceiling who apply for subsidies
will be approved to receive subsidies. In the second experiment,
families in the treatment group with incomes above the eligibility
ceiling who apply to be recertified to continue using subsidies will
remain eligible. In addition, each experiment will test the effects of
a longer certification period by certifying eligibility for some
families for six months and other families for one year. Families in
the two treatment groups will retain eligibility for subsidies over the
two-year study period, provided their income remains below the
experimental limit and they comply with other requirements (e.g.,
continue to work). Outcomes will be measured through administrative
records and periodic interviews with parents.
Washington. In Washington State, the study will test a co-payment
schedule that smoothes out what are currently abrupt increases in copayments that occur when a family moves from one income category to the
next and reduces the co-payment burden for many
[[Page 55403]]
families. Families that apply (or reapply) for subsidies and are
determined to be eligible under current rules will be randomly assigned
to the experimental co-payment schedule or the existing schedule.
(Families with co-payments from the experimental schedule will either
pay the same amount, or less, than families whose co-payments are
calculated using the existing schedule.) Families will retain the same
co-payment schedule for two years, provided they continue to be
eligible for subsidies. Outcomes will be measured through analysis of
administrative data and periodic interviews with parents.
Abt Associates Inc.

OMB Clearance Request

Massachusetts. In Massachusetts, the study is an experimental test
of the effectiveness of a developmental curriculum implemented in
family child care homes. Family child care providers who serve
subsidized and other low-income children and are linked to family child
care networks will be randomly assigned to a treatment or control
group. Providers in the treatment group will use the developmental
curriculum and be trained through regular visits to the home by
specially trained mentors. These providers will receive materials to
use with children from 0 to 5 years of age. Providers in the control
group will receive the more general technical assistance and support
visits that they currently receive. Impacts on provider behavior and
the home environment will be measured through direct observations in
the homes. Child assessments will be conducted through provider reports
for the younger children and through standardized tests for children 30
months and older.
Respondents: Illinois. Parents who apply (or reapply) for subsidies
and are eligible and agree to be in the study will be interviewed by
telephone up to three times in the 24 months after they enter the
study.
Washington State. Parents who apply (or reapply) for subsidies and
are eligible and agree to be in the study will be interviewed by
telephone up to three times over the 24 months of the study.
Approximately 30 state employees working at the Department of Health
and Human Services in the Division of Child Care and Early Learning or
the Division of Community Service will be interviewed as part of the
implementation study.
Massachusetts. Children will be assessed 7 months after
implementing the curriculum, after 11 months, and after 23 months.
Providers will be asked to respond to a brief survey 7 and 23 months
after the study begins.
Annual Burden Estimates
--------------------------------------------------------------------------------------------------------------Number of
Average
Instrument
Number of
responses
per
burden hours
Total burden
respondents
respondent
per response
hours
--------------------------------------------------------------------------------------------------------------Illinois parent survey.........................
5,000
1.5
.58
4,350
Washington parent survey.......................
2,000
1.5
.58
1,740
Washington process study interview.............
30
.5
.5
8
Massachusetts child assessments................
700
1.5
.5
525
Massachusetts provider questionnaire...........
350
1
.16
56
--------------------------------------------------------------------------------------------------------------Estimated Total Annual Burden Hours: 6,679.
In compliance with the requirements of Section 3506(c)(2)(A) of the
Paperwork Reduction Act of 1995, the Administration for Children and
Abt Associates Inc.

OMB Clearance Request

Families is soliciting public comment on the specific aspects of the
information colleciton described above. Copies of the proposed
collection of information can be obtained and comments may be forwarded
by writing to the Administration for Children and Families, Office of
Administration, Office of Information Services, 370 L'Enfant Promenade,
SW., Washington, DC 20447, Attn: ACF Reports Clearance Officer. E-mail
address: [email protected]. All requests should be identified by
the title of the information collection.
The Department specifically requests comments on: (a) Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility; (b) the accuracy of the
agency's estimate of the burden of the proposed collection of
information; (c) the quality, utility, and clarity of the information
to be collected; and (d) ways to minimize the burden of the collection
of information on respondents, including through the use of automated
collection techniques or other forms of information technology.
Consideration will be given to comments and suggestions submitted
within 60 days of this publication.
Dated: September 15, 2005.
Robert Sargis,
Reports Clearance, Officer.
[FR Doc. 05-18771 Filed 9-20-05; 8:45 am]
BILLING CODE 4184-01-M

Abt Associates Inc.

OMB Clearance Request

Appendix B: Review Procedures for Releasing Public and Restricted Use Data Files
Research Connections, a cooperative effort between the Child Care Bureau (CCB), the National Center
for Children in Poverty (NCCP) and the Inter-University Consortium for Political and Social Research
(ICPSR), provides access to and promotes the effective use of child care and early education research data
that can be used to make informed policy decisions. Data files available through Research Connections
comply with the standards of the organization responsible for their preservation and dissemination,
ICPSR.
Disclosure Review at ICPSR
ICPSR encourages and facilitates research in the social sciences and related areas by acquiring,
developing, archiving, and disseminating data and documentation relevant to a wide spectrum of
disciplines. The sole permitted and intended use of archival data is for statistical analysis of trends,
groups, or categories of cases, not for investigations of specific individuals or organizations.
Nevertheless, whenever data are made available in convenient and readily accessible form, the possibility
of intentional or inadvertent disclosure of confidential or erroneous information on individuals or
organizations is present.
In recognition of this possibility, ICPSR has developed policies designed to provide an appropriate
balance between individual privacy and the essential need of the research community for data. The issue
of confidentiality applies to a variety of data including survey data, personal interviews, observation
records, notes and recordings, organizational and institutional data, location or geographic coordinate
data, and public records. The term "confidential" refers to data that directly identify individuals or
organizations or which can be used indirectly or in combination to identify individuals or organizations.
ICPSR processing staff examine the contents of each data collection for the purposes of identifying
information that presents problems of preserving confidentiality. This is done in recognition of explicit or
implicit pledges of confidentiality given to respondents or research subjects and in accordance with
ICPSR policy, University of Michigan policy, and in the case of some of our topical archives, with federal
laws and regulations. This process, called a disclosure risk assessment, identifies variables and variable
content that must be modified to preserve confidentiality. All such changes in the data are noted in the
data documentation. This modified version of the data is made publicly available.
Common methods used to protect respondent confidentiality include dropping cases (eliminating outliers
from the sample or providing only a sub-sample of the cases in the total sample), dropping variables that
may directly identify respondents or pose a re-identification risk, and coarsening categories by collapsing
original values into groupings or capping the tails of a distribution through top and bottom-coding.
In some circumstances the use of these methods alone may not be adequate to fully protect the data in a
manner that allows for both its public dissemination and optimal analytic utility. Additional methods can
be used to decrease disclosure risk, usually applied in concert with the techniques enumerated above.
These complementary procedures can include record swapping, microaggregation, and blank and impute;
each systematically distorts original values while maintaining the statistical properties of the data. The
application of distortion disables the certainty of respondent identification and provides deniability when
claims of identification are made.
As modifying a dataset for reasons of confidentiality can sometimes compromise the research potential of
the data. ICPSR may elect to retain the confidential information but to restrict the use of the data set to
Abt Associates Inc.

OMB Clearance Request

individual researchers with bona fide research purposes for a specific time period based on agreements
established with the data provider. The decision to retain and disseminate a restricted-use version of the
data is made only with the consent of the data producer and in consultation with them.
Accessing Public-use Data Files
While current configurations permit both registered access to public-use datasets, as well anonymous
registration through a guest authentication procedure, Research Connections is moving toward requiring
data users to register with Research Connections as a requirement of public-use data file access.
Prior to downloading data files, all data users, anonymous or registered, are met with the following
warning:
WARNING! Some studies we distribute are collected and archived with the support of the U.S.
government and thus are covered by government regulations in addition to standard terms of use.
Several Federal laws and regulations forbid using these data for any purpose other than statistical
analysis and reporting. The applicable laws and regulations may be found in the United States
Code, 42 USC Section 3789g(a), the Code of Federal Regulations, 28 CFR 22, and Section
308(d) of the Public Health Service Act (42 U.S.C. 242m). Failure to abide by these laws may
result in criminal prosecution. Your use of these data signifies that you have read and agree to
abide by these laws and regulations. For more information, please contact
[email protected].
In addition, they must agree to abide by the Responsible Use Statement detailed below:
In preparing data for public release, ICPSR performs a number of procedures to ensure that the
identity of research subjects cannot be disclosed. For example, direct identifiers are omitted from
datasets, and some characteristics are recoded or masked if they can be combined with others to
identify individuals.

Any intentional identification or disclosure of a person or establishment violates the
assurances of confidentiality given to the providers of the information. Therefore, users
of data obtained from the ICPSR archive and/or any of its special topic archives agree:


To use these datasets solely for statistical analysis and reporting of aggregated
information, and not for investigation of specific individuals or organizations, except
when identification is authorized in writing by ICPSR



To make no use of the identity of any person or establishment discovered
inadvertently, and to advise ICPSR of any such discovery



To produce no links among ICPSR datasets or among ICPSR data and other datasets
that could identify individuals or organizations



To comply with the request that downloaded material not be redistributed or sold to
other individuals, institutions, or organizations without the written agreement of
ICPSR

ICPSR further asks that any books, articles, conference papers, theses, dissertations, reports, or
other publications that employ data or other resources provided by ICPSR reference the
bibliographic citation provided in the abstract and codebook for each ICPSR data collection.
Abt Associates Inc.

OMB Clearance Request

These citations acknowledge the principal investigators, the data producers, and ICPSR as the
data distributor, in accord with recommended citation procedures for computer files in the social
sciences. Also, authors of publications based on ICPSR data should send copies of their published
works or references to the publications to ICPSR for inclusion in a database of related
publications.
In addition, the user acknowledges that the original collector of the data, ICPSR, and the relevant
funding agency bear no responsibility for use of the data or for interpretations or inferences based
upon such uses.
By continuing past this point to the data retrieval process, you signify your agreement to comply
with the above-stated requirements and give your assurance that the use of statistical data
obtained from ICPSR and/or its Special Topic Archives will conform to widely-accepted
standards of practice and legal restrictions that are intended to protect the confidentiality of
research subjects.
Accessing Restricted-use Data Files
ICPSR employs several methods of disseminating restricted-use data files, depending on the level of
security necessary for the safeguarding of the data.
These methods range from making the data available to a defined subset of the research community, to
members of a specific research team, to researchers who have been closely reviewed following the
completion of a formal application process, including the submission of a data protection plan, to
researchers who may access the data only in a controlled and secure data enclave environment. Each of
these approaches requires an application process to gain access to the data. Some require the submission
of a data protection plan and include time limits after which the data must be destroyed or returned to
ICPSR.

Abt Associates Inc.

OMB Clearance Request