Hsis Part A

U.S. Department of Health and Human Services

Office of Planning, Research & Evaluation

Administration for Children and Families

Tracking of Participants in the Head Start Impact Study

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

October 9, 2008

A. JUSTIFICATION

A.1 Explanation of the Circumstances Which Make the Data Collection Necessary

The Office of the Administration for Children and Families, U.S. Department of Health and Human Services (DHHS), is submitting this Request for OMB Review in support of will collect follow-up information from children and families in the Head Start

Impact Study. In anticipation of conducting an 8th grade follow-up for the study, ACF will collect information necessary to identify respondents' current location and follow-up with respondents in the future.

OMB approved the initial package for the Head Start Impact Study (HSIS) in September 2002 (OMB # 0970-0229, Expiration Date: 09/30/2005 and the HSIS continuation OMB # 0970-0229, Expiration Date: 07/30/2006). This data collection was discontinued and then reinstated, under the same OMB number, for a Third Grade Follow-up of the children on 3/16/2007, with an expiration date of 3/30/2010.

The purpose of this follow-up is to continue tracking families in the studying anticipation of future data collections. The study has been highly successful in recruiting and maintaining participants over time, achieving well over 70% response rate in all waves of the study. This continued tracking is designed to maintain these high response rates during years in which no large-scale follow-up study will be conducted. To maintain adequate sample size, telephone interviews will be conducted in order to update the respondent's location and contact information. This information will be collected from parents or guardians in the spring of 2009, 2010, and 2011. ACF anticipates conducting an eight grade follow-up with these children beginning in the spring of 2012.

Background

Overview of Head Start. Over the years, Head Start has served nearly 23 million preschool children and their families since it began in 1965 as a six-week summer program for children of low-income families. The program provides comprehensive early child development services to low-income children, their families, and communities. Head Start has evolved over time to include a wide variety of program options based on the specific situations and resources of the communities to meet the changing needs of the children and families it serves. Variations in services include, but are not limited to, programs offering center-based services, home-based services, part-day enrollment, full-day enrollment and/or one or two years of services. In addition, many programs are now partnering with non-Head Start agencies and/or combining funds from various sources to coordinate services that best address the needs of children and families.

As Head Start’s Federal appropriation has grown, ($96 million in summer 1965 to $6.8 billion in 2005) so have initiatives calling for improved outcomes and accountability (e.g., Chief Financial Officers Act, Government Performance and Results Act of 1993). During the rapid expansion of Head Start, the U.S. General Accounting Office (GAO) released two reports underlining the lack of rigorous research on Head Start’s effectiveness noting that “…the body of research on current Head Start is insufficient to draw conclusions about the impact of the national program" (GAO, 1997). The 1998 report added, “…the Federal government’s significant financial investment in the Head Start program, including plans to increase the number of children served and enhance the quality of the program, warrants definitive research studies, even though they may be costly” (GAO, 1998).

Based upon GAO recommendation, and the testimony of research methodologists and early childhood experts, Congress mandated in Head Start’s 1998 reauthorization that DHHS conduct research to determine, on a national level, the impact of Head Start on the children it serves. Congress called for an expert panel to develop recommendations regarding the study design to “…determine if, overall, the Head Start programs have impacts consistent with their primary goal of increasing the social competence of children, by increasing the every day effectiveness of the children in dealing with their present environments and future responsibilities, and increasing their school readiness” (42 USC 9801, et.seq.). The research should also consider variables such as whether Head Start strengthens families as the nurturers of their children and increases children’s access to other education, health, nutritional, and community services.

To design such a study, the Department convened a committee of distinguished experts, the Advisory Committee on Head Start Research and Evaluation, that considered the major issues and challenges in designing a rigorous research study that is both credible and feasible, and the committee recommended a framework for the design of the Head Start Impact Study (HSIS). A contract was awarded in October 2000 to Westat, in collaboration with the Urban Institute, American Institutes for Research, and Decision Information Resources to conduct the Head Start Impact Study as mandated by the Coats Human Services Amendments of 1998, PL 105-285.

The National Head Start Impact Study is a longitudinal study that involved approximately 5,000 three- and four- year old preschool children across an estimated 75 nationally representative grantee/delegate agencies (in communities where there are more eligible children and families than can be served by the program). Children were randomly assigned to either a Head Start group that had access to Head Start program services or to a non-Head Start group that could enroll in available community non-Head Start services, selected by their parents. Data collection began in fall 2002 and continued through spring 2006, following children through the spring of their first-grade year. The HSIS data collection included parent interviews, teacher and care provider surveys, child assessments, direct observations of the quality in different care settings, and teacher/care provider ratings of children. A Third Grade Follow-Up Study is currently being conducted, based largely on work that has already been completed for the HSIS.

The Head Start Impact Study must be understood within the history of the preschool and Head Start research and evaluation efforts. Crucial to the study is understanding the evidence as it concerns preschool experiences influence on outcomes in elementary school.

Preschool Intervention Studies. Unfortunately, there is a paucity of experimental design studies examining preschool intervention and even fewer preschool intervention studies with longitudinal designs that stretch beyond kindergarten or first grade. Use of experimental design is concentrated on a few studies such as the Abecedarian Project, Project CARE, and the Early Training Project. These studies randomize families matched on control variables (e.g., income, gender) and place some into preschool intervention while excluding others. This allows researchers to determine the effects of treatment by comparing treated children and families to those that were similar at the start of the study and whose experiences differ only in terms of whether or not they received the intervention. When the experiments involve high intensity programs (i.e. extensive instruction, comprehensive services, home visits), generalization is difficult. High intensity projects are often considered too costly and resource intensive to be replicated on a national scale. It is often their small sample size that makes them feasible for researchers to conduct them. For example, the initial Abecedarian sample consisted of 117 participants (Campbell & Ramey, 1995). Moreover, findings from intense programs cannot always be expected to be replicated by more moderate programs. This is due to evidence that the intensity of a preschool intervention can increase the positive effect those programs exert on child outcomes (Nelson et al., 2003; Ramey & Ramey, 2004).

More common are studies that explore early experience predictors to school-age outcomes (e.g., Miles & Stipek, 2006; Peisner-Feinberg et al., 2001) and quasi-experimental designs intended to determine the causal contribution of specific experiences or programs to those outcomes. Quasi-experimental include wait-list designs that compare children who receive an intervention such as Head Start to those who are waiting for the opportunity to enroll, and regression-discontinuity approaches that rank children on level of need and adjust for these and other differences in comparing outcomes of participants and nonparticipants. The Chicago Longitudinal Study and the work of Abbott-Shim and colleagues at the Georgia State University Quality Research Center (2003) are two examples of such work. Although ethical concerns often call for them, quasi-experimental designs generally can not provide the clarity of data obtained through true experimental studies. Other studies that explore relationships between variables provide useful information to guide research hypotheses regarding the aspects of preschool interventions that are most likely to predict later child outcomes. For example, Peisner-Feinberg and colleagues (2001) reported a positive relationship between the quality of preschool care and elementary math scores. However, the question of long-term impacts of preschool interventions is best answered through longitudinal data from studies allowing the direct comparison of children who received the intervention to those who did not—and, ideally, comparing sets of children who are indistinguishable at the outset by virtue of having been selected at random from a common pool of eligible applicants.

Summarized below are some of the findings about the effects of preschool participation on children’s later outcomes.

Cognitive Outcomes. It is clear that preschool participation can have lasting cognitive and academic effects (Barnett, 1995; Miller & Bizzell, 1984; Nelson, Westhues, & MacLeod, 2003). For example, children who attend preschool are less likely to be held back at grade level or to be in special education classes (Darlington, Royce, Snipper, Murray, & Lazar, 1980). Further, the NICHD Early Child Care Research Study linked high-quality child care with higher school-age math and reading test scores (NICHD ECCRN, 2005).

Social Outcomes. Social outcomes such as socialization skills (e.g., Barnett, 1995; Hubbs-Tait et al., 2002) and juvenile delinquency (e.g., Garces, Thomas, & Curie, 2002; Reynolds, Ou, & Topitzes, 2004) are also positively influenced by preschool attendance. For example, children observed to have close relationships with their preschool teacher have been found to have higher attention and sociability ratings in the second grade as well as displaying fewer problem behaviors (Peisner-Feinberg et al., 2001). Further, in their meta-analysis of over 60 studies, Paro and Pianta (2000) concluded that measures of social outcomes taken soon after preschool explained a significant portion of the variance for assessments of social outcomes taken later in elementary school, albeit with small effect sizes.

Experimental Evidence. Experimental data on preschool provides further evidence of the long-term effects of preschool intervention. For example, the Abecedarian study reported that children in a preschool intervention group performed better on cognitive tests in third grade than those who had not had the intervention (Campbell & Ramey, 1995). Similar results were found for fourth graders who had summer interventions during the preschool period (Gray & Klaus, 1970); children who had received intervention outperformed control children on intelligence tests. It should be noted that the entire sample in the Campbell and Ramey (1995) study saw a decline in cognitive scores following the first grade, however, children who had intensive preschool interventions experienced less change over time.

Head Start Evidence Studies examining the effects of a national program such as Head Start have the potential to be more generalizable. However, taken as a whole, the literature yields inconsistent results as to the program’s success after kindergarten. For example, a study following Head Start children who took part in a Post-Head Start Transition program through the third grade found no achievement gains for the participants (Bickel & Spatig, 1999). However, it is reasonable to question whether the transition program elements were sufficient to maintain Head Start gains. Further, there is no control group with which to compare the progress of the children in the transition program to the progress of children not in the program or to those who had never attended Head Start. Finally, the limited sample in the Post-Head Start Transition program makes generalizations of any findings to a national sample of Head Start difficult.

Other data suggest that Head Start programs can have lasting effects. For example, one study found that female Head Start participants narrowed the gender gap in math (Kreisman, 2003). Findings also suggest that Head Start participation improves school readiness which can lead to enhanced school performance throughout elementary school (Abbott-Shim, Lambert, & McCarty, 2003, Lee, Brooks-Gunn, Schnur & Liaw, 1990). Data relating child care quality to positive child outcomes in third grade also point to potential benefits of Head Start participation (Burchinal, Roberts, Zeisel, Hennon, and Hooper, 2006; NICHD ECCRN, 2005). Considering that a national study of Head Start centers found that, on average, Head Start quality is on par with or better than alternate center-based child care options (Zill et al., 2003) it is reasonable to anticipate positive outcomes as a result of enrollment in the program.

Ongoing Longitudinal Efforts. Currently there are a number of longitudinal study efforts collecting valuable data regarding preschool and school-age experiences. Although not all of these studies focus exclusively on Head Start populations, all are collecting data that will allow examinations of the relationship between Head Start experiences and school-age outcomes. The studies are the Head Start Family and Child Experiences Survey (FACES), the Early Head Start Research and Evaluation Project (EHS), the Early Childhood Longitudinal Survey – Kindergarten Cohort (ECLS-K), and the Early Childhood Longitudinal Survey – Birth Cohort (ECLS-B). While these studies have provided and will continue to provide valuable data regarding the variety and nature of experiences of young children and their families as well as the relationships between those early experiences and later outcomes, the HSIS and the Third Grade Follow-Up Study will expand upon the knowledge gained from these combined research efforts.

The HSIS provides the opportunity to explore questions related to Head Start using a nationally representative sample. The relevance of findings reported by these studies will not be biased by programmatic anomalies or limited by regional sample characteristics but will be applicable to the whole of the Head Start population. Further, the experimental design of the study allows for the comparison of children and families whose only significant difference is the treatment in question (i.e. access to Head Start). The preliminary results from the first year report show that Head Start increases 3-year-old children’s cognitive and social emotional development and children’s health as well as positive parenting practices (all the domains examined in the study). Impacts were found on some measures in each of these four domains. Findings were also positive, though less prevalent, for 4-year-olds. The lasting effect of these impacts is currently being examined in kindergarten, first grade, and third grade. The continued tracking proposed in this package will help ensure that response rates for children in both the experimental and control groups are maintained at a high level. As a result, this tracking will provide a critical foundation for longer-term examination of the effects of Head Start into early adolescence.

Purpose of the Study

The primary purpose of this follow-up is to continue tracking families in the studying anticipation of future data collections. The study has been highly successful in recruiting and maintaining participants over time, achieving well over a 70 percent response rate in all waves of the study. This continued tracking is designed to maintain these high response rates during years in which no large-scale follow-up study will be conducted. To maintain adequate sample size, telephone interviews will be conducted in order to update the respondent's location and contact information. This information will be collected from parents or guardians in the spring of 2009, 2010, and 2011. ACF anticipates conducting an eight grade follow-up with these children beginning in the spring of 2012, and this tracking and supplementary data will be beneficial for the examination of long-term impacts of Head Start.

Research Questions

For the tracking study, there will be no research questions, nor any analyses to examine any research questions. The information being collected is solely for the purpose of ensuring that future data collection efforts will be successful in reaching study participants

The Head Start Impact Study itself, has several research questions. These questions were developed from a set of recommendations by the Advisory Committee on Head Start Research and Evaluation (1999) and are presented in a report of the Head Start Impact Study’s first year findings (U.S. Department of Health and Human Services, 2005).

A.2 Purpose and Use of the Information Collection

The original HSIS design called for collecting comparable data on two cohorts of newly entering children (a three-year old cohort and a four-year old cohort) and their families who were randomly assigned to either a treatment group (enrolled in Head Start) or a control group (that were not enrolled in Head Start, but were permitted to enroll in other available services in their community selected by their parents or be cared for at home). To draw the national sample, all eligible grantees/Delegate agencies were clustered geographically with a minimum number of eight grantees/delegate agencies within each cluster. The clusters were grouped into 25 strata based on state pre-K and childcare policy¹, race/ethnicity of the Head Start children served, urban/rural status, and region. Next, one cluster with probability proportional to the total enrollment of three- and four-year olds in the cluster, was selected from each stratum and approximately three grantee/delegate agencies were randomly selected from each cluster. From each of the 75 randomly selected grantees/delegate agencies, approximately 48 children per grantee/delegate agency were assigned to the Head Start treatment group and about 32 children were assigned to the control group. Sample children could not have been previously enrolled in Head Start. To avoid a sample size shortfall, small centers on the frame were grouped together within a program to perform center groups, each center group with a combined reported first year enrollment of at least 27 children. The selection and random assignment of approximately 5000 children occurred during the Spring/Summer of 2002.

The distribution of children by cohort or age group and by status (treatment or control group) are presented in Table 1.

Table 1. Number of Children in the Head Start and Non-Head Start Groups by Age Cohort

Age Cohort	Head Start (Treatment Group)	Non-Head Start (Control Group)	Total Sample
Three-year olds	1,530	1,029	2,559
Four-year olds	1,253	855	2,108
Total	2,783	1,884	4,667

This tracking follow-up will follow all the families from this sample. Data collection for the tracking study will occur in the spring of 2009, the spring of 2010, and the spring of 2011.Many challenges are presented by a longitudinal study with a national sample. Over time, families move and become more difficult to locate. We have been successful in gaining high cooperation from respondents for five rounds of HSIS data collection and four rounds of tracking. High response rates were achieved for the parent interviews and the child assessments as presented in Table 2.

Table 2. Percent of HSIS Parent Interviews and Child Assessments Complete by Data Collection Period

			T		T		T		T	T	T		T
	Fall 02	Spring 03	Fall 03	Spring 04	Fall 04	Spring 05	Fall 05	Spring 06 3-year old cohort	Spring 06 4-year-old cohort	Fall 06	Spring 07 3-year-old cohort	Spring 07 4-year-old cohort	Fall 07
Parent	86	83	84	81	83	81	83	80	82	81	81	77	81
Child	80	84		81		78		77				72

Note: Sample size is 4,667

T=Tracking only, no parent interview or assessment in that period

A tracking update form will be used to verify and update if necessary families’ contact information and information. The tracking updates will primarily be conducted over the telephone with in-person follow-up as necessary. Tracking updates will take about 15 minutes to complete.

A.3 Use of Improved Information Technology and Burden Reduction

The tracking updates will primarily be conducted over the telephone with in

person follow-up as necessary.

A.4 Efforts to Identify Duplication and Use of Similar Information

In the late 1990’s, the US General Accounting Office (GAO) released two reports concluding that (1) “…the Federal government’s significant financial investment in the Head Start program, including plans to increase the number of children served and enhance the quality of the program, warrants definitive research studies, even though they can be costly” ⁽U.S. General Accounting Office, 1998)and (2) this information need could not be met because “…the body of research on current Head Start is insufficient to draw conclusions about the impact of the national program”(U.S. General Accounting Office, 1997).

One purpose of the Head Start Impact Study was to measure the impact of Head Start on children’s early development and school readiness. This tracking study will ensure sufficient maintenance of the original Head Start Impact Study sample to allow longer-term follow-up through to early adolescence.

A.5 Impact on Small Businesses or Other Small Entities

No small businesses or other small entities will be involved in the data collection for this tracking study

A.6 Consequences of Collecting the Information Less Frequenly

As recommended by the Government Accounting Office and mandated by Congress, “definitive research studies” are legislatively required to assess the effectiveness of Head Start nationally on the school readiness of participating children. Despite increasing expenditures, including an appropriation of $6.88 billion in fiscal year 2008, “the body of research on current Head Start is insufficient to draw conclusions about the impact of the national program.” The Head Start Impact Study and its Third Grade Follow-up provided the data to allow such an evaluation through third grade. This tracking study will ensure that response rates are maintained to allow for a rigorous, meaningful evaluation through early adolescence.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This tracking study will be conducted in a manner entirely consistent with the guidelines in Title 5, Section 1320.6 of the Code of Federal Regulations. There are no special circumstances that might require deviation from these guidelines.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The public announcement for the Tracking of Participants in the Head Start Impact Study was published in the Federal Register on April 7^th, 2008 (Vol. 73, No. 67, p 18801). The text of the announcement is contained in Appendix A.

A number of methods were used to receive consultation into the original design of the Head Start Impact Study and its third grade follow-up. These efforts included many meetings with Advisory Committee members, as well as a meeting of consultants for the Third Grade Follow-up measurement design. The details of these efforts were presented in the supporting statement submitted to OMB for the Head Start Impact Study Third Grade Follow-up data collection. Given that the tracking data collection included in this package involves only tracking current participants, no consultants are recommended for this effort.

A.9 Explanation of Any Payment or Gift to Respondents

In order to provide an incentive to families to participate participating in the tracking study, thereby maximizing response rates, we will provide parents with a gift card or cash ($20) for each tracking update. This is same amount that was provided for tracking in the Third Grade Follow-up.

A.10 Assurances of Confidentiality Provided to Respondents

All Westat staff members sign the Westat pledge of privacy for the study. In addition, all field staff signed a privacy pledge.

For some parent respondents, the issue of privacy of information, particularly relating to address and telephone information collected for later tracing of respondents, is a matter of great concern. Participants will be assured that the information collected will be used for research purposes only by the research team, and that contact name and address information and other survey data will not be given to bill collectors, legal officials, other family members, etc.

We will implement procedural steps, similar to the steps used in the HSIS, to increase respondent confidence in our privacy procedures. We will generate a set of identification labels with a unique respondent ID number and bar code. These labels will be affixed to each of the data collection instruments for a respondent. The use of bar codes in conjunction with the numbered identification labels enables the receipt control staff to enter cases by reading the bar code with a wand, making receipt of completed interview packages also more efficient.

A.11 Justification for Sensitive Questions

We do not anticipate that any of the questions asked will be of a sensitive nature. The purpose of the interview and how the data will be used will be explained to all participants. Participants will be reassured in person and in writing that their participation in the study is completely voluntary. A decision not to participate will not affect their standing in any government program, and if they choose to participate, they may refuse to answer any question they find intrusive. All individuals’ responses will be held strictly confidential and none of their answers will be reported to any program, agency, or school but will be combined with the responses of others so that individuals cannot be identified. All interviews will take place in a setting where the respondent's privacy can be assured.

A.12 Estimates of Annualized Burden Hours and Costs

Tables 5a presents data on the annual burden for respondents to the tracking study for each data collection point. The burden estimates will be the same for all three years of data collection.

Table 5a. Estimated Annual Response Burden for Respondents in the Tracking of Head Start Impact Study Participants

INSTRUMENTS	NUMBER OF RESPONDENTS	NUMBER OF RESPONSES PER RESPONDENT	AVERAGE BURDEN HOURS PER RESPONSE	TOTAL BURDEN HOURS
Parent Tracking Interview	4,667	1	.25	1167.75
Totals for Spring 2009	4,667			1,167.75

Total Respondents for each Year: 4,667

Total Responses for each Year: 4,667

Total Burden Hours for each Year: 1,167.75

A.13 Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no direct monetary costs to participants other than their time to participate in the study.

A.14 Annualized Cost to the Federal Government

The total cost to the Federal Government for this tracking study is estimated to be $2.5 million. These costs include development of the tracking materials, data collection, and preparation of a data file. Respondent expenses and incentives are included in these costs. The average annualized cost over the three years of the contract are estimated to be approximately $833,000 per year.

A.15 Explanation for Program Changes or Adjustments

This is request to allow ACF to continue tracking participants in a previously approved data collection request. ACF will not be using the existing instruments as part of the tracking, thus, the reduction in burden.

A.16 Plans for Tabulations and Publication and Project Time Schedule

Since this tracking study is not designed to answer any research questions, there are not plans for statistical analysis of these data. A single indicator, the percentage of families successfully tracked, will be created to indicate the success of the overall tracking effort to advise the study for future data collection.

Publications

No publications will be created from this tracking study

Time Schedule.

Data collection for this tracking study will take place in spring 2009, spring 2010, and spring 2011 for all families.

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

We do not plan to ask for an exception to the OMB rule that the expiration date be printed on all survey documents.

A.18 Exceptions to Certification for Paperwork Reduction Act Submission

There are no requested exceptions to the certification statement in Item 19.

1 Data to define these strata were obtained from Children’s Defense Fund (1999), Seeds of Success report.

2