HSIS Appendices

U.S. Department of Health and Human Services

Office of Planning, Research & Evaluation

Administration for Children and Families

Tracking of Participants in the Head Start Impact Study

Appendices

February 11, 2009

Appendix A

Tracking Instrument

SC Name:	Grantee ID:	RA Center ID:
Grantee:
RA Center:
Child ID Number:
Child Name: Child DOB:
Child Language: RA Group:
Parent Language:

Updater Name:_________________________

(Check one) Telephone: ______ In Person:______

Date: _____/_______/_________

Start Time: ______________AM PM

End Time: _______________ AM PM

Building Futures: Head Start Impact Study

Parent/Primary Caregiver Update

Good [morning, afternoon or evening]. Is this (NAME OF ESPONDENT)? (IF NO, ASK FOR RESPONDENT; IF NOT AVAILABLE, ASK WHEN TO CALL BACK TO TALK WITH HIM/HER.) My name is ______________________, and I’m calling from the Building Futures: Head Start Impact Study that you and your child have participated in. The study has been so successful and the information has been so valuable that the government would like to continue the study with the same children to learn how participation in Head Start or other preschool programs affects children’s learning when the children are older. We’d like to ask you a few questions, much like the ones we asked you last spring, so we are calling you to do a short Parent Interview over the phone that should take about fifteen minutes to complete. We have a few questions about the school and before and after school settings that [CHILD] is in this year. We also will ask some questions to help make it possible to contact you in the future. We would like to thank you for completing this brief phone interview by sending you a check in the amount of 20 dollars. We would like to remind you that all information collected is confidential and will be kept private except as required by law. Your participation is voluntary. You may quit at any time. Your choice will not result in the loss of any current benefits you may have. We truly appreciate your help and your continued support of this important study. May we begin now? (IF AGREES, CONTINUE WITH THE INTERVIEW. IF NO, ASK: When would you like to schedule a date and time to complete this short interview?)

A. CONTACT INFORMATION UPDATE

A-1. Have you moved since March 1, 2008?

YES 1

NO 2

A-2. What is your current address and telephone number? Also, please tell me whether this is the correct spelling of your name.

(INTERVIEWER SPELL NAME AS LISTED ON CHILD PROFILE, VERIFY WITH RESPONDENT, AND RECORD BELOW WITH ADDRESS AND TELEPHONE NUMBER.)

Name:

Address:

City State Zip

Telephone: (__________)___________-________________________________

A-3. Is this the name and address where we should mail your 20 dollar check?

YES 1 (GO TO A-5)

NO 2

(NOTE: IF RESPONDENT STATES THAT HE/SHE CANNOT CASH A CHECK, SAY THAT WE WILL SEND A MONEY ORDER AND CHECK BOX BELOW.)

SEND MONEY ORDER □

A-4. What is the name and address where we should mail the check?

Name:

Address:

City State Zip

A-5. Are you planning to move between now and March 2010?

YES 1

NO 2 (GO TO A-8)

A-6. Do you know what your new address will be or the general area where you are planning to move?

YES 1

NO 2 (GO TO A-8)

A-7. What is the area where you are planning to move and, if you know, what will be your new address and telephone number?

(RECORD AS MUCH INFORMATION AS THE RESPONDENT KNOWS.)

Address:

City State Zip

Telephone: (__________)___________-________________________________

A-8. Just in case we have trouble reaching you, who can we contact who will be able to tell us where to reach you next time we call?

Name:

Address:

City State Zip

Telephone: (__________)___________-_______________________________

DIRECTIONS FOR SECTION B – CURRENT SCHOOL AND/OR CHILD CARE ARRANGEMENTS:

COHORT/AGE

• THE CHILD’S COHORT AND DATE OF BIRTH ARE PRE-PRINTED ON THE INTERVIEW LABEL.

• COHORT A CHILDREN ARE LIKELY TO BE IN FOURTH GRADE (WITH A FEW IN THIRD OR FIFTH)

• COHORT B CHILDREN ARE LIKELY TO BE IN FIFTH GRADE (WITH A FEW IN FOURTH OR SIXTH)

• DATE OF BIRTH IS ANOTHER CLUE. CHECK THIS DATE PRIOR TO OBTAINING THE SETTING INFORMATION TO GET A SENSE OF WHICH SETTINGS THE CHILD WILL MOST LIKELY BE ENROLLED IN BASED ON AGE.

• IF THE CHILD WAS BORN BEFORE 9/98, THE CHILD IS PROBABLY IN FIFTH GRADE.

• IF THE CHILD WAS BORN BETWEEN 10/98 AND 12/98, THE CHILD MAY BE IN FIFTH GRADE.

• OTHERWISE, THE CHILD PROBABLY IS IN FOURTH GRADE.

• THERE WILL BE SOME EXCEPTIONS. FOR EXAMPLE, SOME CHILDREN MAY BE IN AN EARLIER GRADE AND SOME MAY BE ADVANCED.

• NOTE: WE WILL COLLECT INFORMATION ABOUT OTHER CHILD CARE ARRANGEMENTS IN ADDITION TO OR IN LIEU OF SCHOOL.

SETTINGS

• HSIS GUIDELINES FOR NON-HEAD START CLASSIFICATIONS CONTINUE TO APPLY.

• A SCHOOL- OR CENTER-BASED SETTING MEANS THAT THE PROGRAM OPERATES FROM A SPACE THAT IS NOT A PRIVATE HOME (E.G. A FAITH-BASED BUILDING; A COMMUNITY CENTER).

• A DAY CARE HOME MEANS THAT THE PROGRAM OPERATES FROM SOMEONE’S HOME. THIS CAN REFER TO BOTH FORMAL AND INFORMAL TYPES OF CARE OFFERED BY A HOME SETTING.

• OWN HOME REFERS ONLY TO THE STUDY CHILD’S RESIDENCE.

B. CURRENT SCHOOL AND/OR CHILD CARE ARRANGEMENTS

Now I have a few questions about where your child is currently in school or other child care.

B-1. Is your child currently enrolled in Sixth Grade, Fifth Grade, Fourth Grade, or Third Grade?

YES, SIXTH GRADE 1

YES, FIFTH GRADE 2

YES, FOURTH GRADE 3

YES, THIRD GRADE 4

NO, UNGRADED 5

B-2. Which of the following best describes the school setting that [CHILD] is in?

Public School 01

Private School 02

Home School 03

Other (Specify) 04

_____________________________________________________

B-3. What is the name, address, and telephone number of this school?

Name:

Address:

City State Zip

Telephone: (__________)___________-________________________________

B-4. What is the name of [CHILD]’s teacher there?

Name:

B-5. What is the name of the principal there?

Name:

B-6. What month did [CHILD] start [GRADE FROM QUESTION B-1] at [SCHOOL NAME FROM QUESTION B-3]?

|____|____|

Month

B-7. In addition, does [CHILD] regularly spend time in any other enrichment program, or other before or after school arrangement, including care by relatives or neighbors, Monday through Friday, 8:00 a.m. to 6:00 p.m. for 5 or more hours per week? Do not include time with you or another parent.

YES 1

NO 2 (PROBE RE: ANY ENRICHMENT OR REGULAR ARRANGEMENT. IF NONE, GO TO SECTION C)

B-8. How many different arrangements does [CHILD] attend?

_____________________

Number of Arrangements

B-9. Please name each arrangement, tell us the month and year your child started to attend, and choose

the setting description that best applies to each. (CIRCLE ONE)

(ASSIST THE RESPONDENT WITH PROBES TO DETERMINE THE SETTING TYPE IN TERMS OF HSIS’S DEFINITIONS.)

a. 1. Arrangement Name: ____________________________________

2. Start date: |___|___| |___|___|

MONTH YEAR

3. Arrangement Type: (CIRCLE ONE)

School- or Center-Based Program 01

Someone else’s home (day care home) with relative 02

Someone else’s home (day care home) with non-relative 03

Own home with relative 04

Own Home with non-relative 05

Other (Specify) 06

_____________________________________________________

b. 1. Arrangement Name: ____________________________________

2. Start date: |___|___| |___|___|

MONTH YEAR

3. Arrangement Type: (CIRCLE ONE)

School- or Center-Based Program 01

Someone else’s home (day care home) with relative 02

Someone else’s home (day care home) with non-relative 03

Own home with relative 04

Own Home with non-relative 05

Other (Specify) 06

_____________________________________________________

c. 1. Arrangement Name: ____________________________________

2. Start date: |___|___| |___|___|

MONTH YEAR

3. Arrangement Type: (CIRCLE ONE)

School- or Center-Based Program 01

Someone else’s home (day care home) with relative 02

Someone else’s home (day care home) with non-relative 03

Own home with relative 04

Own Home with non-relative 05

Other (Specify) 06

_____________________________________________________

C. UPCOMING CHANGES IN SCHOOL OR MAIN CHILD CARE ARRANGEMENT:

C-1. Between now and September, are you planning to change [CHILD’S] school?

YES 1

IF YES, approximately when?

MONTH

NO 2 (GO TO END

SCRIPT)

C-2. Do you know the name, address or telephone number of that school or where it will be located?

YES 1

NO 2 (GO TO END

SCRIPT)

C-3. What is the area where the school will be located and, if you know it, what is the name, address and telephone number of that school and the name of your child’s teacher or the person responsible for your child’s care in this setting? What is the name of the Principal in that setting (if applicable)? (RECORD AS MUCH INFORMATION AS THE RESPONDENT KNOWS.)

Name:

Address:

City State Zip

Telephone: (__________)___________-________________________________

Teacher/Provider Name:

Principal Name:

END SCRIPT:

That’s all the questions I have. Thank you for your cooperation. You will receive your check for $20 as soon as possible, but it may not be for 6-8 weeks.

[END OF INTERVIEW].

Appendix B

60 Day Federal Register Notice

[Federal Register: April 7, 2008 (Volume 73, Number 67)]

[Notices]

[Page 18801]

From the Federal Register Online via GPO Access [wais.access.gpo.gov]

[DOCID:fr07ap08-69]

-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families

Proposed Information Collection Activity; Comment Request

Proposed Project:

Title: Tracking of Participants in the Head Start Impact Study.

OMB No.: 0970-0229.

Description: The Administration for Children and Families (ACF)

within the Department of Health and Human Services (HHS) will collect

follow-up information from children and families in the Head Start

Impact Study. In anticipation of conducting an 8th grade follow-up for

the study, ACF will collect information necessary to identify

respondents' current location and follow-up with respondents in the

future.

The Head Start Impact Study is a longitudinal study involving

approximately 5,000 first time enrolled three- and four-year-old

preschool children across 84 nationally representative grantee/delegate

agencies. Participants have been randomly assigned to either a Head

Start group or a control group. Data collection for the study began in

fall of 2002 and has been extended through late spring 2008 to include

the participants' 3rd grade year.

ACF will continue to examine outcomes for the sample through the

spring of the participant's 8th grade year. To maintain adequate sample

size, telephone interviews will be conducted in order to update the

respondent's location and contact information. This information will be

collected from parents or guardians in the spring of 2009, 2010, and

2011. A small set of additional items will provide information on the

parents' perception of the children's well-being.

Respondents: Treatment and control group members in the Head Start

Impact Study.

Annual Burden Estimates

-----------------------------------------------------------------------

Instrument Number of Number of Average burden Total

respondents responses per hours per burden

respondent response hours

-----------------------------------------------------------------------------

Tracking 4,667 1 .25 1,167

Estimated Total Burden Hours 1,167

In compliance with the requirements of section 3506(c)(2)(A) of the

Paperwork Reduction Act of 1995, the Administration for Children and

Families is soliciting public comment on the specific aspects of the

information collection described above. Copies of the proposed

collection of information can be obtained and comments may be forwarded

by writing to the Administration for Children and Families, Office of

Planning, Research and Evaluation, 370 L'Enfant Promenade, SW.,

Washington, DC 20447. Attn: ACF Reports Clearance Officer. E-mail

address: [email protected]. All requests should be

identified by the title of the information collection.

The Department specifically requests comments on: (a) Whether the

proposed collection of information is necessary for the proper

performance of the functions of the agency, including whether the

information shall have practical utility; (b) the accuracy of the

agency's estimate of the burden of the proposed collection of

information; (c) the quality, utility, and clarity of the information

to be collected; and (d) ways to minimize the burden of the collection

of information on respondents. Consideration will be given to comments

and suggestions submitted within 60 days of this publication.

Dated: March 31, 2008.

Brendan C. Kelly,

OPRE Reports Clearance Officer.

[FR Doc. E8-7138 Filed 4-4-08; 8:45 am]

BILLING CODE 4184-01-M

Appendix C

Head Start Impact Study Sampling Plan

B.2 Description of Sampling and Information Collection Procedures

Sampling Procedures

Two principles guided the development of this sampling strategy:

• National Representation—the sample needed to support our ability to extrapolate estimated program impacts to the population of all Head Start programs and children.

• Creation of a Randomized Comparison Group—sampled Head Start programs had to be able to provide a sufficient number of additional applicants to allow the use of an experimental design, i.e., the selected programs had to have more applicants than could be served.

The process used to select and recruit the study sample, under these two overarching principles, is described below and summarized in Exhibit 6.

Legislative Mandate and Advisory Committee Guidance. The legislative mandate required that the Head Start Impact Study provide:

“…a national analysis of the impact of Head Start” based on the selection of Head Start grantees/delegate agencies that “…operate in the 50 states, the Commonwealth of Puerto Rico, or the District of Columbia and that do not specifically target special populations.”

Furthermore, the Advisory Committee recommended that the sample of Head Start grantees/delegate agencies should reflect variation in a variety of characteristics including,

“…region of the country, race/ethnicity/language status, urban/rural, and depth of poverty in communities,” and “…design of program as a one-year or two-year experience for children; program options (e.g., center-based, home-based, part-day, full-day); auspice (e.g., Community Action Agency, public school, non-profit organization); community-level resources; alternative childcare options for low-income children; and, the nature of the childcare market and the labor market in the community studied.”

The sampling plan, therefore, began with the inclusion of all operating Head Start grantees/delegate agencies listed in the 1999 and 2000 Head Start Program Information Report (PIR), excluding two types of programs that target “special populations”—those serving migrant children, and those operated by Tribal Organizations—and those programs that are “extremely new to the program” because they may not represent stable Head Start operations. Children enrolled in Early Head Start (i.e., those younger than three years of age) were also excluded from the study sample because they are assured entry into regular Head Start (and cannot, therefore, be randomly assigned).

Geographically Cluster Grantee/Delegate Agencies. Using the resulting list, grantees/delegate agencies were clustered by county on the basis of their primary business address. Every county in the US with at least one operating grantee was included in the sampling frame with a minimum of at least eight (8) grantees/delegate agencies included per cluster. Very

Exhibit 6: Overall Plan for Sample Selection

Create Sampling Frame From All FY 1999-2000 Head Start Grantees and Delegate Agencies (DAs) in all 50

States, District of Columbia, and Puerto Rico (N= 1,715 Grantees/DAs).

Excluded “new”, Migrant, Tribal and Early Head Start programs and programs involved in FACES 2000.

Create County-Based Geographic Grantee Clusters (GGCs)

And Stratify (N= 161 clusters).

Clusters were stratified into 25 groups by HHS region, urban location, State Comprehensive Preschool Program, and race/ethnicity

Randomly Select One Cluster In Each Of 25 Strata.

Contacted all 261 grantees/DAs in selected clusters by telephone to determine if they are “saturated” —do they have sufficient extra applicants for a comparison group? A total of 223 grantees/DAs (85.4%) were determined to be eligible for further consideration.

Combine Eligible Small Programs and Stratify.

Some grantees/DAs are too small to supply the needed sample of children so they were combined into 184 grantee/DA groups. Groups were then stratified by urban location, auspice (e.g., school-based, community agency), race/ethnicity, program option (e.g., full- vs. part-day), and percent of three-year olds.

Select and Recruit Sampled Grantees/Delegate Agencies.

Three grantee/DA groups were sampled per cluster—a total of 76 groups representing 90 separate grantees/DAs. Site visits were conducted to 87 of the 90 selected grantees/DAs. One grantee/DA has closed and two are part of the QRC study.

All Sampled Grantees/Delegate Agencies Complete a Center Information Form (CIF)

for Each Active Head Start Center.

Data from Center Information Form used to determine saturation at the center level, a total of 1,411 centers. Twelve percent of the centers are eliminated due to saturation, leaving 1243 available for further sampling.

Combine Small Centers to Form Center Groups.

Small centers are combined to form center groups with grantee/DAs(N=683), and then sorted by race/ethnicity, program option, school-based, and percent of three-year olds.

Randomly Select a Sample of Centers/Center Groups.

An average of 3 centers/center groups is selected from each grantee/DA, for a total N=220 center groups, comprised of 411 individual centers. [A “reserve” sample was also selected for possible later use.]

Conduct Second Site Visit to Determine Center-Level Saturation.

Saturated centers will be eliminated.

Conduct Random Assignment of Newly-Entering Applicants in Non-Saturated Centers.

Sample targets: 3,406 children in the treatment group and 2,272 children in the comparison group.

Non-Participating Grantees (N=5)

Treatment Group

3-year olds – 1,882 children

4-year olds – 1,524 children

Comparison Group

3-year olds – 1,255 children

4-year olds – 1,017 children

Obtain Data on Non-Participating Sites

Select 3 centers per grantee/DA (N=15),

and an average of 6 Head Start children/

center. Also use FACES data to examine

under-capacity and “saturation” sites.

small grantees/delegate agencies, with fewer than 90 total 3- and 4-year old Head Start children, were combined with another grantee/delegate agency in the same county if possible or with one in an adjacent county. A total of 160 clusters were formed.

Stratify the Sample to Ensure National Program Representation. Next, the clusters were combined into 25 strata, each stratum having approximately the same number of three- and four-year old Head Start children. The following stratification variables were used:

• Region. Defined as five regions¹: (1) Northeast, Head Start Regions 1, 2, and 3; (2) South, Head Start Regions 4 and 6; (3) North Central, Head Start Region 5; (4) Plains, Head Start Regions 7 and 8; and (5) West, Head Start Regions 9 and 10.

• Urban Location. Three categories were defined²: (1) a county containing a central city of a Metropolitan Statistical Area (MSA) with 1 million or more persons; (2) a county in an MSA not included in the first category (i.e., a suburban county or any county in a small MSA); and, (3) all other areas of the country (i.e., areas not in an MSA, predominantly small towns and rural).

• State Comprehensive Programs for Low-Income Preschool Children. Three groups: (1) States with comprehensive state-funded pre-kindergarten programs that are similar to Head Start; (2) States with state-funded pre-kindergarten programs that have some comprehensive program components; and, (3) States meeting neither of the previous two requirements.

• Race/Ethnicity. Three categories: (1) High concentration of Hispanic Head Start children, the percentage of Hispanic children served by the grantees/delegate agencies in the cluster is at, or above, 40 percent; (2) High concentration of African American children (but not of Hispanic children), the percentage of non-Hispanic African American children served by the grantees/delegate agencies in the cluster is at, or above, 40 percent and the percentage of Hispanic children below 40 percent; and, (3) Other, all other clusters not included in the preceding categories.

These variables were used to create 25 strata each of which included about the same number of Head Start children.

Select Sample of Geographic Grantee Clusters. Once the strata were formed one cluster from each stratum was selected with probabilities proportional to size, i.e., clusters with larger numbers of Head Start children had a higher probability of being selected into the sample. This plan ensured that each Head Start grantee/delegate agency and participating child has a known probability of selection into the study sample, and—at later points in the sampling process—that the probabilities will be approximately the same for each child. The decision to sample a total of 25 clusters was based on a tradeoff between (1) the need to control the cost and quality of data collection and our ability to monitor random assignment (argues for fewer clusters), and (2) concerns about the magnitude of the confidence intervals around the estimated program impacts (argues for more clusters).

The 25 clusters included 355 grantee/delegate agencies. At this point, eight grantees/delegate agencies that were involved in the FACES 2000 study were dropped.

Identify Grantees/Delegate Agencies Eligible For The Study. In each of the 25 clusters, an attempt was made to contact all grantees/delegate agencies by telephone, except in three very large clusters—Los Angeles, Chicago, and Brooklyn—where twelve programs (or groups of programs) were randomly subsampled in each cluster prior to screening to reduce the amount of telephone calls that would have to be made. Eighty six additional grantees/delegate agencies were eliminated. In all, an attempt was made to screen a total of 261 programs.

Information gathered during the telephone calls was used to determine if the grantee/delegate agency was still actively serving three- and/or four-year old children, and “not saturated,” i.e., if it had enough applications from newly-entering children in one or more centers (possibly only in one service option) to fill all federally-funded Head Start slots and have enough applicants left over for a comparison group.

Of the 261 programs, a total of 223 (85%) were either determined to be eligible, or their eligibility could not be established because not enough information was provided (this included a total of 28 programs that remained in the pool for possible selection).

Collapsing, Stratifying, and Selecting Grantees/Delegates Within Sampled Clusters. As in an earlier step, small grantees/delegate agencies were combined to avoid a sample shortfall of children (and to avoid the added variability and response burden that would arise if all children in a small program had to be taken with certainty). This procedure reduced the total sampling frame from 223 to 184 stand-alone grantees/delegate agencies or combined groups (groups are treated as a single unit for sampling purposes).

The 184 grantee/delegate agency groups were then stratified to ensure representation of important groups of programs and children. The variables used for this purpose (from highest to least importance) were:

• Urban Location (Central City, Other Urban, Rural/Small Town)

• Auspices (School based vs. all other agency types)

• Percent Hispanic enrollment (> 40% vs. <= 40%)

• Percent Black enrollment (> 40% vs. <= 40%)

• Program Option (Part day only, Full day only, Both)

• New 3-Year Old Enrollment as a percent of total new 3- and 4-year old enrollment

Within each sampled cluster, grantees/delegate agency groups were sorted by the stratification variables, and an initial sample of 3 programs per cluster was selected with probability proportional to the total number of newly entering three- and four-year olds. A total of 90 individual programs were selected for the main sample, representing a total of 76 grantee/delegate agency groups. (Exhibit 7 shows the extent to which the sample is representative of the frame across the stratification variables).

Three of the 90 programs were eliminated prior to beginning recruitment. One program closed and two others were part of another current Head Start study, the Quality Research Center Study. Each of these programs was part of a grantee/delegate agency group, so all 76 groups remain with a total of 87 programs.

Exhibit 7:

Distribution of Sample vs. Frame vs. PIR, Weighted by Total Age 3 to 5 Enrollment

Variable	Percent for Sample (n=90)3	Percent for Frame (N=223)4	Percent for PIR (N=1715)
Urban Location5
Central City	49	46	40
Other Urban	30	31	35
Rural/Small Town	21	23	25
Head Start Region
Northeast	25	25	23
South	39	41	34
North Central	22	19	20
Plains	4	3	7
West	10	12	16
Auspices
School-based	7	10	13
Other	93	90	87
Pre-K Program State Expenditures
Has comprehensive programs similar to Head Start	24	27	28
Has some comprehensive programs similar to Head Start	21	20	18
Remaining States	55	53	54
Percent Hispanic
High (> 40%)	28	31	28
Low ( 40%)	72	69	72
Percent Black
High (> 40%)	44	42	41
Low ( 40%)	56	58	59
Hispanic Enrollment	32	31	28
Black Enrollment	39	39	37
Other Enrollment	29	30	35
Age 3 Enrollment	35	34	35
Age 4 Enrollment	62	61	60
Age 5 Enrollment	3	5	5

Initial Recruitment Contacts. The core of the approach in working with grantees/delegate agencies selected for the main study was to establish strong partnerships with the grantees/delegate agencies, actively address potential concerns, and work to mitigate Head Start staff and study participants’ concerns regarding issues such as random assignment, to the extent possible. Additionally, our goal was to coordinate program recruitment efforts in close collaboration with regional office staff in order to minimize any potential confusion with the requirements of study participation and normal program compliance or monitoring requirements.

To accomplish these goals, study staff were divided into recruitment teams, with each team responsible for recruiting about ten grantees. The recruitment process involved ongoing telephone and in-person contact with each grantee and their selected centers. On average, three site visits were made to each site at strategic points in the recruitment process. Each grantee and associated center also had a local site coordinator assigned to maintain ongoing personal contact to allay concerns about random assignment, help gather the information necessary to conduct random assignment, and to eventually supervise the data collection.

Once the 87 grantees were selected, letters, followed by telephone calls were made to grantee directors to begin to engage their cooperation. Site visits were conducted in the summer of 2001. All 87 grantees were visited and provided information about the study and its random assignment design. Meetings were held with key personnel at the grantee level and initial participation agreement was sought. Overall there were acceptance of the study and willingness on the part of all but two grantees to continue negotiating the conditions of participation. Various issues were raised, but the grantees were very willing to continue the dialogue to address the issues. Our approach was to consider ways in which the procedures can be adjusted to meet individual site needs while at the same time maintain the integrity of a random assignment design

A key to successful site recruitment is to ensure program administrators that the use of random assignment will not impose too many burdens on potential participant families, nor generate dissatisfaction in this vital client population. The largest step toward this goal was the decision to conduct random assignment only in Head Start grantees/delegate agencies operating at or near full capacity and where there was an additional pool of unserved families in the community known to be interested in services. This ensured that the evaluation randomly assigned children to the control group only in communities where Head Start grantees/delegate agencies currently cannot serve all eligible applicants due to limitations in the number of funded slots.

Selecting Centers Within Grantees/Delegate Agencies. After the first site visit, all of the recruited grantees/delegate agencies were asked to complete a Center Information Form (CIF) in late 2001 for each of their currently operating centers, as well as for any centers that were expected to be in operation as of Fall 2002. This information provided center-level verification of recruitment and enrollment data comparable to PIR data and was used to determine if a particular center could supply the “extra” applicants needed for the comparison group. Centers were initially eliminated from further consideration using the grantee-reported information if they met all three of the following conditions:

1. Overall Under-enrollment: Center is under-enrolled by more than 10%, i.e., [(Total slots) – (Total enrollment)] > 10 percent of Total slots.

2. No Waiting List: There are no income-eligible children on the center’s waiting list.

3. No fully-enrolled options: None of the center’s available service options are fully enrolled.

This was a process that tried to eliminate centers that were clearly saturated without eliminating too many centers on the basis of what is likely to be uncertain information. 1,411 center information forms were submitted and this step eliminated about 168 centers, (approximately 12 percent of all centers). Regional offices were also consulted about the enrollment numbers of centers. A list of centers, along with any saturation concerns were sent to regional office staff for their verification.

Forming Center Groups. The sample design called for sampling 16 treatment and 11 control children in their first year of Head Start per center. However, only about 48% of the centers mailing in CIFs reported a first year enrollment of at least 27 children. As with grantees/DAs, “small” centers (i.e., those with relatively few newly-entering children) were combined into “center groups,” ⁶ each with a combined reported first year enrollment of at least 27. The likelihood of each center providing the required sample sizes of treatment and control children was taken into consideration in forming center groups. If a center was underenrolled relative to the number of funded slots allocated to it, or if it reported having no waiting list, it was more likely to be combined with another center, even if its reported first year enrollment exceeded 27. Geographical proximity was also a consideration in grouping centers. Each center was ranked from 1 to 8, with 1 indicating the center was very likely to meet the study requirements and 8 indicating very unlikely. Centers with a score of 8 were deleted from the frame. Each center group was treated as a single unit for sample selection, and the average target sample size of program and comparison group children was sampled from the combined unit.

Stratification of Centers. The center groups were implicitly stratified within each program prior to sampling by sorting them. When used in conjunction with systematic sampling, sorting improves the representativeness of the sample. We used implicit rather than explicit stratification because the number of potential strata in each program generally exceeded the target sample size. Note, however, that in many programs the sorting variables were not especially useful because of the extremely high sampling fractions in most center groups.

The variables used for sorting were obtained from the CIF. These variables were, in order of priority, 1) whether the center is school-based (i.e. teachers are hired by the school district:Yes/No), 2) the percent Hispanic enrollment and/or the percent Black enrollment, 3) program option (full-day, part-day, both/home-based/other), and 4) the percent of first year three year old enrollment. When the school-based or program option variables differed for centers comprising a center group, the group was assigned to the category with the largest first year enrollment. In programs where there was little or no variation on a particular variable, the variable wasn’t used for sorting. In a few programs, the priority of the sorting variables was altered to reflect the particular distribution of centers within the program. When the percent Hispanic or percent Black were not the last sort variables, they were categorized into High/Low based on a cutoff which could differ for each program, depending on the distribution of race/ethnicity in the program.

Initial Center Selection and Screening. A larger than needed sample of centers (or center groups) was randomly selected from each grantee/DA, up to about 6 centers/groups per program. Centers/groups were selected with probabilities proportional to each center’s estimated enrollment of newly-entering three- and four-year old children (i.e., larger centers/groups had a higher chance of being selected). We then randomly selected an average of 3 centers/groups per grantee/DA to serve as a “main” sample with the remainder serving as a “reserve” to be used if we can’t obtain our desired sample of children from particular types of centers, or for particular types of children, across the entire study, not just from within any given grantee or cluster. Very small programs were given a minimum allocation of one center to ensure they could participate in the study, since they had already been recruited. In a number of programs the target sample size exceeded the number of center groups available for sampling, due to extensive collapsing in the program. In these programs we sampled all center groups, and there was no reserve sample.

When the target sample size did not exhaust the number of center groups available, we assigned a reserve sample size of two center groups if the target was less than five, and three if the target was five or more. In a few programs there was only one center group available for the reserve sample.

The sample was drawn in two stages in order to select both a main and a reserve sample. At the first stage, the center groups were first sorted within each program, then sampled with probability proportional to the first year enrollment for the center group. The first stage sample size was calculated to be large enough to accommodate both the target and reserve sample sizes. At the second stage, the first stage sample was sorted in the order of selection and an equal probability sample of center groups was selected. The sample size at the second stage was equal to the target. Center groups selected in the initial sample but not at the second stage became the reserve sample.

In programs where all first stage center groups were sampled with certainty (i.e. a census) but the target sample size did not exceed the number of center groups available, a reserve sample was possible and a second stage sample was selected as described above. In other programs the first stage sample contained a mixture of certainty and noncertainty selections due to a large measure of size for some center groups. In these programs the certainty selections at the first stage were retained for the main sample, and the remaining number needed to complete the target sample size were sampled at the second stage. The certainties from the first stage and center groups subsampled at the second stage are the main sample, and the remaining center groups not selected at the second stage are the reserve sample.

Two centers were given a zero measure of size to avoid sampling them at the request of the ACYF, to avoid conflict with another Head Start study (QRC). An adjustment to the center base weights will be done to compensate for their exclusion from the frame. Center sampling was completed in 85 of the 90 main sample Head Start programs.

Final Center Selection. A second round of discussions and site visits conducted in January-April, 2002 focused on the centers selected for the “main sample” to determine if they were able to provide the desired sample of treatment and comparison group children (i.e., they are not “saturated”). This involved verifying enrollment information for each center/group. The results of this screening were well documented before we decided to eliminate a particular center from further consideration. It is also important to note that for the small number of centers that have multiple service options (e.g., full- and part-time), we sought to conduct random assignment for any un-saturated service option(s). In a similar manner, we conducted random assignment for only one age group (e.g., only three-year olds) if the other was found to be saturated due, for example, to competition for four-year olds among preschool programs in the community.

Select Appropriately-Sized Samples of Head Start Children. In the selected Head Start centers, spread across the sample of study grantees/delegate agencies, we proposed to select an initial sample of 3,137 newly entering three-year-old participants and 2,541 newly entering four-year-old participants. As shown in Exhibit 8, we anticipate that a total of 1,882 three-year-olds will be assigned to the treatment group and 1,255 to the comparison group, while a total of 1,524 four-year-olds will be assigned to the treatment group and 1,017 4-year-olds to the comparison group. This exhibit also indicates the anticipated sample sizes for each wave of data collection. To obtain a final sample of 1,667 three-year-olds and 1,667 four-year-olds at the end of the study period—the size needed for adequate statistical precision—we estimate a beginning sample size of 3,137 three-year-olds and 2,541 newly enrolled four-year-olds.

Exhibit 8

Expected Sample Size At Each Wave Of Data Collection⁷

COHORT 1: Two Year Participants (Three-Year-Olds)

	Participating Grantees/Delegate Agencies
Treatment	Comparison	Total
At Random Assignment	1,882	1,255	3,137
Fall 2002 HS	1,694	1,130	2,824
Spring 2003 HS	1,524	1,017	2,541
Fall 2003 HS	1,372	915	2,287
Spring 2004 HS	1,235	823	2,058
Spring 2005 K	1,111	741	1,852
Spring 2006 1st grade	1,000	667	1,667

COHORT 2: One Year Participants (Four-Year-Olds)

	Participating Grantees/Delegate Agencies
Treatment	Comparison	Total
At Random Assignment	1,524	1,017	2,541
Fall 2002 HS	1,372	915	2,287
Spring 2003 HS	1,235	823	2,058
Spring 2004 K	1,111	741	1,852
Spring 2005 1st grade	1,000	667	1,667

Final Recruitment Steps. The final steps of recruitment included verifying any saturation problems of selected centers, gaining center staff cooperation, developing specific random assignment procedures, and coming to a final agreement about study procedures in each individual site. Site visits were conducted in January – April 2002 and when needed, again in the early summer of 2002. Partnership agreements were signed with each grantee/delegate agency, outlining agreed upon study procedures. Random assignment procedures were implemented in each site as soon as their enrollment decisions were made. The recruitment process entailed ongoing contact with the sites via personal visits and telephone calls. Each recruitment effort worked through any potential concerns about participating in the study, develop individualized study plans with the grantees, and obtained information on the community context.

1 Northeast: Maine, New Hampshire, Vermont, Massachusetts, Connecticut, Rhode Island, New York, New Jersey, Puerto Rico, Pennsylvania, Delaware, Maryland, West Virginia, Virginia, and the District of Columbia, a total of 15 states. South: North Carolina, South Carolina, Georgia, Florida, Mississippi, Tennessee, Kentucky, Alabama, Louisiana, Oklahoma, Texas, New Mexico, and Arkansas, a total of 13 states. North Central: Ohio, Indiana, Illinois, Michigan, Wisconsin, and Minnesota, a total of 6 states. Plains: Nebraska, Iowa, Missouri, Kansas, North Dakota, South Dakota, Montana, Wyoming, Colorado, and Utah, a total of 10 states. West: California, Arizona, Nevada, Idaho, Washington, Oregon, Alaska, and Hawaii, a total of 8 states.

2 Defined using Census Bureau designations for Metropolitan Statistical Areas (MSA’s) and urbanicity (i.e., Beale Codes).

3 Also weighted by PSU and program weight to reflect PSU and program probabilities of selection

4 Also weighted by PSU weight to reflect PSU probabilities of selection and by inverse of probability of selection for subsampling programs within LA county, Chicago, and Brooklyn prior to screening.

5 Using Beale code: 0 = Central City, 1-3 = Other Urban, 4-9 = Rural/Small Town

6 To protect ourselves against unexpected problems, those centers that appeared to be “almost saturated” from the previous step were given a reduced estimated number of newly-entering children (this reduces their probability of being selected, and often results in their being combined with other centers thereby expanding the potential pool of study children).

7 Includes an assumed 10% attrition rate each year.

2