Supporting statement Part B

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL
METHODS

B1.

Statistical Design and Estimation

B1.1

Introduction
The spring 2007 eighth grade Early Childhood Longitudinal Study-Kindergarten

Class of 1998-99 (ECLS-K) data collection will be for a longitudinal sample from the student
population who remained eligible (e.g., were not out of the country, deceased, etc.) after the
spring 2004 fifth grade data collection, regardless of their fifth grade response status. The fifth
grade sample included base year (i.e., kindergarten) respondents and a supplemental sample of
first graders who were not enrolled in kindergarten during the base year and so had no prior
chance of being sampled. The base year students were selected randomly within a sample of
schools located within the boundaries of a geographically representative, demographically
heterogeneous sample of 100 primary sampling units (PSUs).
In the spring 2007 eighth grade data collection, care will be taken to use procedures
to avoid the potential deterioration of the sample due to attrition and mobility. Procedures similar
to those used in the fifth grade rounds of data collection will be used to address this concern.

B1.2

Sampling
A major concern of any longitudinal survey is retaining the sampled members across

the entire duration of the study because attrition may bias the estimates. For the ECLS-K, there is
considerable evidence that students who are mobile are more likely to be “at risk” and have
different characteristics than those who are not mobile, thus increasing the chances for bias if
mobile students are not retained in the study.
In the spring 2004 follow-up data collection, various subsampling rates for certain
subgroups of movers to control the costs of the study were used. The subsampling rates varied
between 15 percent for non-language minority movers with missing data points to 75 percent for
language minority movers with longitudinal data. The plan for the eighth grade sample is to

B-1

include all the respondents from the spring 2004 data collection and follow all movers without
any subsampling to ensure that the final eighth grade sample is sufficiently large for data analysis.

B1.3

Analytical Subgroups and Sample Sizes
In the ECLS-K, the student is the ultimate analytic unit; all other respondents are

linked to the student as contextual data sources.
The spring 2004 fifth grade data collection began with 16,143 sample students (after
the exclusions of groups of movers detailed in section B1.2) and resulted in 12,129 students
eligible for the eighth grade data collection. The expected number of respondents at the end of the
eighth grade followup is approximately 9,600, assuming that the moving and response rates of the
students are similar to those from previous rounds. This sample will allow analysis of the eighth
grade students by subgroups such as the type of school (public/private, Catholic/non-Catholic
private), race-ethnicity (Hispanic, black, Asian/Pacific Islander, other), geographic region, level
of urbanicity (rural, nonrural), and language minority status, with subgroup sample sizes ranging
from approximately 800 (Asian/Pacific Islander) to 7,500 (public school).

B1.4

Sample Weights and Standard Error Calculations
Both cross-sectional and longitudinal student sample weights for the eighth grade

will be computed at the student level. The creation of replicate weights (for jackknife repeated
replication) and stratum and PSU identifiers for Taylor series variance estimation has been a
standard feature of the ECLS-K weighting procedures in previous rounds. These procedures will
continue to be implemented for the eighth grade using the methods that are already in place.
Standard errors and design effects for illustrative survey items will be presented in
the user’s manual for all students. Mean design effects will be presented for subgroups defined by
type of school, gender, race-ethnicity, region, and urbanicity.

B-2

B2.

Data Collection Procedures

B2.1

Tracking the Sample
In Phase III a tracking system database was developed for the ECLS-K and the

following sample tracking activities were conducted: (1) the entire household (parent) address
database was submitted to postmasters for address corrections (ACR); and (2) a respondent
mailing was sent to parents, asking them to report any changes in sampled students' schools
and/or home addresses. Household updates resulting from those two activities were recorded in
the ECLS-K tracking system database. Building on these tracking activities from Phase III, the
updated information about the household location of sampled ECLS-K respondents (parents and
students) will continue to be collected. In addition, a function will be added to the ECLS-K
tracking system to collect information about changes in students' schools.
In October of the 2005-06 school year, a respondent newsletter similar to the one
that was produced and mailed after spring-fifth grade was mailed to the ECLS-K households. The
newsletter contained information from the fifth grade data collection. As with the Phase III
respondent tracking mailing, households will be encouraged to contact the contractor regarding
updates to their household location and their students' schools.
Households that are marked as “unlocatable” in the tracking system will be
subjected to telephone followup.
The advance school contact described in the eighth grade data collection procedures
that follow will also identify students who have changed schools. Information about these
students' new schools will be recorded and updated in the ECLS-K tracking system database in
preparation for national data collection.

B2.2

Eighth Grade Study
Topics covered in this section include advance contact with schools, conducting

student assessments, conducting parent interviews, and distributing and collecting hard-copy
instruments.

B-3

Advance Contact. Eighth grade data collection will begin in fall 2006 with an
advance contact with the schools of the ECLS-K students. The advance contact will alert the
school coordinators to the upcoming data collection scheduled for spring 2007 and ask them to
provide some information regarding the sampled ECLS-K students enrolled in their school,
including (1) student’s enrollment status (withdrawn or currently enrolled in their school);
(2) student’s grade; (3) student’s English, mathematics, or science teacher's name, classroom
number, and class time; (4) information on any accommodations the student requires for the
direct assessment; (5) if the student receives any special education services; and if so, (6) the
primary special education teacher’s name. In addition, school coordinators will be asked to
provide any possible information regarding the new (transfer) school into which withdrawn
students may have enrolled.
Hard-Copy Instrument Mailout. The advance school contact will have identified
most of the sampled students' English, mathematics, or science teachers and special education
teachers, as well as the school administrators, in fall 2006. Beginning in mid February 2007, the
contractor will mail the hard-copy questionnaires for these respondents, embedded within
information packets, to the schools. The information packets will be addressed to the school
coordinator with instructions to distribute them to the identified teachers and the school
administrator. Each packet will include the following (1) the questionnaire(s) to be completed;
(2) an introductory letter with instructions on how to complete and return the instruments; (3) an
addressed, postage-paid return mailer; (4) ECLS-K fact sheet and newsletter and (5) a $25 check.
Student Assessment. Each sampled student will complete a self-administered
assessment. Each of the assessment sections (reading, mathematics, and science) has a routing
test that directs the student, depending on the student’s skills, into one of three second-stage tests.
In addition, the assessment will include a self-administered socioemotional questionnaire (see
below), as well as collecting the student’s height and weight. The test will be administered in
small groups or individually in schools where only one ECLS-K student is enrolled and will last
on average about 65 minutes.
Student Questionnaire. The students will complete a questionnaire that asks about
their dietary habits, schooling, and experiences in and outside of school. The student
questionnaire will average about 20 minutes to complete.
Parent Interview. The parent or guardian will be interviewed over the telephone or
in person. The parent instrument will ask about family structure, family practices, and parental

B-4

involvement in school. Parents will also be asked to provide their opinions on their students' level
of physical functioning, health, and disability status. Because it will have been 3 years since
family information was collected, the parent interview will also include updates to the household
composition, family income, education levels, and other demographic indicators.
Teacher, and School Administrator Questionnaires. English, mathematics, or
science teachers and special education teachers of sampled students will be asked to complete
self-report questionnaires about their background and curriculum and instructional practices for
the eighth grade year.
Teachers (i.e., subject matter and special education) of the sampled eighth graders
will also complete rating scale forms, which assess the student’s socioemotional and cognitive
skills. These forms are self-reported and will be scanned after data collection is complete. The
teacher rating scales will provide data from sources that have first-hand knowledge of the student
and his/her activities in a school environment.
School administrators will complete self-report questionnaires. Information about
the school administration, the staff, and the building will be collected through these
questionnaires.

B3.

Methods for Maximizing Completion Rates
This section describes the methods the contractor will use to maximize completion

rates for student assessments, parent interviews, and hard-copy questionnaires.

B3.1

Working with Schools
School participation is integral to the success of the study. Without the school’s

cooperation, there can be no school, teacher, or student data collection activity for the facility.
NCES recognizes that administrators will assess the burden level before agreeing to
participate. To minimize this perceived burden, NCES intends to continue its use of two strategies
that have not only worked successfully on two other major NCES studies (HS&B and NELS:88)

B-5

and its in-school followup studies but were also used in Phase III of the ECLS-K with OMB’s
approval.
First, the contractor will request the school to appoint a member of the school staff
to serve as a liaison with the contractor’s field staff. In a few cases, (i.e., schools with grade spans
to 8th grade), participating schools have already done this in Phase III. The requirements of this
study are extensive, and the coordinator will arrange the following:
Logistics of scheduling and assembling sampled students for groupadministered assessments;
Space for conducting the assessment;
Distribution and collection of teacher questionnaires and rating scale forms;
and
Assembly of specific enrollment and locating information (for following
transfer students).

B3.2

Student Assessments
There are two main areas in which completion rates can be maximized for the

student assessments: (1) accommodating students who are absent on scheduled assessment days
and (2) locating students who move.
Absent Students. Some students will be absent from school during the time that
assessments are scheduled at any given school. The contractor will set aside days throughout the
field period in which some field staff have no assessments scheduled, so that make-up
assessments can be more easily conducted. As in Phase III, a make-up assessment will be
conducted for any student who can be assessed during the field period. If an in-school assessment
cannot be scheduled, field supervisors will contact parents to make arrangements for in-home
assessments for absent students.
Locating Students Who Move. Locating transfer students who move, and
particularly the new school in which they are enrolled, is critical in maintaining high completion
rates for student assessments overall. As discussed in section B2 (Data Collection Procedures,
Tracking the Sample), the ECLS-K has established an effective tracking system and database for
students and parents. The data collection plan to contact schools in the fall of the school year

B-6

provides early information on the schools of previously identified students who move, as well as
newly identified students who move. This early identification of these students gives the
contractor ample time to locate them prior to the beginning of the national study field period and
ensures contact with new districts and schools in time to arrange for assessments early in the
spring of the school year.

B3.3

Parent Interviews
There are three main areas in which completion rates can be maximized for the

parent interviews: (1) flexibility in scheduling interviews, (2) locating parents of students who
move, and (3) avoiding refusals and converting initial refusals to completed interviews.
Flexibility in Scheduling Interviews. Effective calling patterns are essential for
achieving high response rates on all telephone surveys. Previous experience shows that individual
respondent schedules (work, classes, recreational activities, vacations, etc.) have a more negative
effect on the level of nonresponse when call attempts are limited to a short time span. A larger
percentage of the initially noncontacted numbers will be converted to a positive outcome if the
call attempts are distributed. Completion rates improve when interviewers call on different days
of the week and at varying times of the day and evening.
To establish initial contact with a parent of a sampled student, field staff will be
trained to place two day, three evening, and two weekend calls over a two-week period. These
calls will be made in a nonsequential set of targeted time periods called “time slices.” The time
slices and required number of calls are as follows:
Required Number of Calls
Weekday 10 a.m. to 3 p.m.

1

Weekday 3 p.m. to 6 p.m.

1

Weekday 6 p.m. to 9 p.m.

1

Weekday 6 p.m. to 7:30 p.m.

1

Weekday 7:30 p.m. to 9 p.m.

1

Anytime Saturday or Sunday on separate weekends

2

B-7

If, after seven call attempts, no contact has been made with the parent, the field staff
will be instructed to review the case with the field supervisor for additional instruction on how to
proceed. The supervisor may instruct the field staff to conduct one or more of the following
procedures: (1) send a letter to the parent; (2) contact the school coordinator to see if the school
can help or offer any insight into contacting the parent; (3) contact one of the “other contacts
listed for the parent”; or (4) contact the “nonresident” parent, if applicable.
Once contact is established, up to seven additional calls will be made to complete
the parent interview. If the interview is not completed and the respondent has not refused, the
field staff may be instructed to attempt an in-person interview. During the last few weeks of data
collection, noncontact and uncompleted cases will be visited in-person as appropriate to improve
response rates.
Locating Parents of Transfer Students. Locating parents of students who move is
critical in maintaining high completion rates for parent interviews overall. As discussed in section
B2 (Data Collection Procedures, Tracking the Sample), the ECLS-K has established an effective
tracking system and database for students and their parents. Early identification of the students
who move will also provide ample time to locate the parent.
Refusal Avoidance and Conversion Procedures. Another technique that will be
used to maximize the parent interview response rate is to incorporate into the training sessions
techniques for averting refusals based on methodological work conducted by Dr. Robert Groves
and others. The averting refusal training materials include becoming thoroughly familiar with the
study including answers to frequently asked questions (FAQs) and respondent objections, drafting
responses in their own words to FAQs, practicing saying these responses, diagnosing respondent
objections and quickly responding with a response tailored to the objection. The training includes
self-analysis by recording responses and listening to them, preparing answers for different
situations, using the voice effectively, and role-plays between trainer and interviewer and
between interviewer and interviewer. Thus, averting refusal training will focus specifically on
addressing reasons for refusals on the parent interview component of the ECLS-K study.
During the parent interview data collection period, supervisors and field managers
will review initial refusals with the field staff, putting a particular emphasis on reviewing the
interviewer record of calls, which will be available to supervisory staff on a weekly basis. If a
refusal occurs, the interviewer will be instructed to record key demographic information about the

B-8

refusing respondent and the respondent’s reason(s) (if given) for refusing to participate. This
information will be evaluated by the field supervisor to convert refusals.
Cases identified for refusal conversion will be assigned to a select group of field
staff identified as possessing the necessary skills to act as refusal converters. Field managers will
hold telephone conferences with the identified field staff to review the refusal conversion
procedures and discuss strategies for converting refusals.

B3.4

Hard-Copy Instruments
There are three main areas in which completion rates can be maximized for hard-

copy instruments: (1) early distribution of instruments to schools that have participated in prior
rounds of data collections and schools new to the study, (2) effectively communicating the
importance of school administrator and teacher participation, and (3) the efforts made by
supervisory staff to avoid refusals and to convert initial refusals to cooperating respondents.
Early Distribution of Instruments. Feedback from prior round participating school
administrators and teachers indicated that there would be increased participation if they have
more time to complete the hard-copy instruments. As described earlier, the advance school
contact will identify most of the sampled students' English, mathematics, or science teachers and
special education teachers, as well as the school administrators, in the fall of the school year. The
plan is to send these school and teacher questionnaires, along with an information packet,
beginning in January of the school year. This schedule will allow two months of additional time
for these respondents to complete the instruments and hold them for pickup by the field staff.
Effective Communication with New Transfer Schools. The participation of newly
identified transfer school administrators and teachers can be increased by effectively
communicating information about the ECLS-K, including the goals of the study, what the study
measures, the various components, why it is important that schools and teachers participate, the
study activities to date, the plans for the future, and some overall results from previous rounds of
data collection. Effective respondent materials, as well as telephone contact early in the data
collection field period by supervisors who are trained to convey this information efficiently and
completely, will help maximize the participation by new transfer schools.

B-9

Refusal Avoidance and Conversion Procedures. As part of the spring-first grade
hard-copy data retrieval effort conducted in fall 2000, an effective set of procedures to avoid
refusals was developed. These procedures will be incorporated into the supervisor training
sessions. Much like the Averting Refusal Training described above for ECLS-K parent
interviews, the training session for hard-copy refusal aversion will include analyzing the reasons
for refusal on the ECLS-K study, preparing answers for different situations, using the voice
effectively, and role-play situations between trainer and interviewer and between interviewer and
interviewer. Thus, this averting refusal training will focus specifically on addressing reasons for
refusals on the hard-copy instrument components of the ECLS-K study.

B4.

Individuals Responsible for Study Design and Performance
The following individuals are responsible for the study design and the collection and

analysis of the data on ECLS-K.
Chris Chapman, NCES

(202) 502-7414

Elvie Germino Hausken, NCES

(202) 502-7352

Peter Tice, NCES

(202) 502-7497

Jill Walston, ESSI

(202) 403-6155

Karen Tourangeau, Westat

(301) 251-8265

Christine Nord, Westat

(301) 294-4463

Thanh Le, Westat

(301) 610-5105

B-10