Att_Part B_ECLS-K_11_final
Att_Part B_ECLS-K_11_final.doc
Early Childhood Longitudinal Study Kindergarten Class of 2010-2011
OMB: 1850-0750
Collection of Information Employing
|
B |
B.1 Universe, Sample Design, and Estimation
This submission requests clearance for both the field test and kindergarten year full-scale data collection for ECLS-K:11. Section B.1 includes information on the universe of interest and the sampling plan for the national study, which describes the plans for selecting primary sampling units (PSUs), schools, and kindergarten children within schools. Also discussed in Section B.1 are the precision requirements and target sample sizes. The sample design for the field test is described in Section B.4.
B.1.1 Universe and Sample Design
The universe for the ECLS-K:11 includes all children attending kindergarten in the 2010-11 school year in the 50 States and the District of Columbia. The sample design for ECLS‑K:11, which will be similar to that for the ECLS‑K, will produce a sample that is nationally representative of this population of children in the United States. In the base year (i.e., kindergarten year), children will be selected using a multistage probability design. In the first stage, 90 primary sampling units (PSUs) that are counties or groups of counties will be selected with probability proportional to size (PPS). In the second stage, public and private schools offering kindergarten programs will be selected, also with PPS, within the sampled PSUs. This stage will include oversampling of private schools to ensure that the sample includes enough students attending private schools to generate reliable estimates about them. The third-stage sampling units will be children in kindergarten programs and children of kindergarten age in ungraded schools and classrooms. Children will be selected within each sampled school using equal probability systematic sampling. Asians, Native Hawaiians, and Other Pacific Islanders will be sampled at a higher rate so as to achieve a minimum required sample size in order to generate reliable estimates for them. Although they will be oversampled as one group, the number of completed interviews for children in each of these groups is expected to be large enough in the kindergarten year to produce estimates for each group separately. Only base year respondents will be included in the sample in subsequent years of the study. Due to the high cost of following children who change schools (i.e., “movers”), children who move from the school they attended in kindergarten will be subsampled for follow-up and inclusion in later rounds of collection. The subsampling rate will be around 50 percent but will vary between grade 1 and grade 5 by children's characteristics in order to preserve large enough groups of sampled children that are of particular analytical interest (e.g., language minority children (children from a home in which the primary language is not English)).
B.1.2 Precision Requirements and Sample Sizes
An objective of ECLS‑K:11 is to obtain a minimum level of reliability for estimates pertaining to analytical subgroups, such as Asians, Native Hawaiians and other Pacific Islanders , Blacks, Hispanics, private school kindergartners, and language minority children. Four precision requirements for the survey are identified and form the basis for the base year sample design and plans for the subsequent rounds. These requirements are the ability to do the following:
Measure a relative change of 20 percent in proportions across waves;
Measure a relative change of 5 percent in a mean assessment score across waves;
Estimate a proportion for each wave with a coefficient of variation (CV) of 10 percent or less; and
Estimate a mean assessment score for each wave with a CV of 2.5 percent or less.
The precision requirements that drive the sample design, which are the same as those used in the ECLS-K, are related to the ability to estimate changes over time and the precision of estimates in the grade 5 data collection for the sample as a whole, as well as for subgroups of analytic interest. The ECLS-K:11 sample design began with the assumption, based on the ECLS-K experience, that at least 10,300 completed cases were needed by the end of 5th grade to satisfy the study's precision requirements.
For the ECLS-K:11, the minimum subgroup sample size is determined by first solving for the sample size needed to achieve the precision requirements under simple random sampling with 100 percent overlapping samples between waves using the formula:
where n is
the sample size per wave, α is the
significance level, β is the power term, z has the
standard normal distribution,
is the correlation between two waves, P1
and P2
are the two proportions being compared, Q1=1-
P1,
Q2=1-P2,,
,
and
.
When α=0.05,
β=0.80,
=0.75,
P1=0.30
and P2=0.36,
the sample size needed per wave is 241.1
Assuming a design effect of 4 (based on the ECLS-K), this subgroup
sample size would need to be further increased by a factor of 4 to
964, since the effective sample size is equal to the sample size
actually obtained divided by the design effect.
The assumptions used to arrive at the sample size by the end of the longitudinal study include the rates at which children move from the base year sampled school to other schools, the rates at which the movers will be subsampled, the rates at which the subsampled movers will be located, and the child completion rates. We now know the movements of ECLS-K children after each data collection year and how successful we were at locating them for follow-up, and we have modeled the assumed rates for the ECLS-K:11 on this experience. In ECLS-K, children who moved to another school (but not necessarily residence) were followed at a rate of 50 percent in grade 1, slightly higher in grade 3 so that all language minority children were retained, and slightly lower in grade 5 to accommodate a reduction in the overall sample size. The grade 5 subsampling rates varied according to child characteristics with the highest rate applied to language minority children. For the ECLS-K:11, the overall subsampling rate will be 50 percent, with differential rates for subgroups of interest (e.g., a higher rate for Asians, Native Hawaiians and other Pacific Islanders, and language minority children).
For the ECLS-K:11, a sample of 900 responding schools (720 public and 180 private) with an average sample size of 23 children in each school will yield approximately 20,700 sampled children in the base year. Assuming a sample of approximately 20,700 kindergarten children in the base year, we expect that the sample size at the end of the grade 5 followup should be approximately 11,226 completes, which is higher than the minimum sample size of 10,300 needed to meet the study precision requirements. With the sampling rates for subgroups of interest described in the next section, the fifth grade sample size should be large enough to generate estimates that satisfy the precision requirements for each of the subgroups as well.
The four precision requirements are of equal importance for Hispanics, Blacks, and children of other races who are not part of the Asian or Native Hawaiian and Other Pacific Islander groups. However, these subgroups do not have an impact on determining the oversampling rates for special groups because their expected sample sizes exceed the required sample size for meeting the precision requirements. At the end of the grade 5 data collection for the ECLS-K, the distribution of completed cases for children who were a race/ethnicity other than Asian, Native Hawaiian, or Other Pacific Islander was 11 percent Black, 19.1 percent Hispanic, and 61.4 percent all other races. For the ECLS-K:11, by the end of the fifth grade collection, we expect to have approximately 1,399 Blacks, 2,150 Hispanics, and 6,900 children of other races who are not part of the Asian or Native Hawaiian and Other Pacific Islander groups. These sample sizes are large enough to satisfy all four precision requirements.
We noted earlier that language minority children are another subgroup of analytical importance. We expect that 25 percent of the base year sample and 26 percent of the fifth grade sample will be language minority children, based on the data from the ECLS-K. The higher percentage of language minority children in the fifth grade year was due to language minority movers having been retained at a higher rate than other groups of children. For the ECLS-K:11, with a base year sample of approximately 18,630 expected completes and a fifth grade sample of approximately 11,226 expected completes, we expect to have approximately 4,650 language minority completes in the base year and at least 3,000 language minority completes in the fifth grade year, given our plan to use higher mover retention rates for language minority children than for other groups. Language minority children will be identified by questions in the parent and teacher survey instruments asking about language spoken at home.
B.1.3 Sample of Primary Sampling Units
The first sampling stage is the selection of geographic areas or PSUs. Clustering the sample into relatively compact geographic units is necessary to control the cost of data collection. PSUs will be counties or groups or counties, instead of states or school districts. In most cases, a state is too large of a unit to reduce data collection costs, while school districts do not administratively include private schools and there is no clear mapping of which private schools fall within the geographic boundaries of public school districts. In addition, information on district-level enrollment would be needed to reflect the enrollment of the corresponding private schools under such a design. Counties, on the other hand, have well-defined boundaries and the use of combined counties has the additional benefit of providing a more heterogeneous area, which may reduce the variance of estimates due to clustering.
The sampling frame for ECLS-K:11 will be a sampling frame of PSUs. This PSU frame will be created using the 2007 population estimates from the Census Bureau. It will cover the 50 states and the District of Columbia. The PSUs will respect state boundaries and be created such that each one has a minimum population size of 380 5-year-old children. 2
A stratified sample of 90 PSUs will be selected with probabilities proportional to size. The PSU measure of size will be a function of the 5-year-old population in the PSU. Members of the Asian and Native Hawaiian and Other Pacific Islander race/ethnicity groups for which oversampling is required will contribute more to the measure of size, so that the probability of sampling PSUs with a large proportion of children in these groups is increased. Thus, the measure of size for a PSU will be the number of 5-year-olds who are not Asian, Native Hawaiian, or Other Pacific Islander plus 2.5 times the number of 5-year-olds who do belong to one of these race/ethnicity groups.
PSUs with large measures of size will be included with certainty. The remaining noncertainty PSUs will then be grouped into strata and two PSUs will be sampled from each stratum. Census region, level of urbanization, minority status (e.g., percent of the population in the PSU who are Black, Hispanic, Asian/Native Hawaiian/Other Pacific Islander,3 or American Indian), and income level will be used as stratification variables.
B.1.4 Sample of Kindergarten Programs
The second stage of sampling involves selecting samples of public and private schools that have kindergarten programs from within the sampled PSUs. The targeted sample size (i.e., the number of schools from which participation is needed) is 180 private and 720 public schools, for a total of 900 schools. We will initially sample larger numbers of schools to account for a certain level of expected school nonresponse. In the ECLS-K, private schools participated at a rate of 65 percent and public schools at a rate of 70 percent. Even though efforts will be made to raise the school response rate, we used the ECLS-K rates to compute the number of schools to be sampled initially. We will sample 180/0.65 or about 280 private schools, and 720/0.70 or about 1,030 public schools.
School Frames
Within each sampled PSU, the sampling frame will be the list of all public and private schools offering kindergarten in the PSU. For the ECLS-K:11, we will use the sampling frame being developed for the 2010 National Assessment of Educational Progress (NAEP 2010). The primary sources for the NAEP school frame are the most recent Common Core of Data (CCD) and Private School Survey (PSS), which are NCES universe surveys of all public and private schools in the United States. This frame will be supplemented with schools that educate children in the grades in which NAEP is administered that are not found in the CCD and PSS for some reason (e.g., the school is a new school and was not reported in the most recent CCD or PSS available). This sampling frame is expected to be ready by January 2009. It will include all grades from pre-kindergarten to grade 12 (even though NAEP only selects children in grades 4, 8, and 12). School enrollment by grade and race/ethnicity as reported in the CCD and PSS will be included in the NAEP school sampling frame. Charter schools will be included in the public portion of the school frame, and we expect to be able to sample them identically to all other schools in the frame.
Within each PSU, schools with fewer than 23 kindergarten children will be clustered together before sampling to ensure that the target sample size of about 20,700 kindergarten children is met. Schools (or combined schools) will be selected with probability proportional to size. As with the PSU sample, a weighted measure of size will be computed taking into account the oversampling of Asians, Native Hawaiians, and Other Pacific Islanders as follows:
where 2.5 is the
oversampling rate for Asians, Native Hawaiians, and Other Pacific
Islanders, 4
is the estimated count of Asian/Native Hawaiian/Other Pacific
Islander kindergarten children in the school, and
is the estimated count of all other kindergarten children in school j
in PSU i.
The target number of sampled schools per PSU will be calculated separately for public and for private schools and adjusted upward to offset anticipated school nonresponse and ineligibility rates. The number of schools allocated to each PSU will be set proportional to the weighted measure of size of the PSU.
An issue in the sampling of schools is the potential for overlap between ECLS‑K:11 and the 2010 NAEP school samples. The largest PSUs and a few of the smaller PSUs in both surveys will overlap, and the largest schools within these PSUs may be selected for both ECLS‑K:11 and NAEP because they are selected with probability proportional to enrollment. This could result in a high response burden for the schools selected for both surveys, resulting in lower school cooperation rates. We will consider minimizing the overlap between school samples in PSUs selected for both ECLS‑K:11 and NAEP. It should be noted, however, that no child within a school sampled for both the ECLS-K:11 and 2010 NAEP would be sampled for both studies, because the targeted grade levels are different (ECLS:K-11 will sample kindergarteners while NAEP will sample fourth graders).
B.1.5 Sampling Kindergarten Children, Parents, and Teachers
The goal of the ECLS-K:11 sample design is to obtain an approximately self-weighting sample of children, with the exception of Asians, Native Hawaiians, and Other Pacific Islanders who need to be oversampled to meet the sample size goals. Within each sampled school, field staff will obtain a complete list of kindergartners enrolled. Special care will be taken to avoid excluding children from the list because of disability or language barriers by having the field staff ask the school representative questions from a list designed to prompt for any exclusion of children with a disability or children whose native language is not English.
Two independent sampling strata will be formed within each school, one containing Asian/Native Hawaiian/Other Pacific Islander children and the second containing all other children. Asian/Native Hawaiian/Other Pacific Islander children will be sampled from the Asian/Native Hawaiian/Other Pacific Islander stratum with a sampling rate 2.5 times the rate of sampling for children who are not Asian, Native Hawaiian, or Other Pacific Islander. Within each stratum, children will be selected using equal probability systematic sampling. In general, the target number of children sampled in any one school will be 23. If the sampling unit is a cluster of small schools, the school rosters will be kept separate by appending them one after the other in the sampling frame. Sampling will be done systematically and with equal probability from the list so that if a school is small fewer children will be sampled from the school. If a school has 23 children or more according to the school frame but turns out to have fewer than 23 kindergartners enrolled at the time of child sampling, all children in the school will be sampled. Twins will not be identified prior to sampling but they may enter the sample through this method of probability sampling.
Once the rostered children are sampled, parent contact information for each child will be obtained from the school. The information will be used to locate a parent or guardian and obtain parental consent for the child assessment and for the parent interview.
Teachers who teach the sampled children also will be included in the study and will be asked to fill out a series of teacher questionnaires. All teacher data will be linked to the sampled ECLS-K:11 children they teach. This procedure for identifying teachers for the ECLS-K:11 is different from the procedure used in the ECLS-K where, during the fall-kindergarten data collection, a census of kindergarten teachers taken at each school.5
B.2 Procedures for the Collection of Information
The previous section (B.1) addressed the statistical methodology for the study, including stratification and sample selection, estimation, and precision requirements. This section addresses the procedures for the collection of information.
The national kindergarten data collection will include direct child assessments, height and weight measurement, a screening of children's hearing, parent interviews, school administrator and teacher questionnaires (both regular classroom and special education teachers), and child care provider interviews. Data will be collected twice, once in the fall and once in the spring. Computer assisted interviewing (CAI) will be the mode of data collection for the child assessment and the parent and child care provider interviews. School administrator and teacher data will be collected via self-administered questionnaires.
Pre-Assessment Activities. Data collection staff assigned to recruit schools for the study will work with the school coordinators to establish details of the schools’ participation. They will determine:
On-Site Schedule. The recruiter will discuss the schedule for data collection with the school coordinator. The dates for the fall assessment schedule and for obtaining the sampling lists will be set, making sure to avoid conflicts with any special events in the school’s calendar.
Assessment Locations. The locations within the school where the assessments will take place will also be determined. The goal will be to identify assessment locations that provide as little distraction as possible, that protect the privacy of the children, and that are as nondisruptive of school routine as possible.
Field supervisors will make an advance visit to each school. Based on our experience with the ECLS-K and other NCES studies such as NAEP, it should be possible to complete all the pre-assessment activities with one in-person visit and telephone follow-up. Throughout these pre-assessment activities, we will establish a positive and cooperative working relationship with school personnel and the school community.6
During the visit, field supervisors first will address any questions that the school coordinator or principal may have, then they will review the school coordinator’s role with him/her. A primary task to be conducted during the school visits is confirming the logistical arrangements for conducting data collection within the school that were determined at the end of the recruitment process. A checklist of the arrangements that need to be agreed upon and the tasks to be completed will guide the pre-assessment visit. At the time of pre-assessment visit, the supervisor will also determine whether the school coordinator is willing and available to conduct certain data collection activities, such as following up on consent forms. Other activities for the pre-assessment visit include: sampling kindergartners; identifying parents and teachers of sampled kindergartners; collecting contact information for parents; meeting with teachers; and distributing the teacher questionnaires.
Child Assessment
About 3 days will be required for the assessment visit in each school. The assessment team will arrive at the school on the appointed first day of assessments; they will immediately contact the school coordinator. The field supervisor will introduce the assessors to the school coordinator. The procedures to be used during the on-site data collection period will be discussed to ensure there is a common understanding of those procedures and that no changes in the location of assessments or other aspects of the on-site data collection will be made. The field supervisor also will confirm that all sampled children are still enrolled in the school and determine which children are at school that day. New contact information will be obtained for any children who may have left the school after sampling or between the fall and spring collections.
Assessors will be taken by school personnel to their assessment areas. The assessors will arrange their areas to remove potential distractions as much as possible and establish a comfortable environment for conducting the assessment. They will set up the assessment materials and log into the child assessment program on laptops that they will carry with them. All field staff will be provided with backup batteries, cords, etc., to ensure that data collection activities are not disrupted by equipment problems.
Once the assessment areas have been set up and assessors are ready to begin work, the school coordinator will introduce the ECLS-K:11 team members to the teacher(s) whose children will be assessed. The teacher, in turn, will introduce the assessors to the class. Assessors will then escort the sampled children to the assessment areas, one-by-one, and conduct each 60-minute assessment. As discussed in section A, the assessments will consist of the following: a direct cognitive assessment of reading, mathematics, executive functioning, and science skills and knowledge; measurement of children's weight and height, which will be obtained using instruments brought by the assessors; and a hearing screening.7 The hearing screening will be conducted with specialized equipment brought by the assessors. Students will be screened at the same time that the other parts of the study are being conducted, but because the screening must conducted in a quiet environment to obtain accurate readings from the instruments, students will be pulled out into a separate room to participate in this part of the study. We will ask the schools to volunteer a staff member familiar to the students to be present during the screening to help the children feel comfortable during the screening. Each child will then be returned to the classroom and the next sampled child will be assessed. At the end of each day, the data for completed assessments will be transmitted to a central database.
It is expected that some children will be absent from school when the assessments are scheduled. Certain days throughout the field period will be designated as days on which some field staff will have no assessments scheduled, so that make-up assessments can be more easily conducted on those particular dates. During the first wave of data collection, attempts will be made to conduct a make-up assessment for all children absent on their school’s assessment day who can be assessed at some point during the field period. Missing assessment data for sampled children during the first wave of data collection has implications for the ECLS‑K:11 throughout subsequent rounds, because it prevents the ability to look at growth in achievement across time, which is key purpose of the study. Therefore, it is important to conduct kindergarten assessments with as many children as possible.
Teacher and School Administrator Questionnaires
The field supervisor will identify the teachers of the sampled children who will receive the self-administered teacher questionnaire and enter their names into the field management system (FMS), creating a link between each sampled child and his or her teacher. This linking system was first developed and used successfully for the ECLS-K. When a child has more than one regular teacher, the field supervisor will apply a set of rules to select the focal teacher, defined as the one with whom the child spends the most time. If the child spends equal amounts of time with more than one teacher, a coin toss will be used to select one teacher.
The field supervisor will prepare the teacher materials at the time of the pre-assessment visit and will distribute the materials when he/she is introduced by the school coordinator to the teachers who will be asked to participate in the ECLS‑K:11. These materials will consist of a letter describing the ECLS‑K:11, a summary sheet enumerating the teacher activities involved in the study, and a copy of the ECLS‑K:11 brochure (see Appendix F for examples of these materials).8 The introductory meeting will afford an opportunity to explain in person the purpose and importance of the study and to establish a positive relationship with the teacher.
Distributing the Teacher and School Administrator Questionnaires. In the fall and spring kindergarten collections, teachers will be asked to complete self-administered questionnaires about their background, curriculum, and instructional practices. Teachers of the sampled kindergartners also will complete rating scale forms about the ECLS-K:11 children, which indirectly assess the children’s socioemotional and cognitive skills. The teacher rating scales will provide data from a source that has first-hand knowledge of the child and his/her abilities. In the fall, the field supervisor will identify teachers and distribute the teacher questionnaires and rating assessment forms. In the spring, teacher questionnaires will be sent to school coordinators who will be asked to distribute the teacher questionnaire and rating assessment forms. Teachers will be asked to complete the teacher questionnaires before the 3-day school assessment visit. The average number of children per teacher is expected to be about 4. We plan to offer each teacher an incentive of $7 per child rating form. The incentives will be included in the package of instruments the teachers receive in the fall and in the spring. Field supervisors will collect completed teacher questionnaires during the assessment visits.
In the spring, the teachers of sampled children who are receiving special education services, i.e., special education teachers, will be asked to complete questionnaires about their background and qualifications. They also will be asked to answer questions about the types of services the ECLS-K:11 child receives. The special education questionnaires will be distributed and collected in the same manner as the regular teacher questionnaires described above. We plan to offer each special education teacher an incentive of $7 per child rating form. The incentives will be included in the package of instruments the special education teachers receive in the spring.
Also, in the spring, school administrators will be asked to complete self-administered questionnaires. Information about the school administration, the staff, and the building will be collected through these questionnaires. The school questionnaire will be mailed to the principal or school administrator in advance of the spring assessment visit; the field supervisor will remind the school coordinator of the need to complete this instrument on the first day of assessments at the school. The field supervisor for each school will collect the school questionnaire during the on-site assessment visit. School administrators will receive a $25 incentive for completing the questionnaire, which will be attached to the school administrator questionnaire during the spring data collection. If the school questionnaire has not been completed by the beginning of the last day on-site, the field supervisor will remind the school coordinator about it again. If the school questionnaire still is not completed by the time the team has finished its assessment work at the school, the field supervisor will ask for a specific date from the school coordinator by which the school will send a completed questionnaire. Follow-up will continue until the questionnaire has been received.
Parent Interview
ECLS‑K:11 field staff who conduct the child assessments also will conduct telephone interviews with parents using a computer assisted telephone interviewing (CATI) instrument. Having the same staff members conduct both components of the study better links the activities that take place in the school with the parent interview, which may in turn promote greater parent participation. The list of parent interviews assigned to the field staff will be transmitted to the field staff via computer, with new cases being transmitted as they become available.
Flexibility in Scheduling Interviews. Explicit parental consent for child participation in the study is expected to be required in a significant portion of the schools. As a result, our field staff, who will be conducting most parent interviews, may have already contacted the parent to obtain this consent, at which time they may have been asked questions about the study, including whether the parent could schedule the telephone interview for a particular time. If the parent had not expressed a preference regarding the timing of the interview, procedures for conducting telephone interviews at times that are most convenient for parents and that allow sufficient flexibility will be used.
To establish initial contact with a parent of a sampled child, field staff will be trained to place two day, three evening, and two weekend calls over a 2-week period. If, after these seven call attempts, no contact has been made with the parent by telephone, the field staff will visit the child’s home to explain the study and attempt to complete an in-person interview. If telephone contact is established, up to seven additional calls will be made to complete the parent interview. If the interview is still not completed after seven calls and the respondent has not actively refused to participate, the field staff will attempt an in-person interview. During the last few weeks of data collection, cases that have not yet been contacted or completed, and that have not actively refused to participate, will be attempted as in-person interviews to improve response rates.
Non-English Interviewing. The ECLS‑K:11 sample design is expected to include a large proportion of children from linguistically isolated households. (A household is considered linguistically isolated if no one older than 14 speaks English very well.) In order to include these families in the ECLS‑K:11, special measures are required. It is expected that Spanish will be spoken in the majority of these households. Therefore, the parent interview will be fully translated into Spanish, and we will recruit field staff who are bilingual in Spanish and English to conduct these parent interviews. A number of Asian and other languages also will be spoken by parents of sampled children, but in much smaller numbers. The cost of having bilingual staff representing all these languages available in a large number of PSUs would be prohibitive, as is the cost of sending such staff out for extensive traveling across PSUs. We will recruit and train some bilingual staff for our field effort; however, our primary approach for conducting parent interviews in non-English languages will be to provide for bilingual interviewing by having staff working from their homes through a Telephone Research Center. This is a highly efficient approach to addressing the need for bilingual interviewing in a national sample during a short field period. However, it is cost-prohibitive to develop a full translation of the questionnaire for less common languages. For this reason, we plan on using professional interpreters, as available, for interviews in non-English, non-Spanish languages. This approach was used for telephone interviewing in another NCES study (the Early Childhood Longitudinal Study‑Birth Cohort (ECLS-B)). An interpreting service was used to obtain interpreters for about 20 languages who were connected in a three-way conference call with an English-speaking interviewer and the respondent. Evaluations of the quality and cost of data obtained in this way have established that it can be an efficient way to collect data from respondents who speak less common languages.
Wrap-Around Care and Education Provider Interview (WECEP)
Telephone interviews will be conducted with primary care and education providers for children who spend at least 5 hours a week in before- or after-school care arrangements. If a child has more than one weekly care arrangement, the primary provider will be identified as the one who provides the most care to the child on a weekly basis. The WECEP will only be conducted with the parent’s permission, which will be requested during the parent interview. If permission to contact the provider is granted, parents will be asked to provide contact information for the provider. For interviews with center-based care providers, the first portion of the WECEP interview, related to administrative aspects of the center, will be asked of directors of center-based programs, including school-based settings. However, many of the same questions are asked of home-based care providers in their interview as well. The second portion, the primary care provider section, will be administered to both center-based and home-based providers. Administration time is designed to be 10 minutes for the administrator section and 20-30 minutes for the primary care provider section. Provider interviews will not be conducted by the same field interviewers who assess children and interview parents. Instead, trained telephone interviewers will conduct the interviews from a centralized Telephone Research Center (TRC) using a CATI instrument. The TRC scheduling software will be used to deliver cases to interviewers after parent permission to contact the care providers is obtained. Interviews will be conducted in both English and Spanish. The Spanish interview will use a fully translated instrument. Translators will be used for interviews in non-English, non-Spanish languages, as was described above for the parent interview.
B.3 Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse
This section describes methods for securing cooperation and gaining consent for the ECLS-K:11 and the methods that will be used to maximize completion rates for child assessments, parent and wrap-around care provider interviews, and teacher and school administrator questionnaires.
A major challenge in any survey today is obtaining high response rates, and this is even more important in longitudinal surveys where nonresponse can occur at multiple time points and nonresponse at the base year affects all subsequent waves of data collection. As in most longitudinal surveys, attrition is closely associated with those persons who move between waves; however, moving is defined somewhat differently in ECLS-K:11 than in other surveys because it is triggered by a change in the school the sampled child attends, whether or not the child’s residence changes. In ECLS-K, 25 percent of children changed schools between kindergarten and first grade, and by the fifth grade 56 percent of children were in different schools than they were in for kindergarten. To the extent that parents take advantage of the opportunity to transfer their children from schools in “need of improvement” under NCLB, school mobility may be greater in ECLS-K:11 than it was more than a decade earlier for the ECLS-K.
The main problem associated with nonresponse is the potential for nonresponse bias in the estimates produced using data collected from those people who do respond. Bias can occur when the people who do respond are systematically different from the people who do not. Two approaches that will be used to reduce the potential for bias are designing the data collection procedures and methods wisely to reduce nonresponse and using statistical methods of sampling and weighting to reduce the effect of nonresponse on the estimates. While the statistical approaches are important in controlling biases and costs, the data collection procedures are at the heart of a successful longitudinal survey.
B.3.1 Gaining Cooperation from a Variety of Sources
Cooperation issues loom large in any major school-based survey today. The demands of required testing, which have increased since the enactment of NCLB, may reduce time for and willingness to participate in voluntary studies like the ECLS-K:11, so districts and schools may be increasingly less likely to cooperate. Parents are increasingly skeptical about the value of surveys and non-required tests for their children. Teachers are heavily burdened and often reluctant to spend time on non-teaching activities. The additional burden of a longitudinal survey (and the need to communicate clearly to parents and schools the expected burden of participation in a longitudinal survey) makes securing cooperation in the base year even more challenging. The base year must pave the way for concerted follow-up efforts in later rounds by collecting high quality data to help maintain cooperation and track movers.
The data collection plan approaches the school as a community. We aim to establish rapport with the whole community—principals, teachers, parents, and children. The school community must be approached with respect and sensitivity to achieve high initial response rates and maintain cooperation for future rounds of data collection. The ECLS‑K:11 field staff will be trained that all tasks—securing school and teacher cooperation, completing child assessments and parent interviews, and obtaining parent consent for the wrap-around care and education provider interview—are but different aspects of a single case in their assignment, which it is their responsibility to complete. Therefore, field staff will be responsible for conducting the direct assessments as well as the parent interviews and any required followup on the teacher questionnaires and ratings or the school questionnaire. In securing the cooperation of teachers, we will meet with each school's kindergarten teachers during the pre-assessment on-site visit. This will provide the opportunity to explain the purpose and importance of the study in a way that printed information cannot. It will enable the distribution of the teacher questionnaires directly by the field staff, and it also will provide a forum for responding to teachers' questions and concerns and assuring them of the confidentiality of the study. Incentives have proven to be effective tools in achieving high response rates, and we plan to offer monetary incentives to schools, teachers, and wrap-around care and education providers.
Secure State, District, and School Cooperation
A proactive and intensive approach will be used to secure the cooperation of the states, districts, dioceses to which sampled Catholic schools belong, and schools. The process by which cooperation is sought will be customized based on conditions in the local school systems. For example, for many states and districts an informational package may be sufficient as the first state-level step, but in some states (e.g., Florida) and districts (e.g., Detroit Public Schools, which requires a research proposal submitted a year ahead of the data collection), explicit consent to participate is required from the state by most districts and schools. A national database of district cooperation requirements will be used to develop a highly efficient, tailored approach to gaining cooperation from states and districts. The ultimate goal of the recruiting activity across all levels (state, district, school) is to make sure that the ECLS‑K:11 is on each school’s calendar before the beginning of the 2010‑11 school year. Therefore, the state and district enrollment process will begin as early as possible in fall 2009 and school recruitment in January 2010, 9 months before the kindergarten fall data collection.
The study’s procedures will be flexible enough to address the concerns and needs of states, districts/dioceses, as well as schools, to the greatest extent possible without compromising the systematic procedures that are essential to high-quality data collection. Securing the cooperation of states and districts/dioceses in the ECLS‑K:11 will be handled by a small group of senior field supervisors and in-house staff with extensive experience in this capacity, with field supervisors having responsibility for school enrollment.
Senior field staff with district contact experience will conduct the district enrollment for districts requiring more than notification, as follows.
District and diocesan packets (described further below) including a personalized letter will be shipped via FedEx (with a bright label stating “Important: ECLS‑K:11 Material Inside”) to the senior district/diocesan education official.
A telephone contact will be made to confirm receipt of the packet and identify the contact person. If the packet has not been received at the time of the first call, a callback appointment will be made. If a remail of the packet is necessary, it will also be sent by an express service.
The field staff member will complete a checklist concerning the district/diocesan approval procedure and time frame.
A district coordinator will be identified and will receive all the pertinent material about the ECLS‑K:11.
The district-wide requirements for obtaining parental consent for children’s participation in the study will be ascertained, or confirmed if already known. For the ECLS-K, some schools requested explicit consent from parents even though the district policy was one of implied consent. Thus, ascertaining the policy at the district level and having district representatives assure schools that the ECLS‑K:11 will be complying with district policy may help to limit the instances in which a more stringent and costly approach must be used for some schools.
Any additional information that is requested (extra packets, study instruments, or additional study details) will be sent promptly. Any requests for material not already approved for distribution will be referred to the NCES study director immediately.
Any refusals will be referred to the data collection task leader, along with the case record and reasons for the refusal. In some cases, the contractor project director will discuss appropriate strategies to try and convert the refusal to a completed case with the NCES study director and will make recommendations for actions to be taken.
For some districts, the data collection task leader may recommend an on-site visit to prevent or convert a refusal. Such cases will be brought to the attention of the NCES study director and no on-site visits for this purpose will be made without the approval of NCES.
Contact with the districts will continue until participation is secured or a final refusal is received. The timing of contacts will be customized to the specific case based on their approval procedures and information obtained during the contacts.
Information Packets For States, Districts, Dioceses, and Schools. An information packet will be developed for the ECLS-K:11 to convey the study’s legitimacy, importance, and support to various school entities. The information in the packet will be presented in a way that is clear and specific, yet concise, and will emphasize that the study team will work closely with schools to accomplish the study with the least burden and disruption possible. The cover letter transmitting the packet will be concise, engaging, and convincing. It will come from a recognized public officer (e.g., the Commissioner of NCES) and will list the study endorsers in the letter head, so that the recipient will immediately understand the importance and the widespread support of the study.
Information packets will be customized for each intended audience; for example, materials for state agency personnel will address the procedures that will be used for contacting the school districts and schools. Materials for the districts will address the selection of and procedures for contacting individual schools within their districts. The information packet to all levels of the school system also will include letters of endorsement from appropriate organizations; these letters will be customized for public and private schools (for example, lists of endorsers will be customized by those most pertinent to a public or private school). Endorsements will be solicited from the following organizations (Note: * indicates that organization endorsed the ECLS-K):
|
|
Summary sheets will be incorporated in higher-level mailings. (See Appendix F for examples of respondent materials from the ECLS-K.)
Secure Participation of Schools. Field staff with experience enrolling schools in the ECLS‑K and other NCES studies will be assigned to this activity and trained in January 2010. Each school recruiter will be assigned a specific set of schools and will receive a laptop containing field management system files for the schools.
We will obtain district approval to contact the schools before doing so. Some school districts or diocesan contact persons will wish to contact the schools before we do so. We will adhere to any district requirements that district personnel contact the schools first, that they disseminate the school information packets, or that they receive copies of all communications with schools. We will document and adhere to the procedures to be used within each school district.
For the ECLS‑K:11, it may be necessary for school recruiters to schedule meetings with the district contact person and school representatives to discuss the study and its procedures. Such meetings will be scheduled after the information packets have been received by each principal, in order to maximize the usefulness of the meeting and limit the amount of time spent presenting material that is available in the information packets. We do not plan to proactively offer such meetings to all districts, as that could result in a substantial cost burden to the project. However, we will be prepared to hold such meetings when they are necessary to gain the cooperation of schools and districts.
The school information packets will be sent to the principal or headmaster of each sampled school by 2-day FedEx. Several business days after the mailing, school contacts will begin. The school contact procedure will be similar to the district contact procedure described earlier. The first step will be confirming the receipt of the packet or mailing a new package if the first one was not received. The ECLS-K:11 study staff member who contacts the school will complete a checklist concerning additional information regarding the school needs, the school cooperation decision, the school coordinator for data collection, the parent notification/consent procedures that must be followed (emphasizing district policy), and the dates of visits for child sampling and assessments. The data collection task leader will review cases that initially refuse to participate and decide on appropriate strategies to try and convert the refusal, consulting the project director as needed, and possibly enlisting the assistance of district or diocesan staff to encourage their participation. The school coordinator for data collection will be provided with all the necessary materials before the pre-assessment visit. We propose to offer the school coordinators $25 by check for their assistance.
Secure Cooperation of the Children’s Parents
Reaching parents and obtaining their consent must be accomplished quickly to support the data collection goals within the time frame of the study. After the sampling of children within schools, the field supervisor will prepare an information/consent packet for the parent/guardian of each child. The packet will contain a letter to the parents, a study summary sheet, and frequently asked questions. The packet will introduce the ECLS‑K:11, explain its purpose and importance, tell parents what is involved in participation, and specify the consent procedure that is being used by their school.
Before compiling the parent packets, we will ascertain the policy of the district regarding parental notification and consent, assuring schools that we are following district procedure. Parent consent procedures will be discussed in the district and school recruitment process, as noted above. We know from experience on the ECLS-K that schools generally require one of two types of consent: implicit or explicit. Both types of consent require that parents be notified that their children have been selected for the study. With implicit consent, the school does not require verbal or written consent for the child to participate. Parents are asked only to notify the appropriate person if they do not want their child participating in the study. If there is no effort to refuse participation, consent is assumed. With explicit consent, children may participate only if their parents provide written or oral consent for their children to do so.
For the ECLS-K:11, all parents will be sent a consent form in the advance mailing with information about the study. Depending on school requirements, one of two consent forms will accompany the parent notification letter and advance materials. Explicit consent forms will be sent if the school requires that the parent sign a consent form and return it to the school before their child can participate in ECLS-K:11. Parents will be asked to return the form promptly with their agreement to participate. An implicit consent form will be sent if the school does not require a signed consent form for participation; parents will be asked to return the form only if they object to their child participating in the study. Compared to implicit consent, the use of the explicit consent procedure requires considerably more follow-up than implicit consent, which is costly and can be problematic when the data collection period is brief. For this reason, we will encourage the use of implicit consent whenever possible.
Obtaining a high response rate in the ECLS‑K:11 requires that active parent follow-up be conducted in schools using an explicit consent procedure. The field supervisor will ascertain during the pre-assessment visit whether the school coordinator is available and willing to undertake this follow-up. However, the field supervisor may need to conduct this activity in order to obtain responses from parents before the first day of assessments. The general study approach assumes that responsibility for parent follow-up will be placed with data collection staff, who are trained for this procedure, rather than with the school coordinator, who may have limited time and may be reluctant to prompt parents to reply or to attempt to persuade a reluctant or initially refusing parent to take part in the ECLS-K:11. Data collection staff will call each parent to confirm receipt of the study materials and consent for the child to participate. Such confirmation calls will be made regardless of the type of consent schools require, because we want to be sure parents understand their child is participating in ECLS-K:11 and that any questions or concerns they have about the study can be addressed. For parents with telephones, follow-up will be conducted initially by telephone; bilingual interviewers will make calls to language minority parents. In-person visits will be made for those parents without telephones.
Parent Contact Information. When possible, the field supervisor will obtain contact information, from the schools, for the parents of sampled kindergartners at sampling time, enter the information into the FMS, and transmit it for inclusion in the study respondent database. We understand that some schools may be reluctant to provide this information before parents have agreed to participate in the study. We will work with schools to identify any steps field staff can take to help the school obtain consent from parents and will assure that study procedures accommodate schools who want to provide contact information only after explicit consent is received from parents.
In cases in which explicit consent is required, we will include a space on the consent form for the parent to enter the name of the parent who is most knowledgeable about the child's education, a current address, and home and work telephone numbers (if any). Collecting this information will be beneficial for future rounds of data collection because it will help in locating those families that move without providing updated information to the school.
Monitoring Receipt of Parent Consent Forms. Regardless of whether the school chooses to handle the receipt of consent forms or gives study staff the responsibility for this task, the field supervisor will maintain close contact with the school regarding receipt of parent consent forms. If the school coordinator has chosen to have the ECLS‑K:11 data collection staff conduct follow-up on the parent consent forms, the field supervisor and assessors will check frequently with the school to ascertain whether forms have been returned to the school. The school coordinator will receive a form listing the children and parents and can use the form to check off receipt of the consent forms from parents. A folder will also be provided to store the forms.
Audio-Documentation of Parent Consent. As an additional step in obtaining parent consent, we will seek explicit verbal consent from all parents for their children to participate in the ECLS‑K:11 in an audiofile at the beginning of the fall kindergarten parent interview. This will serve as additional confirmation of consent for those parents who already provided consent, either implicit or explicit, before they were called. It also provides an additional method by which study staff can obtain explicit consent from those parents for whom it is required but who have not yet returned their consent form. Because we must obtain parent permission, whether explicit or implicit, before the child assessment takes place, this strategy should help to have the necessary consent in place at the time field staff visit schools to conduct assessments, since the fall parent interview occurs concurrently with the fall assessments. Also, having recorded parent consent makes it easier to encourage school and teacher participation in instances where students transfer from one school to the next; this issue is discussed further below.
B.3.2 Methods to Maximize Response Rates
Parent Interviews
There are four main areas that can be focused on in order maximize completion rates for the parent interviews: (1) flexibility in scheduling interviews, (2) non-English interviewing, (3) locating parents of children who transfer schools, and (4) avoiding refusals and converting initial refusals to completed interviews.
Flexibility in Scheduling Interviews. Effective calling patterns are essential for achieving high response rates on all telephone surveys. Previous experience shows that individual respondent schedules (work, classes, recreational activities, vacations, etc.) have a more negative effect on response when call attempts are limited to a short time span. A larger percentage of the numbers that are noncontacts after the first call attempt will be converted to a positive outcome if the call attempts are distributed across a longer time span. Completion rates improve when interviewers call on different days of the week and at varying times of the day and evening.
To establish initial contact with a parent of a sampled child, field staff will be trained to place two weekday, three evening, and two weekend calls over a 2-week period. These calls will be made in a nonsequential set of targeted time periods called “time slices.” The time slices and required number of calls are as follows:
Required Number of Calls
Weekday 10 a.m. to 3 p.m. 1
Weekday 3 p.m. to 6 p.m. 1
Weekday 6 p.m. to 9 p.m. 1
Weekday 6 p.m. to 7:30 p.m. 1
Weekday 7:30 p.m. to 9 p.m. 1
Saturday or Sunday, 10 a.m. to 8 p.m. on separate weekends 2
If, after seven call attempts, no contact has been made with the parent, the field staff will be instructed to review the case with the field supervisor for additional instruction on how to proceed. The supervisor may instruct the field staff to do one or more of the following: (1) send a letter to the parent; (2) contact the school coordinator to see if the school can help or offer any insight into contacting the parent; (3) contact one of the other contacts listed for the parent, if any; or (4) contact the nonresident parent, if applicable.
Once contact is established, up to seven additional calls will be made to complete the parent interview. If the interview is not completed after these seven additional calls and the respondent has not explicitly refused, the field staff may be instructed to attempt an in-person interview. During the last few weeks of data collection, noncontact and uncompleted cases will be visited in-person as appropriate to improve response rates.
Non-English Interviewing. In the base year of the ECLS-K, approximately 7 percent of the parent interviews were conducted in a language other than English. With the growth of immigration of non-English speakers, the percentage of parents who prefer to participate while speaking a language other than English is expected to be larger in the ECLS-K:11. To achieve high response rates, it is important that study procedures work to include these parents to the greatest extent possible. As described in the data collection procedures section, we will hire and train field staff who are bilingual in Spanish and English to conduct fully translated parent interviews in Spanish and use professional interpreters, as available, for interviews in non-English, non-Spanish languages.
Locating Parents of Transfer Children. Locating parents of transfer children is critical for maintaining high completion rates for parent interviews overall. It is expected that some children will transfer schools between the fall and spring data collections. A tracking system database with household contact and school information will be developed at the beginning of the study and sample tracking activities such as the following will be conducted to locate children who transfer schools: (1) the entire household (parent) address database will be submitted to postmasters for address corrections (ACR); (2) a respondent mailing will be sent to parents asking them to report any changes in sampled children’s schools and/or home addresses; and (3) schools will be asked to provide updated contact information, if available. Household and school updates resulting from these activities will be recorded in the ECLS-K:11 tracking system database. While this OMB package requests approval for the fall and spring kindergarten collections, long-term study plans are to follow the sample children through fifth grade. Development of this tracking database is an important activity for the maintenance of the sample over time; it will be updated with new information over the lifetime of the study.
Refusal Avoidance and Conversion Procedures. Achieving an acceptable parent response rate will require active and effective refusal conversion efforts. This activity must begin as soon as the parent consent forms start arriving at the school. A key factor in converting refusals is the ability of the field supervisors and assessors to clearly and confidently convey the purpose and importance of the study and the benefits that will be derived from it. This will be a focus of the field staff training. The training materials for averting refusals include information about becoming thoroughly familiar with the study, including answers to frequently asked questions (FAQs) and respondent objections, drafting responses in the interviewer's own words to FAQs, practicing saying these responses, and diagnosing respondent objections and quickly responding with a response tailored to the objection. The training includes self-analysis by recording responses and listening to them, preparing answers for different situations, using the voice effectively, and role-plays between trainer and interviewer and between interviewer and interviewer. Averting refusal training will focus specifically on addressing reasons for refusals on the parent interview component of the ECLS-K:11 study.
During the parent interview data collection period, supervisors and field managers will review initial refusals with the field staff, putting a particular emphasis on reviewing the interviewer record of calls, which will be available to supervisory staff on a weekly basis. If a refusal occurs, the interviewer will be instructed to record key demographic information about the refusing respondent (e.g., sex, approximate age) and the respondent’s reason(s) (if given) for refusing to participate. This information will be evaluated by the field supervisor to determine the best strategy for converting refusals. Cases identified for refusal conversion will be assigned to a select group of field staff identified as possessing the necessary skills to act as refusal converters. Field managers will hold telephone conferences with the identified field staff to review the refusal conversion procedures and discuss strategies for converting refusals.
Child Assessments
There are two main areas that can be focused on in order to maximize completion rates for the child assessments: (1) conducting assessments with children who are absent on scheduled assessment days and (2) locating transfer children.
Absent Children. It is expected that some children will be absent from school during the time that assessments are scheduled at any given school. Days will be set aside throughout the field period in which some field staff have no assessments scheduled, so that make-up assessments can be more easily conducted. A make-up assessment will be conducted for any child who can be assessed during the field period. If an in-school assessment cannot be scheduled, field supervisors will contact parents to make arrangements for in-home assessments for absent children, if possible.
Locating Transfer Children. As was the case with the parent interview, locating transfer children, and particularly the new school in which they are enrolled, is critical in maintaining high completion rates for child assessments overall. The activities described above for the parent interview with respect to tracking the sample serve to help maintain high response rates for the child assessments as well, since parents must be located to gain consent for the child assessments.
There is an additional consideration to locating children who transfer schools, which is the need to contact their new schools and teachers and encourage them to participate as well, allowing the children to be assessed in the school. This issue is discussed further in the next section.
School and Teacher Instruments
There are three main areas that can be focused on in order to maximize completion rates for the school and teacher hard-copy instruments: (1) early distribution of instruments to schools and teachers, (2) effectively communicating the importance of school administrator and teacher participation to school personnel in schools to which ECLS-K:11 children have transferred between the fall and spring of the kindergarten data collection, and (3) the efforts made by supervisory staff to avoid refusals and to convert initial refusals to cooperating respondents.
Early Distribution of Instruments. Feedback from school administrators and teachers in the ECLS-K indicated that there would be increased participation if they had had more time to complete the hard-copy instruments. The preassessment visit in the fall of the kindergarten school year will identify most of the sampled children’s regular classroom and special education teachers, as well as the school administrators. During the preassessment visit, field supervisors will prepare regular classroom teacher questionnaires and distribute them, along with teacher incentive checks, and ask teachers to complete them before assessment day. For the spring data collection, the plan is to send these school and teacher questionnaires, along with their incentive checks, in February of the school year. This schedule will allow 2 months of additional time for these respondents to complete and return the instruments to the school coordinator for field staff to collect on assessment day.
Effective Communication with New Schools to Which Sample Children Transfer. The participation of newly identified school administrators and teachers can be increased by effectively communicating information about the ECLS‑K:11, including the goals of the study, what the study measures, the various study components, why it is important that schools and teachers participate, the study activities to date, the plans for the future, and perhaps some results from the ECLS-K. Effective respondent materials, as well as telephone contact by supervisors who are trained to convey this information efficiently and completely, will help maximize the participation of schools to which sample children transfer. In addition, parental consent will be recorded for all children (as mentioned earlier), so that a record of consent will be available for new schools, making it easier to get new schools and teachers to participate.
Refusal Avoidance and Conversion Procedures. Much like the averting refusal training described above for ECLS‑K:11 parent interviews, the training session for hard-copy refusal aversion will include analyzing the reasons for refusal, preparing answers for different situations, using the voice effectively, and role-play situations between trainers and trainees and pairs of trainees. Thus, this averting refusal training will focus specifically on addressing reasons for refusals on the hard-copy instrument components of the ECLS‑K:11 study.
Special Considerations in Obtaining Cooperation. District and school personnel have stated that they face increasing demands upon their schools for a variety of noninstructional activities, including requirements for state and district assessment. Sensitivity to these concerns is essential to gaining cooperation for the ECLS‑K:11, and it must be made clear to school system personnel at all levels that the ECLS‑K:11 staff is more than willing to work with them to facilitate their participation with the least burden and disruption possible.
We plan to provide incentives to schools in the form of donations to unrestricted school funds. Most schools have some type of fund that can be used for school needs or activities. Sometimes such funds are managed by the school itself or by the PTA. Making a donation to such a fund provides the school with the means to use the funds in a way that best meets its needs. An average honorarium of $200 per school is recommended.
Wrap-around Early Care and Education Provider Interviews
As discussed previously, the wrap-around care and education provider instrument (WECEP) was first fielded as part of the ECLS-B. Experiences on that study show that it is important that care providers be aware that they have parents' permission to provide information about the study children in their care. Prior to the WECEP interview, a packet of information about the study will be sent to the care provider. This packet will contain a consent form signed by the parent giving ECLS-K:11 staff permission to contact the care provider. Child care providers will be sent a $35 incentive in appreciation for their completing the WECEP in order to build response rates. Providing child care providers with an incentive to participate will alleviate the need to extend the field period in order to convert refusals and, thereby, help avoid delays in data delivery. The incentive will be included in the pre-interview information packet.
As with the parent interview, scheduling for the care provider interview will be as flexible as possible to accommodate the care providers' schedules. Interviews will be conducted in Spanish by bilingual interviewers, and translators will be used for other languages in order to include care providers who prefer to participate in languages other than English or Spanish.
Statistical Approaches to Nonresponse
High response rates in the initial kindergarten data collection are very important because any nonresponse biases incurred at that stage are likely to persist in the longitudinal estimates even if attrition is minimized in the subsequent waves. We will analyze nonresponse in the initial school cooperation rates and focus efforts on obtaining rates that are as equal as possible across the major subgroups. For example, non-Catholic private schools have the lowest response rates in surveys of this type and thus additional efforts are required to improve their response rates. This approach contrasts with blindly trying to increase overall response rates by spending resources on those groups that already have the highest response rates, which might raise the overall response rate more but does not overcome bias that would be present in the estimates due to lower response rates for certain subgroups.
We also will subsample movers using a scheme that is similar to that used in ECLS-K to reduce nonresponse bias. The subsampling itself does not reduce nonresponse bias, but the subsampling does enable the same fixed resources to be allocated to a lower number of children so that higher response rates for subgroups can be achieved. The most expensive children to survey are the movers because collecting data on movers requires additional efforts to get permission from the all the entities from which consent is required (e.g., new districts and school administrators). Also, cost per completed case is increased when there are fewer children per school, and it is often the case that when children change schools, they are the only study child in the school to which they moved.
The form of subsampling that we propose is to include all movers in the rare groups of policy interest in the subsample to protect the power for analyzing these children and to subsample other movers. For example, for the ECLS-K Asians, Native Hawaiians, and other Pacific Islanders, as well as language minority children, were retained at much higher rates in the subsamples. Other children who move will be subsampled at a rate of about 50 percent. This subsampling rate reduces data collection costs for the movers and ensures that the variances of the estimates are not greatly inflated by using much lower subsampling rates.
B.4 Test of Procedures and Methods
The ECLS-K Class of 1998-99 study has informed the approach for the ECLS-K:11. By design, the ECLS-K:11 data collection instruments are in large part a collection of items used in the ECLS-K, to allow comparisons between the two cohorts of kindergartners. However, because there will be some changes to the ECLS-K instrumentation and study procedures, and most particularly in the child assessment, a field test for the ECLS-K:11 is included in this request for OMB approval. As mentioned in section A.1.d, a majority of items in the ECLS-K:11 assessment will be the same as the earlier ECLS-K base year assessment in order to enable researchers to conduct cross-cohort analyses. However, experiences with the ECLS-B, which used an assessment battery very similar to that of the ECLS-K, demonstrated that the assessments no longer had enough items to accurately the measure of skills of children functioning at higher levels. Additionally, there are new domains, in particular executive functioning and science, that will be included in the ECLS-K:11 direct cognitive assessments for which items must be tested. The field test will include a kindergarten through second grade assessment instrument,9 a test of procedures for hearing and vision screening,10 and a test of the teacher rating scales. This section presents the field test plan.
B.4.1 Field Test Sample
In the fall of 2009, NCES will conduct a field test of the ECLS‑K:11 base year data collection. By the time of the field test, we will have already selected the sample of schools to include in the full study in the manner described in section B.1. We propose to begin the process of securing cooperation of states, districts, dioceses, and schools for the field test as soon as possible after the selection of the full study sample to avoid the burden of overlap; that is, schools sampled for the full collection will not be selected as field test schools.
Ten schools in five geographic sites will be selected for the field test (i.e., 50 schools in total). One of these geographic sites will be located in Washington, DC or the surrounding area to allow for NCES staff observation of the collection methods. Other sites will be distributed throughout the country. Within each geographic area, the field test sample will represent different urbanicity levels and will include public and private (both secular and parochial) schools. Special attention will be given to schools with a high proportion of Black, Hispanic, and Asian, Native Hawaiian, and other Pacific Islander children so that the sample used to evaluate the functioning of the assessment will be diverse.
In each field test school, 18 children will be selected from each of the following grades: kindergarten, first, and second. As mentioned in section A.1.d, a particular purpose of the field test is to gather information for developing test parameters for the direct cognitive assessments. A valid assessment must contain at least some items at a more difficult level for the highest functioning children. For a K-2 assessment, this means including some items at a third grade level. For this reason, 9 children will be selected from grade 3 within each school. In addition, 24 Spanish-speaking kindergarteners will be selected to obtain a measure of English reading scores and Spanish basic skills. This field test sample of 900 English-speaking kindergartners, 1,200 Spanish-speaking kindergarteners, 900 grade 1 English-speaking children, 900 grade 2 English-speaking children, and 450 grade 3 English-speaking children will be used to develop test parameters for each grade (a total of 4,350 children). In each field test school, two kindergarten, two first grade, and two second grade teachers (a total of 300 across all field test schools) will also be recruited to complete the teacher instruments. NCES will not begin recruiting schools for the field test sample until the design is cleared by OMB.
B.4.2 Field Test Data Collection Procedures
Once field test schools are identified, contact will be made with the schools to identify field test school coordinators to assist in coordinating the field test data collection efforts (similar to the role of the school coordinator for the national data collection, described previously). Data collection for the field test will begin in August 2009 and will be completed in October 2009. Training and data collection protocols for the child assessment and teacher questionnaires (including all response rate maximization procedures) developed for the full-scale survey will be used during the conduct of the field test.
Unlike the direct cognitive assessments in the full-scale collections, the field test direct cognitive assessments will be conducted using paper and pencil. Hard-copy field test booklets for the child assessment will be administered individually and assessors will record children’s responses in a paper-and-pencil format using a separate score sheet. In order to test many different items without overburdening the children, the field test assessment booklets will be developed from items divided into four reading forms, two math forms, and two science forms. These forms will be spiraled such that each child in the field test will receive one of four versions of the reading and one of two versions of either the math or the science (e.g., reading 1 and math 1, reading 2 and math 2, reading 3 and science 1, reading 4 and science 2, etc.). Spanish-speaking kindergarteners will receive an English reading form and a Spanish basic skills form. Assessment materials will be prepared for each school to assure that the appropriate number of booklets of each version is administered in each grade. Field staff will be trained to record observations about children’s behaviors and response to the assessment and will also keep a more general diary of field test experiences. These field test observations will be used to prepare the field test report.
As part of the field test, all participating kindergarten through third grade children who are English speakers in the 50 field test schools will take part in a hearing and vision screening. The screenings will be conducted by a group of field staff trained specifically on administering the hearing and vision assessments; these staff will not be conducting other portions of the child assessments. A health screening/pediatric specialist will train the staff for the hearing and vision screening; the training will cover not only the use of the specialized screening equipment and the assessments, but also general issues relating to research with child participants. The hearing assessment will consist of a few questions asked of the child to determine if the child has any conditions that may be affecting the child’s hearing on the day of the assessment (e.g., the child has a cold), a visual inspection of the ear, and a hearing test. An otoscope will be used to visually inspect the ears, and children’s hearing will be measured using a tympanometer/audiometer. The vision screening will assess optimal characteristics of the eyes (e.g., does the child need glasses, are there differences between the eyes, does the child have a lazy eye) using an autorefractor. Visual acuity also will be tested using an electronic visual acuity (EVA) tester.
Additional data in the field test will be collected from two teachers in each of grades kindergarten through 2 in each field test school. Kindergarten, grade 1, and grade 2 teachers will be asked to complete the teacher questionnaires and child rating forms. Each teacher will be given $7 for each child rating form he or she completes. Field test school coordinators will receive a $25 honorarium for their assistance during the field test.
B.5 Individuals Responsible for Study Design and Performance
The following individuals are responsible for the study design and the collection and analysis of the data on ECLS-K:11.
Gail Mulligan, NCES (202) 502-7491
Chris Chapman, NCES (202) 502-7414
Karen Tourangeau, Westat (301) 251-8265
Christine Nord, Westat (301) 294-4463
Thanh Lê, Westat (301) 610-5105
1 The assumptions underlying the calculation of sample size noted here are: a two-tailed test of differences with significance level alpha of 0.05 and power beta of at least 80 percent; estimating proportions of 30 and 36 percent (i.e., 20 percent relative change); and a correlation between assessment scores from different waves of 0.75. This assumed correlation of assessments comes from experiences in the ECLS-K. Specifically, looking at difference estimates computed between grade 1 and grade 3, and between grade 3 and grade 5 of the ECLS-K, the estimated correlations in assessments between consecutive waves were found to be very high (between 0.72 and 0.98), for an average of 0.75.
2 Since the target population for the study is kindergarteners, and age 5 is the modal age for kindergarteners, number of 5-year-olds is used as the measure of size for each PSU.
3 For the purpose of PSU selection, these groups will be included together in the stratification scheme.
4 Children in these racial/ethnic groups will be oversampled together as one group.
5The ECLS-K study included the collection of data from a nationally representative sample of kindergarten teachers, allowing for the generation of estimates at the teacher level. The ECLS-K:11 will include only a nationally representative sample of children.
6During the time we are trying to secure parent cooperation, with the school's permission we will look for ways to inform the school community about the study through the school’s communication systems and tools, for example by posting news about the survey on the school’s web site; submitting articles about the survey to the school newsletter; and discussing the survey at a PTA meeting or on Back-to-School Night. To make it easier for field supervisors to initiate this communication, sample letters to parents, articles for newsletters, and PowerPoint presentations that can be customized will be included in an ECLS‑K:11 Best Practices Guide that will be given to field supervisors.
7The feasibility of conducting the hearing screening efficiently will be determined in the field test. If the screening takes too long or is too difficult for field interviewers to conduct accurately in the field, these screenings will be dropped from the full collection.
8 The materials in appendix F are those used in the ECLS-K and are provided as examples of materials that will be used for the ECLS-K:11 . Some have been updated for the ECLS-K:11 while others have not. Those that have not been updated yet will be updated to accurately reflect the study year and procedures and will be provided in the revised OMB package, to be submitted at a later date, that describes the results of the field test and any changes that are made based on those results.
9Another field test will be conducted later, under a different contract, that will cover third through fifth grade.
10At this time, funds are available to conduct the vision screening only during the field test. Results of both the hearing and vision field test screenings, as well as basic demographic information about the children collected during the field test, will be provided to the federal agencies sponsoring these screenings (the National Center for Deafness and Other Communication Disorders and the National Eye Institute, respectively) for their own analysis of the characteristics of children in the field test sample who exhibit hearing and vision problems.
|
|
|
Early
Childhood Longitudinal Study |
B- |
|
| File Type | application/msword |
| File Title | Early Childhood Longitudinal Study Kindergarten Class of 2010-2011 |
| Last Modified By | #Administrator |
| File Modified | 2009-03-20 |
| File Created | 2009-03-20 |
© 2024 OMB.report | Privacy Policy