Volume I:
Request for Clearance for the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Spring Second-Grade Cognitive Interviews
OMB# 1850-0803 v.64
The Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) is a survey that focuses on children’s early school experiences beginning with kindergarten and continuing through the fifth grade. It includes the collection of data from parents, teachers, school administrators, and nonparental care providers, as well as direct child assessments. Like its sister study, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K),1 the ECLS-K:2011 is exceptionally broad in its scope and coverage of child development, early learning, and school progress, drawing together information from multiple sources to provide rich data about the population of children who were kindergartners in the 2010-11 school year. As with the ECLS-K, the ECLS-K:2011 is sponsored by the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education. Fall and spring data collections in the study children’s kindergarten and first-grade years were conducted for NCES by Westat. The Educational Testing Service (ETS) serves as the subcontractor developing the child assessments. Clearances for studying the ECLS-K:2011 cohort have been granted under OMB clearance number 1850-0750 for the fall 2009 field test data collection, the fall 2010 and spring 2011 kindergarten national data collections, the fall 2011 and spring 2012 first-grade national data collections, and the fall 2012 second-grade national data collection. Additional clearances have been received for the spring first-grade cognitive interviews (OMB No. 1850-0803 v. 43), cognitive interviews to test Response to Intervention (RtI) questions (OMB No. 1850-0803 v. 51), and a field test of a computerized version of the Dimensional Change Card Sort (OMB No. 1850-0803 v. 60).
The ECLS-K:2011 is the third in an important series of longitudinal studies of young children sponsored by the U.S. Department of Education that examine child development, school readiness, and early school experiences. It shares many of the same goals as its predecessors, the ECLS-K and the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), but also advances research possibilities by providing updated information and addressing recent changes in education policy.
Like its predecessors, the ECLS-K:2011 will provide a rich and comprehensive source of information on children’s early learning and development, transitions into kindergarten and beyond, and progress through school for a new cohort of children.
The ECLS-K:2011 will provide data relevant to emerging policy-related domains not measured fully in previous studies.
Coming more than a decade after the inception of the ECLS-K, the ECLS-K:2011 will allow cross-cohort comparisons of two nationally representative kindergarten classes experiencing different policy, educational, and demographic environments.
This is a request for clearance to conduct cognitive interviews to inform the instruments planned for the ECLS-K:2011 spring second-grade data collection. Specifically, the goals of these cognitive interviews are 1) to identify and correct problems with ambiguity in or misunderstanding of new and revised items for the spring second-grade teacher questionnaire and parent interview and 2) to finalize the parent interview through timing tests.
NCES has contracted Westat to conduct the spring second-grade data collection for the ECLS-K:2011, including these cognitive interviews activities.
Cognitive interviews (described below) will be conducted to evaluate new and revised items in the ECLS-K:2011 spring second-grade teacher questionnaires (appendix A). Timed telephone interviews will be conducted to determine the timing of the different sections of the draft second-grade parent interview (appendix B) and to evaluate a small number of new and revised items. Appendix C includes the cognitive interview protocols for teachers and parents. For easier review of the specific cognitive interview questions and probes that are embedded within the full parent interview, the protocol document for the parent interview only includes those questions for which parent understanding is being tested, along with the follow-up questions and probes related to those questions. It does not include the full interview.
Most items from the teacher questionnaire have been fielded over multiple rounds of the ECLS-K and ECLS-K:2011 and, therefore, will not be specifically tested in these cognitive interviews. The new items being tested in the cognitive interviews have been included in the questionnaire to address issues that have recently been identified as important areas for research. Other items are being included in the cognitive interviews because they have been revised as they were found to have relatively low item response rates in the kindergarten round of ECLS-K:2011 data collections or used wording that was not as clear as it could be; revisions are intended to increase the rate at which respondents answer these items and improve the quality of data gathered from them. In general, participating teachers will be asked to discuss their interpretation of selected items, to discuss the process they would use to obtain an accurate response, and to provide suggestions for item and response category wording for any unclear items or items that are difficult to answer. In addition, because there was no second-grade data collection round in ECLS-K, teacher respondents will be asked to briefly review the full questionnaire so that they understand the general context in which the tested questions are asked and to assess the appropriateness of the full instrument to collect data from second-grade teachers. At the end of the cognitive interview, teacher respondents will also be asked if they have any other comments on the items that were not discussed or the questionnaire in general.
Parents will be asked to complete the entire parent interview mainly to obtain timing information for the different sections in the interview. As with the teacher questionnaire, most items from the parent interview have been fielded over multiple rounds of the ECLS-K and ECLS-K:2011 and, therefore, will not be specifically tested in these cognitive interviews. Questions probing on a select set of new and revised questions (questions about children’s use of technology at home, accessing reading materials online, and bullying at school, as well as parents’ opinions about asking questions directly of second graders) have been incorporated within the parent interview instrument and will be asked as part of the regular parent interview.
The request to conduct the national spring second-grade data collection will be submitted at a later date. The ECLS-K:2011 collections are authorized under 20 US Code section 9543, which states that the purpose of NCES is “to collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations.”
Cognitive Interviews with Teachers. The cognitive interviews will be conducted as one-on-one interviews between a responding teacher and an experienced qualitative interviewer. Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer a series of questions about the items he or she just answered. This approach includes asking probing questions, as necessary, to clarify points that are raised in the think-aloud comments.
Cognitive interviews will be conducted with 25 second-grade teachers. To gather information about items related to instructional programs for English language learners (ELLs), we will attempt to recruit teachers from at least seven schools with a student population that includes at least 15-20 percent English language learner students. This is a purposive sample, however we will attempt to recruit teachers from among elementary schools with various characteristics, e.g., from public (including charter) and private schools and rural and urban schools located throughout the country.
We will recruit teachers from the 41 schools that participated in the 2011 ECLS-K:2011 first-grade cognitive interviews and the Response to Intervention (RtI) cognitive interviews, as well as from the 59 schools that participated in the 2009 ECLS-K:2011 field test. These schools are being selected due to their willingness to participate in past field test work and the fact that Westat has developed a good working relationship with them. The schools that participated in the 2011 field tests are in 17 states in different regions of the country (i.e., northeast, south, central, and west) and include 4 private schools and 1 charter school, and 8 schools that were “in need of improvement” in the 2009-2010 school year. The schools that participated in the 2009 field test are in 9 states in different regions of the country (i.e., northeast, southeast, central, and west), and include 9 private schools, 3 charter schools, and 12 schools that were “in need of improvement” in the 2009-10 school year.
During recruitment, we will prioritize schools that have been identified as having the desired proportion of ELL students. We will contact schools, beginning with the schools that have this characteristic, until we reach the target number of teachers. To recruit respondents through these schools, we will contact the person who was designated as the school coordinator for the previous field test. The school coordinator will be asked to identify one second-grade regular classroom teacher and provide the contact information for the teacher. If the teacher declines participation we will ask the school coordinator to identify an additional second-grade teacher. In the event that the school coordinator from the 2009 field test is not available, we will contact the school principal to provide contact information for a second-grade teacher.
If we are unable to recruit 25 teachers from schools that participated in the previous ECLS-K:2011 field test or cognitive interviews, we will also recruit some second-grade teachers through the network of contacts that Westat has with teachers, as well as through advertisements placed on listservs that teachers may subscribe to (e.g., teacher association listservs, university education department listservs). Recruitment through this method will continue until the sample size target is reached.
After speaking with study staff and agreeing to participate in a cognitive interview, teacher respondents will be mailed a letter explaining the purpose of the cognitive interviews (appendix D) and a copy of the draft second-grade teacher questionnaire to review prior to the cognitive interview (appendix A). The full teacher questionnaire covers a wide range of topics including classroom and teacher characteristics, class organization and resources, instructional activities and curricular focus, parental involvement, evaluation and grading practices, school and staff activities, school climate, school environment, and teacher background. A cover letter will instruct respondents to take 5 minutes to briefly review the entire questionnaire prior to the cognitive interview. While the interviews will focus only on selected items, reviewing the entire questionnaire will provide some additional context for the respondent. Interviewers for the cognitive interviews will follow a prewritten protocol (appendix C) but will be free to deviate from the protocol in order to address specific issues or anomalies in the respondents’ verbal reports. Question wording may change during the testing period in response to suggestions from teachers who are interviewed earlier in the testing period.
In the cognitive interviews with teachers, focal items come from the sections on classroom characteristics, class organization and resources, and instructional activities and curricular focus. Cognitive interviews with teachers are expected to last about an hour on average. Thus, the total burden per cognitive interview, including 5 minutes to review the questionnaire, is anticipated to be about 65 minutes for teachers.
Based on school response rates from the spring 2011 first-grade cognitive interviews of the ECLS-K:2011, we anticipate a response rate of about 66 percent.2 Thus we anticipate contacting about 38 teachers to obtain 25 responding teachers. During recruitment, a recruiter will explain the purpose and anticipated length of the cognitive interview and ask the respondent if he or she would like to participate. The recruitment burden is anticipated to be about 5 minutes on average (see appendix E for the teacher recruitment script).
Interviews with Parents. The interviews with parents will be conducted as telephone interviews by experienced interviewers. Interviewers will make an appointment with participating parents and conduct the interview with a hard-copy paper version of the parent interview, following the proper skips in the interviewer instructions (appendix B). Timings will be recorded for each section and for the interview overall. In addition, parents will be asked for feedback on a select set of items in the interview that were revised or newly developed based on input from experts that was received during expert review panel meetings. In addition, experts strongly recommended asking questions directly of the children themselves, starting in second grade, to obtain information about self-concept and efficacy. Parents will be asked for their reactions to asking such questions of second-graders at the end of the interview. The interviews are expected to take about an hour per parent.
Interviews will be conducted with 18 parents of second-grade children. Parents participating in the interviews will be recruited through the network of contacts that Westat has with parents, as well as through advertisements (e.g., Craig’s list).
Based on parent response rates from cognitive interviews conducted for the National Household Education Survey (NHES), we anticipate that about 90 percent of parents who express interest in participating in the parent interview tests will end up participating. Thus we anticipate contacting about 20 parents to obtain 18 responding parents. During recruitment, a recruiter will explain the purpose and anticipated length of the timing tests and ask the respondent if he or she would like to participate (see appendix E for the parent recruitment script).
In preparation for the first- and second-grade data collections, one Technical Review Panel (TRP) and three Content Review Panels (CRP) were held in March 2011 to review and comment on issues related to the instruments. The members of these panels included experts in child development, teacher education and classroom activities, education policy, special education, bilingual education, and the learning environment. During the TRP and CRP meetings, experts made recommendations for items to be tested in these cognitive interviews. The items proposed for the second-grade cognitive interviews reflect the recommendations made by the TRP and CRP.
To attract teachers to participate in the cognitive interviews and to thank them for their time and effort, participating teachers will receive $35.
Teachers who participate in cognitive interviews will largely be recruited through schools in which Westat has an established relationship. This includes schools that are local to Westat (i.e., in the DC metropolitan area) and those that are located throughout the country and thus will provide some diversity in respondents. Most of these schools participated in the 2009 ECLS-K:2011 field test and either the cognitive interviews conducted in the spring 2011 or those conducted in the summer of 2011. In addition to recruiting volunteers through Westat’s network of contacts, advertisements may be placed on listservs geared toward teachers.
Parents participating in the timing tests will be recruited through the network of contacts that Westat has with parents, as well as through advertisements. To attract parents to participate in the cognitive interviews and to thank them for their time and effort, participating teachers will receive $35.
At the beginning of the cognitive interview respondents will be informed that their participation is voluntary and that all information they provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573]. Oral consent will be obtained from all respondents at the time of the interview. The script for oral consent is incorporated into the cognitive interview protocol and can be found in appendix C. No personally identifiable information will be maintained after the interview analyses are completed. With respondent permission, interviews will be audio recorded for later analysis. If the respondent indicates that he/she does not want to be audio recorded, only written notes will be taken. The recordings and notes will be destroyed at the conclusion of the ECLS-K:2011.
We anticipate contacting approximately 38 school coordinators to help identify teachers for recruitment, and we anticipate contacting approximately 38 teachers and 20 parents during recruitment for interviews to test the teacher questionnaires and the parent interview, respectively. The expected burden for recruitment per teacher is 5 minutes on average plus 5 minutes on average for obtaining information from each school coordinators, for a total of about 7 burden hours. Cognitive interviews will be conducted with 25 teachers. Including the time to review the instrument prior to the interview, response burden for teachers is expected to be about 65 minutes for a total of about 27 burden hours. Thus, total burden hours for teachers, including recruitment, time to review the questionnaire, and cognitive interviews is 34 hours. The expected burden for recruitment per parent is 5 minutes on average for a total of approximately 2 burden hours. Interviews will be conducted with 18 parents. Response burden for parents is expected to be about 65 minutes for a total of approximately 20 burden hours. Total burden hours for parents, including recruitment and timing test is 22 hours. Total burden hours for both teachers and parents is 56 hours.
There is no direct cost to the respondent.
The cost to the government to conduct the cognitive interviews is $146,132.
Recruitment for the cognitive interviews and interviews are expected to begin as soon as possible in February 2012. The results of the cognitive interviews will be initially discussed via memorandum with NCES immediately following the completion of the cognitive interviews, then a report summarizing the findings will be finalized in November, 2012.
1Throughout this package, reference is made to the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99. It is referred to as the ECLS-K. The new study for which this submission requests approval is referred to as the ECLS-K:2011.
2 In the spring 2011first-grade cognitive interviews, a cooperating teacher was found in every school that agreed to participate. That is, the teacher response rate had 100 percent correspondence with the rate at which schools agreed to participate.
| File Type | application/msword |
| File Modified | 2012-02-06 |
| File Created | 2012-01-27 |