Volume I:
Request for Clearance for the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Response to Intervention (RTI) Items Cognitive Interviews
OMB# 1850-0803 v. 51
June 2011
The Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) is a survey that focuses on children’s early school experiences beginning with kindergarten and continuing through the fifth grade. It includes the collection of data from parents, teachers, school administrators, and nonparental care providers, as well as direct child assessments. Like its sister study, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K),1 the ECLS-K:2011 is exceptionally broad in its scope and coverage of child development, early learning, and school progress, drawing together information from multiple sources to provide rich data about the population of children who are kindergartners in the 2010-11 school year. As with the original ECLS-K, the ECLS-K:2011 is sponsored by the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED). Fall and spring collections in the kindergarten year are being conducted for NCES by Westat, with the Educational Testing Service (ETS) as the subcontractor developing the child assessments. Clearances for studying the ECLS-K:2011 cohort were granted for the fall 2009 field test data collection, fall 2010 and spring 2011 kindergarten national data collections, and the fall first-grade data collection (OMB No. 1850-0750).
ECLS-K:2011 is the third in an important series of longitudinal studies of young children sponsored by the U.S. Department of Education that examine child development, school readiness, and early school experiences. It shares many of the same goals as its predecessors, the ECLS-K and the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), but also advances research possibilities by providing updated information and addressing recent changes in education policy.
Like its predecessors, ECLS-K:2011 will provide a rich and comprehensive source of information on children’s early learning and development, transitions into kindergarten and beyond, and progress through school for a new cohort of children.
ECLS-K:2011 will provide data relevant to emerging policy-related domains not measured fully in previous studies.
Coming more than a decade after the inception of the ECLS-K, ECLS-K:2011 will allow cross-cohort comparisons of two nationally representative kindergarten classes experiencing different policy, educational, and demographic environments.
This is a request for clearance to conduct cognitive interviews to inform the instruments planned for the ECLS-K:2011 spring first-grade data collection. Specifically, the goals of this field test are to identify and correct problems with ambiguity in or misunderstanding of items that have been adapted from other studies or have been newly developed for the spring first-grade teacher and school administrator questionnaires. These items are intended to evaluate the extent to which schools and teachers are implementing all or some components of a Response to Intervention (RtI) framework.
RtI is a multi-step approach to providing early and progressively intensive intervention and monitoring within the general education setting. RtI can be defined as a system for general, remedial, and special education that integrates assessment, evidence-based intervention, and student monitoring within a multi-tiered system designed to maximize student achievement and reduce behavior problems by tailoring the type and intensity of interventions based on individual student performance. RtI can also be used to identify students with learning disabilities.
RtI is intended to support improved academic achievement for all students. It offers a model for early intervention to prevent failure by identifying students who are struggling with the general curriculum in the classroom. Children’s placement in different levels or tiers of services is data-driven. A hallmark of RtI is an integrated system of assessment and monitoring at every stage of the process (Burns & Ysseldyke, 2005, Coleman et al., 2006). All students are periodically compared to their classmates using pre-determined benchmarks or local or national norms. Students determined to be at risk in the area being assessed (e.g., reading, math, behavior) receive a targeted, evidence-based intervention and the student’s progress is monitored. If the student improves, the student returns to general classroom instruction. Frequent monitoring occurs to ensure that progress is maintained following the intervention. If the student does not improve, the student may receive a more intensive intervention. Thus, the approach calls for dynamic assessment that allows practitioners to respond to children’s needs (Fuchs & Fuchs, 2006). Progress is regularly assessed. RtI has become an increasingly popular educational approach since the 2004 reauthorization of IDEA because the reauthorization allowed for the use of information about students obtained through RtI practices in identifying students with a specific learning disability.
Items related to RtI practices in schools are proposed for inclusion in the ECLS-K:2011 to complement data collected in the RtI Impact Study being conducted by the National Center for Education Evaluation (NCEE). NCEE is the cosponsor for this particular aspect of the ECLS-K:2011. The NCEE RtI Impact Study will focus on schools that are currently implementing RtI programs to evaluate the effects of those programs. The goal for the ECLS-K:2011 will be to collect information at a national level to better understand the extent to which schools across the country are implementing identified RtI programs or are using practices that would be identified as RtI practices, even if the school has not formally adopted an RtI program.
NCES has a contract with Westat to conduct the spring first-grade data collection for ECLS-K:2011, including the field test activities described here.
Cognitive interviews will be conducted to evaluate the items related to RtI practices used in schools and classrooms (appendix A) for the ECLS-K:2011 spring first-grade teacher and school administrator questionnaires. The items in this request were either adapted from national or state-level studies or were newly developed specifically for ECLS-K:2011. They have not been included in past administrations of the ECLS-K or the ECLS-K:2011.
While the cognitive interviews will focus on the RtI-related practices items, respondents will be asked to briefly review the full questionnaire for their respondent type (e.g., teachers will review the teacher questionnaire, school administrators will review the school administrator questionnaire) so they understand the general context in which the tested questions are asked. In general, participating teachers and school administrators will be asked to discuss their interpretation of items, to discuss the process they would use to obtain an accurate response to the items, and to provide suggestions for any unclear items or items that are difficult to answer. Question wording may change during the testing period in response to suggestions from teachers and administrators who are interviewed earlier in the testing period.
The request to conduct the national spring first-grade data collection will be submitted at a later date under OMB clearance for ECLS-K:2011 (OMB # 1850-0750), which is authorized under 20 US Code section 9543, which states that the purpose of NCES is “to collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations.”
The cognitive interviews will be conducted as one-on-one interviews between a responding teacher or school administrator and an experienced qualitative interviewer. Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer a series of questions about the items he or she just answered. This approach includes asking probing questions, as necessary, to clarify points that are raised in the think-aloud comments.
Cognitive interviews will be conducted with 25 first-grade teachers and 25 elementary school administrators. Because we are interested testing questions on RtI practices regardless of whether a school has formally adopted an RtI framework, we will attempt to recruit school administrators and teachers who work in public schools that have adopted an RtI framework in the school as well as those who work in public schools that have not adopted an RtI framework.2 Although this is a purposive sample, we will recruit teachers and school administrators from public elementary schools with various characteristics, e.g., from rural and urban schools located throughout the country.
After speaking with study staff and agreeing to participate in a cognitive interview, respondents will be mailed a letter explaining the purpose of the cognitive interviews (appendix D) and a copy of either the teacher or school administrator questionnaire, depending on their respondent type, to review prior to the cognitive interview (appendices E and F). We will instruct respondents to take 5 minutes to briefly review the entire questionnaire prior to the cognitive interview. While the interviews will focus only on selected items, reviewing the entire questionnaire will provide some additional context for the respondent. Interviewers for the cognitive interviews will follow a prewritten protocol (appendix B) but will be free to deviate from the protocol in order to address specific issues or anomalies in the respondents’ verbal reports. The full teacher questionnaire covers a wide range of topics including classroom and teacher characteristics, class organization and resources, instructional activities and curricular focus, parental involvement, evaluation and grading practices, school and staff activities, school climate, school environment, and teacher background. The full school administrator questionnaire includes questions about school characteristics; school facilities and resources; school-family-community connections; school policies and practices; school programs for particular populations; federal programs including Title I, Adequate Yearly Progress, and Title III; staffing and teacher characteristics; and school administrator characteristics.
The RtI items that we propose to test for the school administrator and teacher questionnaires were either adapted from existing studies or were newly written specifically for the ECLS-K:2011. Some of the items in the school administrator questionnaire ask specifically about RtI practices and will be asked of school administrators who report that an RtI framework has been implemented in the school. Other items address RtI practices and will be asked of all school administrators but do not directly ask about RtI by name.3 Items on RtI implementation include the number of years that RtI has been implemented in the school, the areas (e.g., reading, math) for which RtI has been implemented, the levels of instruction/intervention that are provided, and how the school has communicated with parents about RtI implementation. Items on RtI practices that will be asked of all school administrators include whether a formal program to reduce behavior problems has been implemented in the school, the use of RtI-related practices (without using the term RtI), the types of staff available to support classroom teachers, communication with parents about special education eligibility, and approaches to determining special education eligibility.
The items in the teacher questionnaire focus on RtI-related practices without using the term RtI and will be asked of all cognitive interview teacher respondents. Items obtain information on school-level practices for administering instruction and interventions, the levels of instruction and interventions that are available to students, staffing, professional development, and the frequency and purpose of assessments. Items on RtI-related practices are asked separately for reading and math.
Cognitive interviews with teachers are expected to last about an hour and 15 minutes on average. Because some items for the school administrator will only be asked if the school administrator reports that RtI is used at the school, the cognitive interviews with school administrators are expected to vary in length. For schools with an RtI framework, the cognitive interviews with the school administrator are expected to last about an hour and 15 minutes on average. For schools that do not have an RtI framework, the cognitive interviews with the school administrator are expected to last about an hour on average. Thus, the total burden per cognitive interview, including 5 minutes to review the questionnaire, is anticipated to be about 80 minutes for teachers and between 65 and 80 minutes for school administrators.
Based on current school response rates from the spring 2011 first-grade cognitive interviews of the ECLS-K:2011, we anticipate a response rate of about 66 percent. Thus we anticipate contacting about 38 teachers to obtain 25 responding teachers and about 38 school administrators to obtain 25 responding school administrators. In total, an estimated 76 teachers and school administrators will be contacted. During recruitment, a recruiter will explain the purpose and anticipated length of the cognitive interview and ask the respondent if he or she would like to participate. The recruitment burden is anticipated to be about 5 minutes on average (see appendix C for recruitment script).
In preparation for the spring first- and second-grade data collections, one Technical Review Panel (TRP) and three Content Review Panels (CRP) were held in March 2011 to review and comment on issues related to the instruments. The members of these panels included experts in child development, teacher education and classroom activities, education policy, special education, bilingual education, and the learning environment. The panels also included members who were experts in RtI. During the TRP and CRP meetings, RtI and other experts made recommendations on topics related to RtI to add to the school administrator and teacher questionnaires. In addition, these experts made recommendations on possible sources for existing questionnaire items that could be used or adapted for the purpose of this study. New RtI items for the school administrator and teacher questionnaires were drafted by Westat and then reviewed by selected members of the TRP and CRP who had RtI expertise, as well as NCES and NCEE staff. Comments made by RtI experts have been incorporated into the RtI-related questions proposed for cognitive interviewing.
Participating teachers and school administrators will receive $40 for their participation. This incentive is necessary to attract teachers and school administrators who have many competing priorities.
Because these cognitive interviews will be conducted over the summer months when schools are closed or staffed at lower capacity, we will recruit elementary school administrators and first-grade teachers through a network of contacts that Westat has with teachers and school administrators throughout the country. Members of our Technical Review Panel (TRP) and Content Review Panels (CRP) who have expertise in RtI and have worked with schools that have adopted RtI will assist in identifying schools that have adopted RtI. We will also attempt to recruit through teachers’, administrators’, and general educators’ associations in which teachers and school administrators participate. Cognitive interviews will be conducted via telephone, and teachers and administrators will complete the interviews on their own time (i.e., if they are staffing the school during the summer, the interviews will take place outside of their normal work/school day).
At the beginning of the cognitive interview respondents will be informed that their participation is voluntary. Oral consent will be obtained from all respondents at the time of the interview. The script for oral consent is incorporated into the cognitive interview protocol and can be found in appendix B. No personally identifiable information will be maintained after the interview analyses are completed. With respondent permission, interviews will be audio recorded for later analysis. If the respondent indicates that he/she does not want to be audio recorded, only written notes will be taken. The recordings and notes will be destroyed at the conclusion of the ECLS-K:2011.
We anticipate contacting approximately 38 teachers and 38 school administrators during recruitment for cognitive interviews. The expected burden for recruitment per person is 5 minutes on average for a total of about 6.3 burden hours. Cognitive interviews will be conducted with 25 teachers and 25 administrators. Including the time to review the instrument prior to the interview, response burden for teachers is expected to be about 80 minutes and response burden for administrators is expected to range from 65-80 minutes, depending on whether the school has adopted an RtI framework. This is a total of about 63.5 burden hours.4 Total burden hours, including recruitment, time to review the questionnaire, and cognitive interviews is 69.8 hours.
There is no direct cost to the respondent.
The cost to the government to conduct the cognitive interviews is $66,431.
The recruitment for the cognitive interviews is scheduled to begin in July, 2011. The results of the field test will be initially discussed via memorandum with NCES immediately following the completion of the field test; a memorandum summarizing the results of the cognitive interviews will be delivered to NCES in September, 2011.
1Throughout this package, reference is made to the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99. For ease of presentation, it is referred to as the ECLS-K. The new study for which this submission requests approval is referred to as the ECLS-K:2011.
2 While RtI may be used in private schools, we expect it to be more widely used in public schools. For this reason, we are only including public school staff in our cognitive interview sample,
3 Both TRP and CRP members strongly recommended asking about RtI practices without specifically referencing RtI for a few reasons. First, it was felt that schools that have not adopted a formal RtI program may not be familiar with the term and therefore may be confused by questions that reference RtI or be likely to under-report practices they use that are common to RtI programs if they do not specifically identify their school as using RtI practices. Second, it was thought that schools with a formal RtI program might be likely to report they have a practice if it is identified as RtI even if they do not use that specific practice in the school. Lastly, there was greater interest in knowing what specific practices schools are using rather than whether they believe they have a formal RtI program in place.
4 Burden calculations in this OMB package used an average of 72.5 minutes for the respondent burden for school administrators.
| File Type | application/msword |
| File Modified | 2011-06-30 |
| File Created | 2011-06-30 |