Volume I:
Request for Clearance for the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Field Test of a Computerized Version of the Dimensional Change Card Sort
OMB# 1850-0803 v.60
November 7, 2011
The Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) is a survey that focuses on children’s early school experiences beginning with kindergarten and continuing through the fifth grade. It includes the collection of data from parents, teachers, school administrators, and nonparental care providers, as well as direct child assessments. Like its sister study, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K),1 the ECLS-K:2011 is exceptionally broad in its scope and coverage of child development, early learning, and school progress, drawing together information from multiple sources to provide rich data about the population of children who were kindergartners in the 2010-11 school year. As with the original ECLS-K, the ECLS-K:2011 is sponsored by the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED). Fall and spring collections in the kindergarten year were conducted for NCES by Westat, with the Educational Testing Service (ETS) as the subcontractor developing the child assessments. Clearances for studying the ECLS-K:2011 cohort were granted for the fall 2009 field test data collection, fall 2010 and spring 2011 kindergarten national data collections, and the fall first-grade data collection (OMB No. 1850-0750).
ECLS-K:2011 is the third in an important series of longitudinal studies of young children sponsored by the U.S. Department of Education that examines child development, school readiness, and early school experiences. It shares many of the same goals as its predecessors, the ECLS-K and the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), but also advances research possibilities by providing updated information and addressing recent changes in education policy.
Like its predecessors, ECLS-K:2011 will provide a rich and comprehensive source of information on children’s early learning and development, transitions into kindergarten and beyond, and progress through school for a new cohort of children.
ECLS-K:2011 will provide data relevant to emerging policy-related domains not measured fully in previous studies.
Coming more than a decade after the inception of the ECLS-K, ECLS-K:2011 will allow cross-cohort comparisons of two nationally representative kindergarten classes experiencing different policy, educational, and demographic environments.
This is a request for clearance to conduct a small scale field test to test the feasibility of including in the second-grade cognitive assessment battery a computerized version of the Dimensional Change Card Sort (DCCS) task, which was administered in the ECLS-K:2011 as a non-computerized (i.e., physical) version in the kindergarten and first-grade rounds. As the name suggests, the DCCS is a task in which children are asked to sort cards with different pictures according to a set of rules that change periodically. For example, first the child is asked to sort the cards based on the color of the pictures on them. Then the child is asked to sort the cards based on the shape of the pictures on them. In the third stage, the child is told that the cards should be sorted either by color or by shape depending on whether the card has a black border. How accurately a child sorts the cards is taken as a measure of the child’s cognitive flexibility, which is one aspect of executive functioning. Executive functioning can be conceptualized as consisting of three components (cognitive flexibility, working memory, and inhibition) and its relationship to academic achievement is currently of great interest to the research community.
For the second-grade round of data collection, the ECLS-K:2011 Executive Function Content Review Panel (CRP)2 recommended switching to a computerized version of the DCCS, which has the ability to measure card sorting response time, as well as card sorting accuracy. (The physical DCCS only measures card sorting accuracy.) According to the Executive Function CRP, it becomes more important to capture response time when measuring cognitive flexibility as children get older. One reason for this is that there is less variability in results on the physical version of this task as children age (i.e., more children sort accurately for more of the task, thereby producing a ceiling in the assessment), so additional data need to be captured in order for the card sort to be a meaningful measure of executive function. The computerized version of the DCCS captures this additional information.
The DCCS was developed by researcher Philip Zelazo. The computerized version of the DCCS has been field tested for inclusion in the NIH Toolbox for the Assessment of Neurological and Behavioral Function (henceforth referred to as the NIH Toolbox). The NIH Toolbox consists of a set of relatively brief assessments with good psychometric properties designed to work over a large range of ages (3-85 years). While the computerized DCCS is being field tested through the NIH Toolbox project,3 it is a newly developed system that has not been used in large-scale studies, such as the ECLS-K:2011. Additionally, the scope of the ECLS-K:2011 national data collection presents a host of potential logistical challenges to successfully collecting DCCS data with a computerized version of this task, necessitating a field test of the instrument for this particular study. For example, one challenge the ECLS-K:2011 faces is integrating the DCCS computer program with the ECLS-K:2011 child assessment computer –assisted personal interviewing (CAPI) application. Currently, each of the child assessment sections (i.e., reading, math, science) is programmed in a single application so, for example, when completed, the reading section flows smoothly into the math section. However, the computerized DCCS may be programmed on a different platform and it is unclear whether the assessment sections would be able to smoothly transition from one section into the computerized DCCS or the assessor would need to exit the CAPI application and launch the computerized DCCS separately. In terms of staffing, another challenge is designing an effective training program to prepare our staff to conduct the computerized DCCS.
Three main issues related to administering the DCCS via computer will be examined in the proposed ECLS-K:2011 field test. First, the field test will examine whether there are any administration issues that arise when the DCCS is conducted by field staff in the school-based settings typically available in the ECLS-K:2011 data collection, rather than in the smaller-scale research setting that has been the primary setting for the use of the computerized DCCS to date, (the exception being the ongoing NIH Toolbox testing). Another administration issue to examine in the field test is child behavior. In the prior rounds of ECLS-K:2011, reports from the field staff have been that the physical version of the DCCS has served as a break for the participants from focusing on the cognitive assessment easels. (During the kindergarten and first grade rounds, the physical DCCS is administered between the math and science cognitive assessments.) Our field staff reports that the physical DCCS task, which has the child physically sorting picture cards into different trays according to the periodically-changing rules given by the assessor, seems like a game to the children. The computerized DCCS is administered completely through the laptop computer. The field test will examine how second-grade children react to the different data collection mode. The field test will also examine if child fatigue is likely to occur when completing the task on the computer and what the assessor can do to help the child stay engaged and “on-task.” Also, we will be investigating whether the children have trouble with the instructions of the task, which are communicated by the computer, and what the assessors can do to help them understand the task. Another issue to examine in the field test is whether any technical issues arise with the computerized DCCS application (e.g,. the computer freezing up or crashing) that assessors will need to address during administration at a school. While these are examples of the types of administration issues we will examine during the field test, an additional important goal of the field test is to identify possible administration issues with the computerized task that we have not anticipated, as this is a new assessment component for the ECLS-K:2011. The second main issue to be addressed during the field test is the adequacy of the assessor training protocol for the administration of the computerized DCCS application. For example, information from the field test can identify steps in the computerized assessment protocol that may need to be highlighted or clarified in the training of the national data collection staff. Finally, the field test will examine the issue of whether a computerized DCCS application will work on the ECLS-K:2011 field laptops, can be integrated with the current CAPI applications, and can be customized so that DCCS data can be transmitted to the contractor’s home office in the same way as other child assessment data.
In sum, the overarching goal of the field test is to determine whether or not the computerized DCCS can be fielded in the national second grade assessment. To that end, the field test will:
Examine the feasibility of administration of the DCCS to second-grade children in school settings as part of the larger ECLS-K:2011 cognitive assessment;
Obtain feedback from field staff to improve training and administration for the national data collection;
Be a full systems test of all aspects of the DCCS;
Ensure the DCCS software opens and closes properly within the CAPI child assessment program;
Confirm the DCCS data are saved, stored, and transmitted properly; and
Evaluate the overall success of the administration of the DCCS (for example, how easily the instrument can be integrated into the child assessment and administered, how successful the training on the administration of the instrument was, and how the children respond to the instrument).
NCES has a contract with Westat to conduct the field test activities described here.
The request to conduct the national fall second-grade data collection, which is the first ECLS-K:2011 data collection in which the computerized version of the DCCS would be used, is currently under review at OMB. The ECLS-K:2011 collections are authorized under 20 US Code section 9543, which states that the purpose of NCES is “to collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations.”
In January and February 2012, NCES will conduct a field test of the computerized DCCS, which will be incorporated into the full ECLS‑K:2011 assessment battery for the field test. Schools sampled for the main (i.e., national) data collection will not be selected as field test schools. The field test will be conducted with approximately 220 children from a purposive sample of 9 schools local to the mid-Atlantic region to allow for NCES staff observation of the collection methods. In order to test feasibility in school settings, our goal is to include schools with different characteristics (e.g., large/small school sizes; high/low SES districts; diverse student bodies). In each field test school we will recruit second-grade students to participate in the assessments.
In order to test whether the computerized DCCS application can be integrated with the current CAPI applications, the field test laptops will include the entire cognitive assessment instrument: the two-stage reading, math, and science assessments; the Numbers Reversed task (an assessment of working memory); and the computerized DCCS. Each of these components will be administered to the child one-on-one by a trained assessor (although the DCCS, once initiated on the laptop by the assessor, will be primarily self-administered through use of the computer program). Field staff will be trained to record observations about children’s behaviors and response to the computerized DCCS and will also keep a more general diary of field test experiences. These field test observations will be used to prepare the field test report.
The general assessment methods used will be the same as those that have been used successfully for the fall kindergarten, spring kindergarten, and fall first grade data collection rounds. The assessment visit at each school is expected to take approximately 3 days, with each child’s assessment expected to last approximately one hour. The exact number of days for the site visit will depend on several factors, including the number of participating children at the school, any restrictions on the assessment schedule (e.g., assessments only in the morning), and the amount of space available for simultaneous assessments. The length of the site visit will be worked out with the school prior to the assessment team visiting the school. The assessment team that visits each school will include a team leader and three assessors. The assessment team will arrive at the school on the appointed first day of assessments and, following any of the school’s required check-in procedures, immediately contact the school coordinator.4 The team leader will introduce the assessors to the school coordinator. The procedures to be used during the on-site data collection period will be discussed with the school coordinator to ensure there is a common understanding of those procedures.
The team leader and assessors will be taken by school personnel to the assessment area(s), from which they will arrange to remove potential distractions as much as possible and establish a comfortable environment for conducting the assessment. They will set up the assessment materials and log into the child assessment CAPI program on the laptops they would have brought with them. All field staff will be provided with backup batteries, cords, etc., to ensure that data collection activities are not disrupted by equipment problems.
Once the assessment areas have been set up and assessors are ready to begin work, the school coordinator will introduce the ECLS-K:2011 team members to the teacher(s) whose students will be assessed. The teacher, in turn, will introduce the assessors to the class. Assessors will then escort the sampled children to the assessment areas, one-by-one, and conduct the full assessment which, as mentioned above, is expected to take about one hour. After completing the assessment, the child will be returned to the classroom and the next child will be assessed.
In preparation for the spring first- and second-grade data collections, a Content Review Panel (CRP) meeting was held in March 2011 to review and comment on issues related to the executive function assessments used in the ECLS-K:2011. The focus of the meeting was to discuss issues seen in the administration of the executive function tasks in the kindergarten rounds of data collection, as well as the best ways to measure executive function longitudinally as children age. The members of this panel included the following experts in the area of executive function: Clancy Blair at New York University, Adele Diamond at the University of British Columbia, Megan McClelland at Oregon State University, and Philip Zelazo at the University of Minnesota.
The experts strongly recommended continuing to use the same assessments of executive function across time and continuing to measure the same aspects of executive function in every round of data collection, as opposed to alternating the measures used between data collections. In order to continue using the DCCS as children age the experts noted that it becomes necessary to capture children’s response time (i.e., how long it takes a child to sort a card accurately) in addition to whether a child sorts a card accurately. This can be accomplished through the use of the computerized version of the DCCS.
The Westat DCCS field manager overseeing this field test will contact the school district selected for the study and ask for permission to contact nine elementary schools within the district and request their participation in the field test. Once district cooperation is obtained, recruitment letters explaining the field test will be mailed to the principals of the selected schools. After the letters are mailed and in cases where there is an established relationship between the selected school and Westat (e.g., schools that have participated in previous field test work for Westat), the DCCS field manager will visit the school personally; otherwise, the field manager will contact the principal by telephone. Whether in person or by telephone, the field manager will ask the principals for their cooperation and to identify second graders whose parents or legal guardians can be contacted about possible participation in the study. Given the main purposes of the field test (i.e., to evaluate the effectiveness of assessor training, to evaluate children’s reactions to the assessment and ease with which the instructions are understood when communicated on the computer, to test systems integration and operation, and to uncover any unanticipated issues with the use of the computerized assessment), a purposive sample, rather than a random sample, can be appropriately employed, because it is not necessary for the resulting data to pertain to a representative sample of children. Therefore purposive sampling will be used for the field test, such that the sample will consist of those children for whom parental consent to participate has been received. The field manager will make arrangements to have consent forms distributed to the parents of the eligible children, either to obtain signed consent or opt-out forms if the school does not require a signed consent form for participation. All children who have signed consent forms, or children whose parents have not opted out at the time of data collection, will be included in the field test.
While we expect an average of 24 children in each school, most likely some schools will have a higher number of eligible second graders, while others may have fewer eligible children. As stated above, we will include all the children for whom we have received consent, up to the target number of total complete assessments (i.e., about 200).
In order to thank school staff for their time and help with arranging rooms and coordinating student, we propose to remunerate schools $5 per child who participates in the study. Children, their parents, teachers, school administrators, and school coordinators will not receive an individual incentive.
School principals and parents will receive materials describing the study that clearly indicate their participation is voluntary. Furthermore, parents will receive consent forms that they must return to the school either to allow their child to participate or to provide notification that they do not want their child to participate, depending on the method of consent the school requires. These materials, which can be found in appendix A, also include the required language about disclosure and legal use of any data collected through this study that has appeared on other ECLS-K:2011 respondent materials. No personally identifiable information will be maintained after the assessment analyses are completed.
Table 1 shows the expected burden for the field test. The assessments are expected to last about an hour per child. However, because the child assessments are not subject to the paperwork reduction act, these hours are not typically included in burden hour estimates for the ECLS-K:2011. The hours included for the administrators, school coordinators, and parents are for recruitment.
Respondent type |
Sample n |
Response rate/
|
Number of respondents |
Burden hours per respondent |
Number of instruments per respondent |
Total number of responses |
Total burden hours |
Child |
216 |
1.00 |
216 |
1.00 |
1 |
216 |
216 |
Parent |
240 |
.90 |
216 |
.25 |
1 |
216 |
54 |
Administrator |
10 |
90 |
9 |
1.00 |
1 |
9 |
9 |
School Coordinator |
9 |
1.00 |
9 |
1.00 |
1 |
9 |
9 |
Study Total |
2591 |
NA |
2341 |
NA |
NA |
2341 |
721 |
NA Not applicable
1 The sample of students taking the direct assessment is not included in this count because it is not subject to the Paperwork
There is no direct cost to the respondent.
The cost to the government to conduct the field test is $265,571.
Recruitment for the field test is expected to begin in November 2011, with data collection taking place over a one-month period from January to February 2012. Recruitment will not begin until OMB approval for the proposed field test activities is received.
The results of the field test will be initially discussed via conference call with NCES at the end of March 2012, after the contractor has had time to analyze the field test results. A full field test report will be delivered to NCES in mid-April 2012, in time to prepare the final child assessment for the national fall second-grade ECLS-K:2011 data collection round.
1Throughout this package, reference is made to the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99. For ease of presentation, it is referred to as the ECLS-K. The new study for which this submission requests approval is referred to as the ECLS-K:2011.
2 The Executive Function Content Review Panel is a panel of experts assembled by the data collection contractor to review and comment on issues related to measurement of executive function. Please see the section “Consultants Outside the Agency” for details and a list of panel members.
3 According to a presentation by NIH Toolbox assessment developers at the 2011 Society for Research in Child Development (SRCD) meeting, the final version of the NIH Toolbox assessments is expected in September 2012.
4 Schools will be asked to assign a staff member to be the school coordinator to help coordinate the assessment activities at the school The school principal will appoint the school coordinator at the time of school recruitment (December). The school coordinator may be a school secretary, a second-grade teacher, or the principal him/herself.
| File Type | application/msword |
| File Modified | 2011-11-08 |
| File Created | 2011-11-07 |