OMB Memo of FT Changes for 2011

MEMORANDUM OMB # 1850-0750 v.8

To: Shelly Martinez, OMB

From: Gail Mulligan, Chris Chapman, Jill Carlivati, NCES

Through: Kashka Kubzdela, NCES

Subject: Memo outlining changes between ECLS-K:2011 field test and the full scale data collection

Date: February 4, 2010

This memorandum summarizes the changes made to the national data collection procedures, protocols, and study materials and instruments that were approved on March 20, 2009 as part of the OMB clearance package (OMB No. 1850-0750 v.5) for the ECLS-K:2011 field test. These changes are being made for the full-scale data collection in response to the field test’s results and subsequent input from experts in other offices within the Department of Education, other federal agencies, and members of the study’s technical and content review panels.

Part A

We added a summary of the purpose, method, and results of the fall 2010 field test and described changes to the national data collection based on the field test results. The description of the field test and its results can be found in section A.1.e. More details on the results, which are provided as memos exchanged between NCES and the ECLS-K:2011 data collection contractor, Westat, can be found in Appendix A.1, A.2, A.3, and A.4.

Hearing screenings. We have removed the hearing screening from the description of the spring national data collection for now. Results of the field test were mixed, and we cannot go forward with exact protocols as field tested. We believe that with more time we can develop a plan that includes just a subset of the protocols already tested and collect reliable data in a way that is not overly burdensome for children. Additionally, it appears as though there may be funding to conduct vision screenings on a subsample of the national Kindergarten sample, which originally were not planned to begin until the first grade. As part of the current submission we are requesting approval to submit a future package with a waiver of the 60-day Federal Register notice that deals specifically with the hearing and vision screenings. In the Supporting Statement Part A, we are also requesting a waiver of the 60-day Federal Register notice for the fall 2011 first grade data collection and first and second grade respondent tracking and recruitment.

Direct assessment of science skills and knowledge

The description of the direct assessment of children’s science skills and knowledge has also changed. Based on the results of the field test, the direct assessment of science skills and knowledge will be administered starting in the spring of kindergarten (spring 2011). There will be no fall kindergarten science assessment. Rather than the two-stage test that was described in the previous package, a brief single-stage test of approximately 15-20 items will be administered to the children in spring 2011.

Direct assessment of English basic reading skills

A goal of the Spanish field test was to obtain an English Basic Reading Skills (EBRS) assessment which, as a stand-alone subset of the full reading assessment items, could be used to accurately measure the low-level English reading ability of kindergarten English Language Learners. Many of the EBRS items were common-functioning for the children in the Spanish field test and the children in the English field test, indicating that we can confidently administer an EBRS subset of the reading items to all children in the study, and that this subset of items will permit reliable estimates of English reading ability for those who have very low English reading ability as indicated by their performance on the EBRS. Having the EBRS assessment as a feature of the ECLS-K:2011 reading assessments will allow us to accurately measure early reading skills and ability for the full range of expected kindergarten English reading abilities, including the emerging abilities of the ELL children who would have been excluded from our assessment in the ECLS-K:1998-99 reading assessment.

While we are including descriptions of the EBRS and SBRS field test results in Appendices A.3 and A.4, we are currently conducting a final review of the Spanish field test results to determine the proper routing of children through the English and Spanish assessments. Our decision on how to route children will be based on additional review of Spanish field test data, a review of how the preLAS2000 data from ECLS-K:1998-99 related to EBRS items taken in 1998 by those who passed the preLAS2000, and IRT modeling parameters that will be needed for accurate score estimations in the national study. In this OMB submission, we are outlining the routing plan that would involve the greatest number of components. However, the additional components are very short such that changes in plans for how to route children through the reading assessments will not significantly affect overall burden. We hope to have our final plan developed within one week, and if it results in any changes to the plan outlined, we will amend this package.

Wrap-Around Early Care and Education Provider Questionnaire (WECEP)

The data collection approach for the WECEP has changed based on information from our field test. Originally, the WECEP was proposed as a computer assisted telephone interview with the child’s care provider. This approach was based on the method used in the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B). However, while the ECLS-B sample design was geographically clustered, it did not have clustering to the same degree as the ECLS-K:2011, which samples many children within the same school. In the field test, we asked the school coordinator or another knowledgeable school staff member about the after-schoolchild care providers that served children in their school, including how many kindergartners attended those programs.  Forty-four field test schools provided information on 58 child care arrangements that were used by children in their school, including an estimate of how many kindergartners attended that arrangement.   Out of 58 arrangements: 5 (9%) had only one child attend; 25 (43%) had two to five children; 11 (19%) had six to 10 children; 17 (29%) had more than 10 children (some schools named other child care arrangements, but could not estimate how many of their kindergartners attended; however, they did indicate that more than one child attended).

 

As a result, only 9 percent of care providers for whom information was obtained had one child in attendance while approximately half of the arrangements were attended by more than five kindergarten children. Granted, school staff likely would not know about the home-based care providers who cared for only one child in the school and about which the study would collect information (for example a grandparent providing care every day for 2 hours after school), but these data do indicate substantial clustering for the arrangements for which school staff could provide information. As the WECEP interview has a lengthy set of questions that ask specifically about the child, a CAPI interview would be unduly burdensome for a respondent who had to report on many children. As a result, we have modified our data collection method from a single CATI interview to a set of 4 self-administered questionnaires: center-based center director questionnaire, center-based provider/teacher questionnaire, home-based provider/teacher questionnaire, and a child-level questionnaire that will be completed by both center- and home-based providers. These questionnaires have been reviewed by the experts in the Administration of Children and Families who are funding this part of the study to ensure that the questions are understandable and appropriate in hard-copy. Where necessary, help text that was available to telephone interviewers has been included in the questionnaire for respondents to read. Contact information (including mailing address and telephone number) for the child-care providers will be collected from the parent in the parent interview and the appropriate set of questionnaires will be mailed to the home-based or center-based provider.

Because of the change in design and the expectation of clustering in the child-care centers, we also are proposing a change to the incentive structure that was previously approved. In the last clearance package, we proposed the incentive structure that was used in the ECLS-B, specifically to give all providers $35. This was appropriate in the ECLS-B because providers typically only cared for one study child. This structure would be unfair in the ECLS-K:2011, because all providers would receive the same amount regardless of how many children we asked them to provide information for, even though burden increases with the number of children in the arrangement. Therefore, in the current submission, we proposed offering WECEP providers an honorarium depending on the number of sampled children in their care. The honorarium will be included with the packet of questionnaires that are mailed to the provider. Center-based providers with one sampled child will be offered a $15 honorarium; center-based providers with two to five children will be offered a $20 honorarium; center-based providers with six to ten children will be offered a $30 honorarium, and center-based providers with more than ten children students will be offered a $35 honorarium. The honorarium structure is slightly different for home-based providers as they have fewer questionnaires to complete and are expected to have fewer sampled children than center-based providers. Home-based providers with one sampled child will be offered a $10 honorarium; home-based providers with more than one child will be offered $15 honorarium. An incentive structure such as this is consistent with the incentive structure used for teachers in the ECLS-K:2011, which also provides honoraria based on the number of study children we ask teachers to provide information for.

Because of the change in approach, we have modified the response burden accordingly.

Part B

As described above, we have made several changes to the data collection design since the previous submission. As a result, the data collection methods in Part B have been modified accordingly.

Part C

Based on the suggestions of technical and content review panel members and federal staff from other offices within and outside of the Department of Education, as well as other modifications to the study instruments, we have modified the lists of the instruments’ covered constructs and related research questions. Also, as requested by OMB during review of the previous package, we have added tables for each instrument that provide links between the instrument items, the covered constructs, and the related research questions, which can be found in Appendix G.

Appendices

In the OMB clearance package approved on March 20, 2009, the Appendices A-E contained item pools for the various ECLS-K:2011 instruments. These item pools contained items fielded in several different rounds of the ECLS-K, along with new items we were considering for inclusion in the ECLS-K:2011. The appendices for the current submission now contain the final, formatted instruments to be fielded in the fall and spring of the kindergarten year (2010-2011 school year), including instructions to the respondent on how to complete the instrument.

Many items that were in the original item pools were included in the current proposed parent interview instruments with no change. However, many items that were in the original item pools were modified or dropped due to recommendations from our Technical Review Panel or other subject matter experts, burden considerations, or grade-level appropriateness. For each of the original appendices, the attached ECLS-K 2011 Memo Outlining FS Changes Attachment.xlsx file provides tables detailing whether each particular field test item was retained with no change, retained with revisions, or dropped from the item pool. In the description of the appendices below, we note which instruments were reviewed by federal staff within and outside of the Department of Education.

Prior Appendix A, now Appendix B (A.1, A.2, A.3, A.4, & A.5)

Now named Appendix B, the proposed fall (B.1) and spring (B.2) computer-assisted telephone parent interview instruments are located here. Please see Attachments A.1, A.2, A.3, A.4, and A.5 to see the item-by-item parent interview list of whether an item was retained with no change, retained with revisions, or dropped from the item pool. The most significant changes made to the parent interview were those to the child health and disability section. This section was reviewed by many people, including experts in NCSER and the OMB-convened panel on measuring disability. Revisions were made to existing items to ensure that the instruments use the most current and accurate terminology related to special education and childhood disabilities, as well to ensure that we are capturing appropriate information. Some items were added to capture more detailed information, and some skip patterns were changed to capture information for a larger group of children. For example, when the list of diagnosed disabilities was first developed for the ECLS-K, it focused only on the categories covered by IDEA. We have updated the list of disabilities to capture a wider range of childhood disabilities. The sections on vision and hearing have been expanded to capture additional information that would be useful in conjunction with results from the hearing and vision screenings. Also, in response to an OMB request during review of the last package, we have added an item asking parents if the study child ever received services through an Individualized Family Service Plan, which will allow us to follow these children with certainty through later waves of the study.

In the current OMB clearance package submission, Appendix A has summaries of the results from the ECLS-K:2011 fall 2009 field test.

Prior Appendix B, now Appendix C

Now named Appendix C, the proposed school administrator questionnaires are located here. Please see Attachment B to see an item-by-item list of whether an item was retained with no change, retained with revisions, or dropped from the item pool. Items pertaining to Title III, which are similar to those pertaining to Title I that were asked in the ECLS-K, were added at the recommendation of technical review and content panel members who are experts in the education of English language learners. These items, as well as the definitions related to language and language instruction, were reviewed by experts in OELA and revised slightly to ensure that the terminology being used is appropriate and that the Title III questions ask about services and programs that would indeed be provided to children using Title III funds.

Prior Appendix C (C.1, C.2, & C.3), now Appendix D

Now named Appendix D, the proposed teacher questionnaires are located here. Previously, we had three questionnaires identified as A, B, and C, which is the structure that was used in the ECLS-K. Questionnaire A had questions about the classroom, questionnaire B had questions about the teacher, and questionnaire C had questions about the study child. We have changed the structure of the questionnaire so that parts A and B are now combined in one teacher questionnaire, and part C is now identified as the teacher child-level questionnaire. There also is a separate questionnaire for teachers new to the study in the spring that asks about their own characteristics. This additional questionnaire is necessary because we do not ask teachers who participated in the fall about their background again in the spring. Please see Attachments C.1, C.2, and C.3 to see an item-by-item list of whether an item was retained with no change, retained with revisions, or dropped from the item pool. The disability items in these questionnaires were reviewed by experts in OSEP and revised slightly to ensure that the disability categories are in line with the categories used in IDEA. It should be noted that the items that ask teachers to rate children in their classroom on social skills, problem behaviors, learning dispositions, and attentional focusing and inhibitory control, and those that ask teachers to describe their relationship with sampled children are not included in this package as they are copyright protected.

Prior Appendix D, now Appendix E

Now named Appendix E, the proposed special education teacher questionnaires (A & B) are located here. Please see Attachments D.1 and D.2 to see an item-by-item list of whether an item was retained with no change, retained with revisions, or dropped from the item pool. Items in these questionnaires were reviewed by experts in both NCSER and OSEP. Revisions were made to existing items to ensure that the instruments use the most current and accurate terminology related to special education and childhood disabilities, as well to ensure that we are capturing appropriate information. Per an OMB request during the review of the last OMB package, we have added an item to the college courses question to determine whether the special education teacher or service provider has taken classes in Response to Intervention techniques.

Prior Appendix E, now Appendix F

Now named Appendix F, the proposed wrap-around early care and education provider (WECEP) questionnaires are located here. As described above, the WECEP instrument has been changed from a telephone interview to four self-administered questionnaires: center-based center director questionnaire (F.1), center-based provider/teacher questionnaire (F.2), home-based provider/teacher questionnaire (F.3), and a child-level questionnaire for both home- and center-based care providers (F.4). Please see Attachment E to see an item-by-item list of whether an item was retained with no change, retained with revisions, or dropped from the item pool.

Prior Appendix F, now Appendix H

In the OMB clearance package that was approved on March 20, 2009, Appendix F contained sample respondent materials for the national data collection based on what had been fielded in ECLS-K. Now named Appendix H, we have included the remaining respondent materials that need OMB approval:

• Teacher letter

• School coordinator letter

• Parent letter (active consent)

• Parent letter (passive consent)

• Parent consent form (active consent)

• Parent consent form (passive consent)

• WECEP letter

For the study record, we also have included those materials that were approved previously by OMB for use in the national collection, including a study brochure, a study timeline, and study letters for states, districts, dioceses, and schools. Also a birthday card that was used in the ECLS-K is included as an example of the cards that will be developed for tracking purposes in future rounds of data collection. Finally, we are including a draft letter to school districts encouraging participation in the ECLS-K:2011; this letter, which is currently under review, would be sent under the signature of Secretary Duncan.

Note that a new Appendix G has been added, per OMB request during review of the last package, which describes the links between the instrument items, the covered constructs, and the related research questions.

Page 10 of 10