Case StudiesTO 17 Supporting Statement B (8-21-14)

August 22, 2014

U.S. Department of Education

PPSS TO 17: Case Studies of the Implementation and Use of Kindergarten Entry Assessments

Supporting Statement Part B: Collection of Information Employing Statistical Methods

Third Draft OMB Package

Contract Number GS-10F-0554N; Order Number ED-PEP-11-O-0090

SRI Project P22275.431

Submitted to:

Erica Lee

Policy and Program Studies Service

Office of Planning, Evaluation and Policy Development

U.S. Department of Education

400 Maryland Avenue, SW

Washington, DC 20202

Prepared by:

Teresa McCaffrey, Kaily Yee, Shari Golan, and Tracy Huang

SRI International

Contents

B. Collection of Information Employing Statistical Methods 1

B.1. Sampling Design 1

B.2. Procedures for the Collection of Information 4

B.4. Test of Procedures and Methods 9

B.5. Consultations on Statistical Aspects of the Design 9

References 11

Appendices

A. Consent Form

B. Recruitment Materials

C. Document Review Protocol

D. State Version Interview Protocol

E. District Version Interview Protocol

F. Principal Interview Protocol

G. Kindergarten Teacher Interview Protocol

H. Test Administrator Interview Protocol

I. Preschool Director Interview Protocol

J. Professional Development Provider Interview Protocol

Exhibits

Exhibit 1. Number of Schools by Recommended Stratifications Across States 3

Exhibit 2. Recommended District Stratifications 3

Exhibit 3. Recommended School Stratifications 3

Exhibit 4. Pilot States and Respondents 9

Supporting Statement, Part B
Paperwork Reduction Act Submission

B. Collection of Information Employing Statistical Methods

B.1. Sampling Design

The objectives of this study are to document the processes, accomplishments, challenges, and solutions of four states implementing Kindergarten Entry Assessments (KEAs) and to share what they have learned with federal and state policymakers and the field. Of particular interest is to identify what is working well in states that are early adopters of KEAs. Data collection will occur in four states implementing KEAs and 12 districts and 24 schools nested within those states. This sample size, as determined by the scope of work, will enable the study team to explore a range of strategies that states, districts, and schools are using to implement KEAs.

Sampling Criteria

Selecting States for the Study

In consultation with the Department of Education and Health and Human Services, the study team used the following criteria to narrow down the number of states for inclusion in the study. The case study states must first meet four basic criteria to ensure relevancy to federal and state policymakers:

1. The state’s KEA is comprehensive and covers all five domains of school readiness.

2. The state is implementing its KEA in the beginning of kindergarten and not at the exit of prekindergarten (PreK).

3. The state is planning to be in full implementation of KEA by fall 2014.

4. The state is not part of the North Carolina KEA-EAG consortium to avoid a conflict of interest. SRI International, the principal researcher on this Task Order, is helping North Carolina and its nine partner states to enhance their KEA as part of their KEA-EAG.

Applying these criteria reduced the potential sample of eligible case study states to five: Illinois, Maryland, Ohio, Pennsylvania, and Washington.

In addition to meeting these fundamental criteria, the Department and HHS further recommended that at least one of the selected states represent one of the following:

5. An RTT-ELC grantee state (IL, MD, OH, PA, WA)

6. A non–RTT-ELC grantee state (None of these states met all the criteria.)

7. An EAG state (MD, OH)

8. A user of a commercially available KEA assessment tool (e.g., TS-GOLD: WA; and DRDP-SR: IL)

9. A user of a KEA assessment tool developed by the state (MD, OH, PA)

Four states—Maryland, Illinois, Washington, and Pennsylvania—met the fundamental and recommended criteria for participation in the case studies and also provide useful variation on the many key KEA design elements. Ohio, a fifth state, also met the fundamental criteria but is part of the Maryland EAG-RTT consortium and using a similar approach to Maryland. If any of the suggested states decline or are unable to participate in the case study, other options exist that may be able to replace them (e.g., Ohio for Maryland, Colorado for Washington, California for Illinois, New Mexico for Pennsylvania). California, Colorado, and New Mexico do not meet the four basic criteria, but they provide the best alternatives.

Selecting Districts and Schools Within the Four Case Study States

After the study team has reached agreement with the Department on the selection of four states and has secured permission from each state education agency (SEA) to conduct the study, it will begin selecting three districts within each selected state and two schools within each of those districts. The sample will be limited to public school districts with noncharter, regular schools¹ that have kindergarten enrollment.

The study team will purposively select the districts and schools within each state. The selection will be informed by a district and school sampling frame based on selected demographic variables (Exhibit 1) and by recommendations from the SEA and district contacts regarding districts and schools that are implementing KEAs in the fall of 2014. This stratification process will ensure that the state case studies represent a variety of schools and student groups and focus on districts and schools fully implementing their states’ KEAs. The steps to identify the districts and schools are described below.

At the district level within each state, the study team will draw a purposive sample of three school districts stratified by urbanization characteristics: city, suburban, and rural/town. By including districts from communities that vary in urbanization in each state, the study team can capture variations and issues in KEA implementation that may be related to community characteristics (e.g., types of challenges in family engagement, access to PreK and early learning programs).

At the school level, the study team will draw a sample of schools with high concentrations of poverty and significant concentrations of English learner (EL) students and schools with lower concentrations of poverty and EL students. Many of the schools with a significant percentage of EL students may also have a high percentage of students living in poverty because the public schools in which EL students are concentrated have, on average, a substantially greater proportion of students qualifying for free or reduced-price school lunches (Fry, 2008). The study team emphasized schools with a significant concentration of students living in poverty and/or students identified as EL because students in these groups tend to start kindergarten with lower school readiness skills than peers in other schools (Halle et al., 2009). Preschools included will be PreK programs serving 4-year-old children located at or near the selected schools.

The resulting sampling frame is a three-category district-level urbanization stratification by three-category school-level demographic stratification, resulting in a total of nine cells (Exhibit 1). Across the states and school-level stratification will be eight schools from each type of urbanization district (city, suburban, and rural/town). Across the states and district-level stratification will be nine high-poverty-enrollment schools, nine significant-EL-enrollment schools, and six schools with neither high-poverty nor significant-EL enrollment. The KEA implementation case studies will enable the study team to see how states may be using KEAs to better understand and address achievement gaps (e.g., improving early learning programs to better address areas in which students are not ready).

Exhibit 1. Number of Schools by Recommended Stratifications Across States

School enrollment	City district	Suburban district	Rural/town district	Total Schools
High poverty enrollment (above 75%)	3	3	3	9
Significant EL enrollment (above 30%)	3	3	3	9
Neither high poverty nor significant EL enrollment	2	2	2	6
Total schools	8	8	8	24

Within each state, the distribution of districts by urbanization will be the same, with one district from each urbanization category (Exhibit 2). In contrast, the distribution of schools by student demographics (i.e., high-poverty-enrollment schools, significant-EL-enrollment schools, and schools with neither high poverty nor significant EL enrollment) will differ across states (Exhibit 3). As six schools from each state will be included in the study, each state will fill only six of the nine cells in Exhibit 1. Exhibit 3 shows how the distribution of each state’s six schools may vary by state. Therefore, the study team will need to examine the selection of schools in a district together with the selection of schools in other districts both within and across states to ensure coverage of all nine shaded cells in Exhibit 1.

Exhibit 2. Recommended District Stratifications
	City	Suburban	Rural/town	Total
State 1	1	1	1	3
State 2	1	1	1	3
State 3	1	1	1	3
State 4	1	1	1	3
Total	4	4	4	12

Exhibit 3. Recommended School Stratifications
	High poverty enrollment	Significant EL enrollment	Neither high poverty nor significant EL enrollment	Total
State 1	2	2	2	6
State 2	3	2	1	6
State 3	2	2	2	6
State 4	2	3	1	6
Total	9	9	6	24

District and School Data Sources and Definitions

Publicly available data sources from the Common Core of Data (CCD) and statewide school data from the four selected states will provide district- and school-level information to help inform selection. The study team will obtain the most recent publicly available district and school data for the four selected states from the CCD.² The most recent year of data available is school year 2010–11. Data will be used to group districts and schools using the following definitions:

• Urbanization—To define the district-level urbanization categories, the study team will use the urbanization characteristics in the CCD: city, suburban, town, and rural.³ The study team will combine the categories of town and rural to form three urbanization categories: city, suburban, and town and rural. Town is combined with the rural category because they both have the same subcategories to describe their distance from an urbanized area (i.e., fringe, distant, and remote).

• High concentration of poverty—High poverty enrollment will be defined as schools where more than 75 percent of the students are eligible for free and reduced-price lunch program. This definition of high-poverty schools was used in the 2010 Condition of Education report published by the Department (U.S. Department of Education, 2010). Data will likely come from statewide school data from the four selected states.

• Significant concentration of EL students—Significant EL student enrollment will be defined as schools where more than 30 percent of the students are identified as EL students (U.S. Department of Education, 2010). This cut point is based on the finding that 53 percent of EL students attend schools with an EL student population of 30 percent or greater (August & Shanahan, 2008). Student EL enrollment data at school level are not available in CCD and will likely come from statewide school data from the four selected states.

• Neither high poverty nor significant EL concentrations—This category will be defined as schools with 75 percent or fewer students who are eligible for the free and reduced-price lunch program and 30 percent or fewer students who are identified as EL students.

B.2. Procedures for the Collection of Information

To learn more about KEA implementation in the four participating states in the case studies, the study team will engage in two different data collection activities: (1) a review of documents (e.g., planning guides, state or local reports, technical or training manuals) from the four selected states and (2) interviews with staff at the state, district, and school levels.

Review of Documents from Four Selected States

The study team will gather state and local documents from representatives and key informants from the four participating states, from training events and meetings hosted in the selected sites, and from federal, state, and district websites. The study team will capture essential information from each of the documents using a document review protocol (Appendix C). The study team will use the document review protocol to document the types of materials collected, the intended audiences (e.g., parents, teachers, school administrators, or the general public), the sources (e.g., website, training, district representative), and the contents of the documents. The documents reviewed will vary by site but will include, at a minimum:

• Relevant KEA legislation, regulation, policies, and guidance documents produced by state agencies;

• KEA design and specification documents, technical manuals, and reports produced by State Advisory Councils working on KEA planning and implementation;

• State and local outreach/communication plans; and

• Training materials.

The contents of the reviewed documents will be organized into topics including: (1) information about the KEA assessment tool (e.g., the purpose of KEA, the selection process, domains of assessment, availability of the tool in multiple languages, and accommodations allowed for children with disabilities), (2) information about administration of the KEA (e.g., assessment timeline and frequency, administration methods, numbers and percentages of various types of staff who participated in professional development for the KEA, if available, and information and training support provided to parents, teachers, and school administrators), and (3) use of KEA results (e.g., recipients of KEA results, data storage and tracking procedures, and information provided to parents, teachers, and school administrators). All collected documents will be stored on a secure website to facilitate sharing and analysis by the study team.

Before all phone interviews and site visits, data collection staff will review background documents on their sites, including state KEA requirements and guidelines, federal applications/proposals, and progress reports that describe KEA plans and activities. This background preparation is essential to use interview time efficiently, to interpret respondents’ answers, and to probe more deeply on relevant topics. During site visits, interviews, and observations, team members will gather additional documents for review and archiving, as relevant and suggested by key informants.

Interviews

To learn about KEA implementation from various perspectives, study team members grouped within state teams will interview up to seven state-level respondents per state, three district-level respondents per district and one professional development provider per district, four school-level respondents per school, and three PreK program directors per elementary school, for a total of 244 interviews. Interviews will take 45–90 minutes to complete, depending on the respondent, and will be audio-recorded with consent from the participant.

Phone interviews. Phone interviews will be used for state-level respondents, district-level professional development providers, and directors from preschool programs linked with case study schools. The study team will use an interview protocol for each of the respondent types: state interview protocol (Appendix D), preschool directors protocol (Appendix I), and the professional development provider protocol (Appendix J). All the protocols include both structured and semistructured questions. Structured questions with discrete response categories will allow for systematic collection of data that can be easily compiled and compared across respondents at the state, district, and school levels. Semistructured questions will gather rich descriptions of activities, challenges, accomplishments, factors that influenced implementation choices, and lessons learned.

Scheduling state-level interviews will begin as soon as the study team receives OMB clearance and the state has been notified about the study. The state case study leader will work with the state’s Race To the Top-Early Learning Challenge (RTT-ELC) grant’s main contact to identify state-level respondents most knowledgeable about KEA decision-making, implementation, and use of data (e.g., state superintendent; individuals responsible for state assessments, professional development, and state data systems; individuals from the state department of health and human services; and non-governmental researchers, advocates, and foundation representatives who were involved in the selection of the KEA). A final set of seven potential (and two alternate) participants will be nominated. The RTT-ELC main contact will be asked to communicate with selected participants about the study and to schedule some or all of the state-level interviews or, if preferred by the RTT-ELC main contact, the state case study leader will initiate contact and schedule interviews using the recruitment letter in Appendix B. The state case study lead together with a second study team member will conduct the state-level interviews by phone using interview protocols for state-level respondents. All interviews will be audio-recorded with participant consent and last approximately 90 minutes.

To learn about the types of supports that educators and other KEA assessors receive to build capacity to administer KEAs and use the data to inform practice, the study team will also conduct a phone interview with one professional development provider per district, for a total of 12 interviews. The study team will begin identifying respondents for these interviews during the state- and district-level interviews. After identifying professional development providers who worked with personnel in the case study districts, the district case study leaders will initiate communication with them to gain consent and schedule interviews at convenient times. The district case study lead together with a second researcher will conduct these interviews by phone using the interview protocol for professional development providers. All interviews will be audio recorded with participant consent and last approximately 45 minutes.

The study team also will conduct phone interviews with directors from three preschool programs per case study school, for a total of 72 interviews. To capture the experiences of a variety of preschool programs, the interviews will include PreK program directors from one Head Start program, one district-funded prekindergarten, and one private preschool, if possible. All interviews will be audio recorded with participant consent and last approximately 45 minutes. Each participating program will receive a $25 gift certificate as a token of appreciation. These interviews will occur after the on-site school-level interviews (discussed below) to ensure that interviewers have ample prior knowledge of the KEA as it pertains to the district and school.

On-site interviews. Two study team members will conduct on-site interviews with district- and school-level staff over a two-day site visit. Before visiting the district, the district case study lead will contact the district superintendent to request nominations for three key informants and two alternates (e.g., the superintendent or assistant superintendent of instruction, early learning director, and assessment director or person responsible for overseeing the collection of KEA data) to participate in district-level interviews. The district case study leader will then work with someone designated by the superintendent to schedule the interviews and school site visits or if preferred by the superintendent, the district case study leaders will communicate directly with nominated interview participants and scheduling interviews. Similarly, the district case study leader will work with the principals at each of the study schools to identify all of their kindergarten teachers and other staff who help administer the KEA in their schools. A sample of names will be drawn from the full list of relevant staff if there are more staff members than needed in a given respondent role. The district case study leader will then work with someone designated by the principal to schedule the school staff interviews.

The same district case study leader and a second researcher will conduct the district-level and school-level interviews using the district-level interview protocol (Appendix E) and school staff interview protocols (Appendix F, G, and H), as appropriate. All interviews will be audiotaped with participant consent. District interviews will last approximately 60 minutes each and school staff interviews will last approximately 45 minutes each.

The two study team members will spend one day visiting each case study school (school visits within the same district will be conducted on consecutive days) and schedule district interviews at the beginning or end of each those days. At each school, interviews will be conducted with the principal, two kindergarten teachers, and a test administrator if people other than teachers are involved in collecting KEA data. Interviews will be guided by interview protocols designed for the various school-level staff members: principals (Appendix F), kindergarten teachers (Appendix G), and other test administrators (Appendix H). During their visit, both study team members will be present at the district and principal interviews, but only one study team member will interview each teacher and test administrator. Thus, two teachers could be interviewed concurrently. This model has been effective in other studies involving similar site visits to districts and schools. Having a team of staff members conduct the district and principal interviews together provides each study team member with a broader perspective of the district and a similar context for conducting the teacher interviews independently. Furthermore, by having study team members interview the teachers and test administrators separately, the team can reach more respondents in a cost-effective and time-efficient manner. If district or school staff members are not available for interviews at the scheduled site visit, they will be interviewed later by phone.

Statistical Methodology and Estimation Procedures

The document review and interviews involve qualitative data collection. Statistical methodology is not applicable. The study team will quantify the qualitative data collected when this quantification can be done reliably.

Degree of Accuracy Needed

The study team will do everything possible to maximize the accuracy of the data collected for each of the case studies. All interviews (subject to the permission of the respondent) will be recorded to improve the accuracy of reporting. Furthermore, site visitors will attend detailed training and will review background information before planning their visit to ensure efficient, consistent, and accurate use of the data collection protocols. Accuracy of analysis will be further facilitated through the use of qualitative research data analysis software that will support reliability across those involved in coding responses.

Use of Periodic Data Collection

The team will visit each site only one time.

B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse

SRI has extensive experience in gaining access to states, districts, and schools for research purposes. The study team will use a top-down strategy for recruitment, starting at the state level followed by districts and then schools, and begin recruitment immediately upon OMB approval. The study team will invest time to thoroughly train all research staff involved in recruitment to clearly describe study purposes and requirements and to gain support and endorsement while being mindful of authority and decision-making structures at the state, district, and local levels. Using these recruitment strategies, the study team expects a response rate of 90 percent.

Upon receiving OMB approval, study team leaders will send a letter to the state superintendent informing him or her of the state’s inclusion in the study (sample letter in Appendix B). The state RTT-ELC coordinator will be copied on the letter, and study team leaders will contact the state superintendent within a week of sending the letter to discuss the study (including its importance and value), confirm KEA implementation in fall 2014, answer any questions, and obtain agreement to participate. If the superintendent agrees to having the state participate in the study, the study team will contact the state RTT-ELC coordinator who oversees the state’s KEA to also discuss the study.

The study team will ask the RTT-ELC coordinator to provide a list of districts implementing the KEA in fall 2014 (if implementation is optional) and to designate which of these districts are examples of implementation worth further study. Once the study team selects the three districts to include in the study, the study team will ask the RTT-ELC coordinator to help with district and school recruitment by letting the districts know about the study in advance of the study team contacting them. Next, the district study team will contact the district superintendent for each district in the sample by email and then follow up by phone within a week to discuss the study (including its importance and value), confirm KEA implementation in fall 2014, answer district questions, find out about any local research approval requirements, and obtain agreement to participate (sample letter in Appendix B). If a district declines to participate, the study team will contact districts on the alternate list.

After the district superintendent agrees to participate in the study, the district study team leader will ask the him or her to identify a main district contact with whom the study team will work to identify appropriate staff for the interviews and select and recruit schools from the list of potential schools identified by the study team through the sample stratification process described above. If there is no one to coordinate the scheduling of interviews at the district, the district case study leader will contact the district staff to be interviewed using the recruitment letter in Appendix B. If there are local requirements to approve research studies, the study team will follow them and submit the necessary application for conducting research.

Once the district has agreed to participate, the study team will ask the district contact to help with school recruitment by letting the school principals know about the study in advance of the study team contacting them. Next, the district study team will contact the principals at the two target schools by email to describe the study (including its importance and value), confirm KEA implementation in fall 2014, and obtain agreement to participate (sample letter in Appendix B). The district study team leaders will call principals within one week of the email to explain the study, answer the principals’ questions, and discuss the benefits of taking part in the study as well as expected level of effort that will be required of school staff. After a principal agrees to participate in the study, the district study team leader will ask the principal to identify a main school contact with whom the study team will work to identify appropriate staff for the interviews and help with the scheduling. If the school does not wish to participate, the study team will contact other schools within the district on the alternate list. In small districts where there are no alternate options, the study team may need to recruit a different district from the list of sample districts. In these special cases, the study team will consult the Department to develop an alternative plan.

To identify professional development providers who have been supporting the implementation of the KEA through training or coaching in the case study districts, the study team will ask the state and district contacts for nominations. The district study team will then contact the providers identified to explain the study and obtain agreement to participate first by email (sample letter in Appendix B). Within one week the email, the study team will call the professional development providers to follow up on the email and schedule an interview. If the professional development provider does not wish to participate, the study team will contact other providers on the list.

To recruit preschools linked to case study schools to participate, the study team will ask both the district and school contact for nominations. The district study team will then contact the preschool directors to explain the study and obtain agreement to participate by email (sample letter in Appendix B) and then follow up by phone within one week of the email. If the preschool does not wish to participate, the study team will contact other preschools on the alternate list. In districts where there are no alternate options, the study team will consult the Department to develop an alternative plan.

B.4. Test of Procedures and Methods

The study team piloted the interview protocol items to ensure clarity with respondents in states and districts that were not selected for the study but were implementing KEAs. Based on recommendations from TWG members, the Department, and the knowledge of the study team, the study team identified four sites (California, Oregon, Minnesota, and the District of Columbia) in which to pilot the interview protocols. The study team contacted state and district level staff in charge of local KEA implementation to request interviews and nominations of staff in other roles who were knowledgeable about KEA implementation and who might be willing to participate in the pilot. These nominated respondents were invited to participate and to nominate additional potential respondents. In total, the study team piloted six interview protocols with at total of 11 individuals at the state, district, school, and pre-kindergarten levels. No interview protocol was piloted with more than three respondents (Exhibit 4).

Exhibit 4. Pilot States and Respondents

	State	District	Principal	K Teacher	Pre-K Directors	PD Provider	Total
California	2			1			1
Oregon	1					1	2
Minnesota	1	2	1	1			5
Washington, D.C.					1		1
Total	4	2	1	2	1	1	11

To conduct the pilot interviews, study team members used a think-aloud technique. Respondents were sent the interview protocols in advance to review so that they could be prepared to provide comments about the clarity, completeness, and relevance of interview questions. Study team members asked respondents to answer a sample of interview questions to see whether questions evoked the desired types of information. Respondents were also asked to provide feedback on questions they found confusing, complicated, repetitive, or unable to answer because they lacked the information. The study team also asked respondents if there were any important issues or experiences regarding KEAs that were not captured by the current list of questions that the study team might want to add to the protocol. The study team also asked respondents to look at the response options for structured questions to identify if there were missing response options that reflected their experiences. Finally, the study team asked respondents whether certain types of documents requested in the protocol exist and whether they would have permission to share such documents with a study team member. Study team members took detailed notes on respondents’ comments and suggestions during the pilot interview, consolidated the findings, and made corresponding changes to the protocols as warranted.

B.5. Consultations on Statistical Aspects of the Design

Agency

Erica S. Lee, Education Research Analyst, in the Policy and Program Studies Service, Office of Planning, Evaluation and Policy Development, at the U.S. Department of Education is the Contracting Officer’s Representative for the study. She can be reached at 202-260-1463.

Contractors

SRI International will be responsible for data collection and analysis, under the direction of
Dr. Shari Golan, who can be reached at 650-859-4007.

References

August, D., & Shanahan, T. (Eds.). (2008). Developing reading and writing in second-language learners: Lessons from the Report of the National Literacy Panel on Language-Minority Children and Youth. New York, NY: Lawrence Erlbaum Associates and The International Reading Association.

Fry, R. (2008). The role of schools in the English language learner achievement gap. Washington, DC: Pew Hispanic Center.

Halle, T., Forry, N., Hair, E. C., Perper, K., Wandner, L., Wessel, J., & Vick, J. (2009). Disparities in early learning and development: Lessons from the Early Chilldhood Longitudinal Study - Birth Cohort (ECLS-B). Washington, DC: Child Trends. Retrieved from http://www.childtrends.org/Files/Child_Trends-2009_07_10_ES_DisparitiesEL.pdf

U.S. Department of Education. (2010). The condition of education. Closer look 2010. High-poverty public schools. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/programs/coe/analysis/2010-index.asp



1 A “regular school” refers to “a public elementary/secondary school providing instruction and education services that does not focus primarily on special education, vocational/technical education, or alternative education, or on any of the particular themes associated with magnet/special program emphasis schools” (http://nces.ed.gov/ccd/commonfiles/glossary.asp).

2 CCD data are available at https://nces.ed.gov/ccd/elsi/ .

3 Urbanization is determined by the urban-centric locale code in the CCD. The CCD uses four urbanization categories: city, suburb, town, and rural. In addition, the CCD includes four categories to describe the population size of the district’s city or town (i.e., large, midsize, small, and fringe). Urban districts will be districts that are coded in CCD as city large, city midsize, and city small; suburban districts will comprise categories of suburb large, suburb medium, and suburb small; and rural will comprise districts coded as town fringe, town distant, town remote, rural fringe, rural distant, and rural remote.