School-Level Respondents

Appendix C

Document Review Protocol

KEA Case Studies

Document Review Protocol

Purpose

The purpose of this protocol is to review documents in a systematic way to provide background information about a state’s KEA for interviews and more technical or detailed information about some aspects of the processes used for selection, implementation, and reporting and use of results. The data will be used to

• Provide background on a state’s KEA for state interviews and site visits on a state’s KEA development/adoption, implementation, and use for state and final report.

• Gather more technical and detailed information about the KEA tool and process.

• Abstract relevant information from documents obtained at interviews.

Resources

Documents may be obtained from the following sources:

1. Applications to Race-to-the-Top Early-Learning Challenge

2. State Department of Education or Department of Early Learning website, which contains the following types of resources:

• KEA strategic plans or planning documentation

• Pilot or field test report

• KEA information for various stakeholders

• KEA training materials

• Assessment manuals

• KEA reports with findings for various stakeholders.

3. State, district, or school interview respondents who share documents with the interviewers.

General Instructions and Tips

1. Populate header information first so you do not forget.

2. This is a cumulative document that will first be populated with information from publically available documents and later updated with documents provided to the study team by interview respondents.

3. There will be one document review per state. All sources of information should be listed in the table on page 2, “Document Source for Information.” Each document source should be numbered sequentially and referenced in each section as the review is being conducted.

4. Make notes of any questions you have when reviewing the documents. When unsure of the information, flag it and note the document name and page number, if available, and ask another team member to review. This will allow the interview team to identify areas in which clarification are needed.

5. You are not limited to the format or the table presented. Add any notes as needed.

Document Source for Information

Source #	Title	Date	Where document came from If website, list URL	Author
1.
2.
3.
4.
5.
6.

Add rows as needed

List sources of information used (by source number) next to each item below.

Section I: Background on KEA Development or Adoption

1. What year did the state begin the initiative to implement the KEA? (Q1)

2. What agency or department was the key driver of the KEA initiative? List any collaborations with another agency or department during the early stages of the KEA initiative. (Q1)

3. What other stakeholders (such as district representatives, Kindergarten teachers, PreK teachers, parents, foundation representatives, outside contractors/expert consultants) were engaged in the KEA selection process and how? (Q1a)

4. For what key purposes was the KEA selected or developed? [Check all that apply.] (Q1b)

Check	Key purposes	Source #
	Screening: To identify children who may need additional supports or further assessment
	Data-informed instruction: To identify specific skills to teach to individual students or groups of students
	Family engagement: To inform parents about children’s strengths and needs and ways to support learning
	Professional development: To inform professional development for teachers about a common student need
	Benchmark: To benchmark learning in order to track progress throughout the year
	EC program improvement: To inform improvement of early learning programs/policies
	Other (specify):

5. What criteria were used in the selection of the KEA? [Check all that apply.] (Q1c)

   Check	Criteria	Source #
   	Cost
   	Match to early learning standards
   	Burden
   	Use with certain populations
   	Connection to a PreK assessment
   	Connection to a K-3 assessment
   	Other (specify):

6. Which of the following KEA measures was selected? [Check all that apply.] (Q1d)

   Check	KEA measures	Source #
   	Teaching Strategies-Gold
   	Desire Results Developmental Profile-School Readiness
   	Work Sampling
   	BRIGANCE
   	DIBELS
   	Other (specify):
   	Other (specify):

7. How was the above measure modified, if at all, to better fit the state’s needs? (Q1d)

8. If state developed its own KEA measure, what was the main reason for doing so? (Q1d)

9. What agency or department is currently overseeing the implementation of KEA? (Q1)

10. Who was involved in selection of KEA measure? [Check all that apply.] (Q1a)

Check	People/agencies involved	Source #
	State department leaders
	District representatives
	Teachers
	Parents
	Outside contractors (specify):
	Experts (specify):
	Foundation representatives (specify):
	Other (specify):

11. What are the domains assessed by the KEA measure? [Check all that apply.] (Q2)

Check	Domains	Source #
	Language and literacy
	Cognition and general knowledge, including math and early science
	Approaches to learning
	Physical well-being, including motor development
	Social and emotional development
	Other (specify):
	Other (specify):
	Other (specify):
	Other (specify):

Section II: KEA Implementation

12. What information on the administration of the KEA is provided to various stakeholders? (Q5)

Stakeholders	Topics covered	Source #
District administrators
Principals
K teachers
PreK teachers
Parents
Other (specify):

13. What training or support do teachers and administrators receive on the administration of the KEA? [Check all that apply.] (Q5a)

Is the training/support offered the same for all schools and districts statewide or is it school or district specific? (Q6g)

Check	Type and source of training/support	How often it is offered	How long the training is	Specific to district or same statewide	Source #
	In-person training by:
	1-on-1 coaching by:
	TA by:
	Webinars by:
	Online self-paced courses or modules List URL:
	Other

14. When is the KEA administered (e.g., within what week of school)? (Q6a)

First assessment: ____________________________________________________

Second assessment (if applicable): ______________________________________

Third assessment (if applicable): ________________________________________

Any other administrations: ____________________________________________

15. Who administers or provides information used in the KEA? Also indicate what evidence or types of information are gathered. [Check all that apply.] (Q6b) (Q2,2b)

Are the procedures the same for all schools and districts statewide or is it school or district specific? (Q6g)

Use multiple rows for each skill/domain, depending on who collects the data and what information is collected.

Skill/ Domain	Who collects? (e.g., teachers, classroom aids, families, others)	What information is collected? (e.g., checklists, direct assessments, observation ratings)	Specific to district or same statewide	Source #

16. How is technology used to support KEA administration and scoring? [Check all that apply.] (Q6e)

Check if used		Source #
	Collect documentation for portfolios
	Provide direct assessment tasks online
	Enter data on ratings and student information
	Score data collected by teachers or others
	Generate reports on results
	Other (specify):

17. What strategies are used to help ensure that children with disabilities and English learners are accurately assessed? [Check all that apply.] (Q6f)

Strategies used	Children with disabilities	English Learners	Source #
State offers special training and support to those administering the KEA with CWD and/or ELs
Offer alternate assessments for CWD
Offer additional accommodations during administration of KEA for CWD
Offer support to conduct assessment in the child’s native language
Provide modified or adapted results
Other (specify):

18. How do schools and districts submit KEA data/results to the state? [Check all that apply.] (Q7)

Check if used		Specific to district or same statewide	Source #
	Paper forms
	Exports
	Data entry into special KEA data system
	Data entry into SLDS
	Other (specify):

19. List how the state or districts monitor and evaluate implementation of the KEA. (Q8)

• Xx

• Xx

Section III: Use of KEA

20. Who receives or has access to the results of the KEAs? [Check all that apply.] (Q9,9a,9b)

Check if used	Who has access to the results?	In what format (e.g., printed reports, web data system)?	When?	What type of information?	Source #
	Families
	PreK educators
	K teachers
	Principals
	District administrators
	SEA administrators
	Other (specify)

21. What supporting information or training do teachers, principals, district staff, and others receive regarding interpreting or using KEA results? [Check all that apply.] (Q10)

Check if used	Staff	Type of training/support (coaching, webinars, online modules, manuals or guidebooks, other)	Dosage of the training (frequency and length)	Source of training (training provider or URL)	Source #
	PreK educators
	K teachers
	Principals
	District administrators
	SEA administrators
	Other (specify)

22. When was the KEA pilot-tested? When was the KEA field-tested? What were the results?

• Pilot test:

• Field test: