National Evaluation of i3 (NEi3) Data Collection Survey

Appendix A:

National Evaluation of i3 (NEi3) Data Collection Survey

This work is supported by the U.S. Department of Education under contract number ED-IES-10-C-0064, ED-IES-13-C-0005 and/or ED-IES-14-C-0007, Abt Associates, Prime Contractor.

NEi3 Data Collection: Welcome Page

Welcome [User_Name]!

The National Evaluation of Investing in Innovation (NEi3) is being conducted by Abt Associates Inc. on behalf of the U.S. Department of Education (ED) Institute of Education Sciences (IES). A primary goal of the NEi3 is to assess the strength of evidence generated by independent local evaluations of i3 grants and to provide summaries of evaluation findings. The purpose of this data collection is to collect information from evaluators like you regarding the findings of your i3 independent evaluation. We look forward to working with you on this important endeavor! 

Abt Associates and its subcontractors follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183), which require that all collection, maintenance, use and wide dissemination of data conform to the requirements of the Privacy Act of 1974 (5 U.S.C. 552a), the Family Educational Rights and Privacy Act of 1974 (20 U.S.C. 1232g), and the Protection of Pupil Rights Amendment (20 U.S.C. 1232h). The study team will not be collecting any individually identifiable information. All of the data requested from independent evaluators will be in the form of aggregated reports of the methods, measures, plans, and results in their independent evaluations.

Responses to this data collection will be used to disseminate information about your study findings. Individual i3 grants will be identified in this study. The characteristics, results, and findings reported by independent evaluators, as well as assessments of the quality of the independent evaluations and i3 projects, may potentially be reported. This study cannot, however, associate responses with a specific school or individual participating in i3 independent evaluations, since we are not collecting that information.

Grantees and their independent evaluators are required to comply with the requirements of the NEi3. Participation in at least one of the NEi3’s data collection periods is required. Your response is critical for producing valid and reliable data. You may skip any questions you do not wish to answer; however, we hope that you answer as many questions as you can. Your answers to questions will not affect your grant—now or in the future. Participation in the NEi3 Findings Survey will not impose any risks to you as a respondent. If you have any questions about your rights as a research participant, you can contact Teresa Doksum, the Chairperson of Abt’s IRB at 617-349-2896 or by email at [email protected].

This survey is authorized by law:American Recovery and Reinvestment Act of 2009 (ARRA), Section 14007, Title XIV (Public Law (P.L.) 111-5.

We estimate that the survey will take you approximately 15 hours to complete.

OMB Clearance Number: xxxx-xxxx Expiration Date: xx/xx/xxxx

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is xxxx-xxxx. The time required to complete this information collection is estimated to average approximately 15 hours per respondent, including the time to review instructions, gather the data needed, and complete and review the information collected. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: U.S. Department of Education, Institute of Education Sciences, 555 New Jersey Avenue, N.W., Washington, DC 20208.

Overview of the NEi3 Data Collection Survey

Throughout the data collection survey, the NEi3 study team has pre-populated information available from the following sources:

• your analysis plan

• your contrast tool

• reports or other documents on interim or final results

Please review this information carefully to ensure its accuracy (you’ll be asked to actively confirm the accuracy of all pre-populated information). 

We realize that it will likely take you a few visits to this website to complete entering your findings into this data collection survey. Therefore, you may log into and out of the survey as many times as necessary to complete the entire data collection survey by MM/DD/YYYY. The information you enter into this survey will be automatically saved once you navigate to a different survey section. 

The data collection website is organized into three main sections:

1. Background Section

2. Fidelity of Implementation Section

3. Impact Study Section

After you enter the survey, you can access each of the above sections by clicking on the appropriate navigation button at the top of your screen. Below, we briefly describe the content of each of these sections. 

The Background Section asks you about the parties involved in developing, implementing and evaluating the i3-funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation. 

The Fidelity of Implementation Section asks you to describe the key components of the i3-funded intervention you evaluated, the measures constructed to evaluate fidelity of implementation for those components, and the findings from your implementation study. 

The Impact Study Section asks you to describe each of your evaluation’s confirmatory contrasts. Specifically, the survey will ask you to provide information about the results from the analysis for each contrast, including the estimated effects of the intervention and the size of the analysis sample. In addition to your confirmatory contrasts, you are welcome to provide the details of any additional exploratory contrasts from this study that you would like the NEi3 to consider including in the NEi3 Final Report. Detailed guidance about how to enter information in each column is available by hovering your mouse on the pertinent column header. This section is divided up into the following nine subsections:

• Contrast Description: Key Features

• Contrast Description: Design

• Contrast Description: Outcome Measurement

• Contrast Description: Baseline Measures

• Confounds

• Outcomes

• Assessing Attrition

• Baseline Measurement

• Reported Findings

The same contrasts appear on each subsection, but different items appear on each subsection. So for each contrast, you’ll be asked to enter/update descriptive information, and information on confounds, outcomes, attrition, baseline equivalence, and findings. 

Monitoring Progress and Submitting Your Results

As you complete each section (and subsection) of the survey, you will check off the “Section Complete” or “Subsection Complete” checkboxes that appear in each section (and subsection). That will allow you to track your own progress. Once you have indicated your completion of each section, check off the “Survey Complete” checkbox at the top of the survey. That will automatically notify the NEi3 AR team that you have completed the survey and it is ready for their review. 

Getting Help

For help with technical issues using this website, please call 617-520-2447. For help selecting appropriate data for entry and/or for further guidance on survey items, please call 301-347-5638.

Accessing the Survey

For each study there is a corresponding data collection survey. If your grant has only one study, then only one survey will appear on the survey selection page. If your grant has multiple studies, you’ll be able to open the desired survey on the survey selection page.

NEi3 Data Collection: Background Survey

Background Information

The following section asks you about the parties involved in developing, implementing and evaluating the i3-funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation.

1. What is the name of the grantee for which you conducted/are conducting the evaluation?

___________________________________________________________

2. What is the name of the grant under which you conducted/are conducting the evaluation?

________________________________________________________

3. What is the i3 grant type of this grant?

 Development  Scale-Up  Validation

4. How many impact studies/evaluations are funded under this grant? ______________________

5. If there are multiple impact studies/evaluations funded under this grant, what is the name of the study reported in this survey? ____________________________________________________________________________

6. What is/are the name(s) of the organization(s) that conducted/is conducting the impact study? ____________________________________________________________________________

7. What is/are the name(s) of the principal investigator(s) that conducted/is conducting the study? ____________________________________________________________________________

8. What is the name of the i3-funded intervention? ____________________________________________________________________________

9. Please review the following summary description of the i3-funded intervention, and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided. Please note that this description is intended for inclusion in the NEi3 Findings Report; however, the NEi3 may condense, or seek clarifications for, proposed revisions.

NEi3 Description: This section will be pre-populated by the NEi3 team with a summary of the intervention in sufficient detail to help readers understand what makes this intervention similar to or different from other interventions. The level of detail will be similar to what would be provided in an introductory section of a typical impact evaluation report. It will include the length of the intervention and the dosage, as well as information about the content, delivery, and implementation of the intervention.

 The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)

 The intervention description requires some revision. (If this box is checked, please enter a revision below.)

Evaluator Revision:

Please insert a revised description here if needed.

10. What are the names of the individual(s) and organization(s) that developed the intervention?

Individual	Organizational affiliation

11. What are the name(s) of the individuals and organization(s) that implemented and/or are implementing the intervention being evaluated under this grant?

Individual	Organizational affiliation

12. Did the following entities review the study findings prior to their release and/or submission to NEi3:

	Yes	No
The individual/organization implementing the intervention (from question 11)		
The individual/organization developing the intervention (from question 10)		
Participating School District(s)		
Participating School Principal(s)		
Other, please specify: ____________________________		

If you answered “yes” to any question above, please explain: ______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response.

13. Did the following entities provide comments on the study findings prior to their release and/or submission to NEi3?

	Yes	No
The individual/organization implementing the intervention (from question 11)		
The individual/organization developing the intervention (from question 10)		
Participating School District(s)		
Participating School Principal(s)		
Other, please specify: _________________________________		

If you answered “yes” to any question above, please explain how comments were addressed: ______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts

are implicated if warranted based on open-ended response.

14. For each entity below, did the entity request changes to the analysis and reporting prior to their release and/or submission to NEi3?

	Yes	No
The individual/organization implementing the intervention (from question 11)		
The individual/organization developing the intervention (from question 10)		
Participating School District(s)		
Participating School Principal(s)		
Other, please specify: _________________________________		

If you answered “yes” to any question above, please explain whether the requested changes were made, and why: ______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts

are implicated if warranted based on open-ended response.

15. For each of the following entities, did the entity conduct any of the statistical analysis for the findings submitted to NEi3?

	Yes	No
The individual/organization implementing the intervention (from question 11)		
The individual/organization developing the intervention (from question 10)		
Participating School District(s)		
Participating School Principal(s)		
Other, please specify: _________________________________		

If you answered “yes” to any question above, please explain: ______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response.

16. Please review the following description of the population served by this grant’s i3-funded intervention and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided.

NEi3 Description: This section will be prepopulated with a description of the population served by this grant’s i3-funded intervention(s). The description will include the target age/grade, or teacher type (for example, fifth grade teacher, elementary school counselor, high school science teacher, etc.), of the intervention. It will also include the location of the intervention and a description of the type and number of schools served.

 The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)

 The intervention description requires some revision. (If this box is checked, please enter a revision below.)

Evaluator Revision:

Please insert a revised description here if needed.

17. Please review the following description of the design of this study’s impact evaluation. If the description requires any revision, please insert a corrected description in the area provided.

NEi3 Description: This section will be prepopulated with a brief narrative description of key features of the study design. This description will include the design and level of assessment.

 This description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)

 This description requires some revision. (If this is checked, please enter a revision below.)

Evaluator Revision:

Please insert a revised description here if needed.

18. Please review the following description of how the study planned to select or assign treatment and comparison groups for the impact evaluation. Please update the description to reflect the selection or assignment process as implemented and to add details as needed. Information that needs to be added is indicated in capital letters.

NEi3 Description of Plans: This section will be prepopulated with the description from the Analysis Plan of the process by which treatment and comparison groups are assigned and/or identified. For RCTs, this description will include: how the sample for random assignment was selected (articulate all inclusion and exclusion criteria); the process for random assignment, and blocks or strata, and its timing; any changes to the analysis sample after the point of random assignment and reasons for sample loss or gain (e.g., identification of ineligible students; joiners); when clusters and individuals were first notified of random assignment; when consent was obtained. For non-experimental designs, this description will include: how the treatment and comparison samples were selected (articulate all inclusion and exclusion criteria); if matching was used, the dimensions on which the groups will be matched and the procedures for matching; whether the treatment sample is comprised of individuals who volunteered for the intervention; whether the comparison sample is comprised of individuals who were not eligible for or actively declined to participate in the intervention; and any differences in eligibility criteria between the treatment and comparison groups. If key information is not available in the Analysis Plan, the prepopulation will indicate this in capital letters.

Evaluator Description of Implementation:

Please insert a revised description here.

19. Please review the following description of the sample eligibility and inclusion for this study’s impact evaluation. If the description requires any revision, please insert a corrected description in the area provided.

NEi3 Description: [Describe how the districts/schools were chosen to be in the study (e.g. universe, random sample from a defined population, convenience sample , etc…). Please articulate all inclusion and exclusion criteria. For example, “The five largest school districts in XYZ County, Texas, were recruited to participate in this study because of their size, and because of the high percentage of ELL students in XYZ County. Within each of the five school districts, two to four elementary schools will be recruited and half will be randomly assigned to treatment and half will be assigned to control. The intervention is a school-wide intervention. All recruited schools will be included in the impact evaluation. Outcomes will be assessed for students in grades three through five. The primary research question concerns the impact on school average student test scores after the intervention has been in schools for three years.” Finally, please ensure it is clear in the summary to whom the sample was designed to generalize.]

 The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)

 The intervention description requires some revision. (If this box is checked, please enter a revision below.)

Evaluator Revision:

Please insert a revised description here if needed.

20. For both the full population served by this grant’s i3-funded intervention, and the sample of the population included in this study’s impact estimates, please provide the following statistics at the student-level if available and at the school-level if the student-level is not available:

	Full Population Served by this Grant’s i3-Funded Intervention	Sample of the Population Included in this Study’s Impact Estimates
Location
City	n_____ (___%)	n_____ (___%)
Suburban	n_____ (___%)	n_____ (___%)
Town	n_____ (___%)	n_____ (___%)
Rural	n_____ (___%)	n_____ (___%)
Demographics
American Ind/Alaskan	n_____ (___%)	n_____ (___%)
Asian/Pacific Islander	n_____ (___%)	n_____ (___%)
Black	n_____ (___%)	n_____ (___%)
Hispanic	n_____ (___%)	n_____ (___%)
White	n_____ (___%)	n_____ (___%)
Two or more races	n_____ (___%)	n_____ (___%)
Region
Northeast	n_____ (___%)	n_____ (___%)
Southeast	n_____ (___%)	n_____ (___%)
Midwest	n_____ (___%)	n_____ (___%)
Southwest	n_____ (___%)	n_____ (___%)
West	n_____ (___%)	n_____ (___%)
Grade Level
Before Kindergarten	n_____ (___%)	n_____ (___%)
K-5	n_____ (___%)	n_____ (___%)
6-8	n_____ (___%)	n_____ (___%)
9-12	n_____ (___%)	n_____ (___%)
School Ranking
Student attending Priority Schools/ Priority Schools	n_____ (___%)	n_____ (___%)
Students attending Focus Schools/Focus Schools	n_____ (___%)	n_____ (___%)
Other i3 Priority Groups
English Language Learners	n_____ (___%)	n_____ (___%)
Free/Reduced-Price Lunch	n_____ (___%)	n_____ (___%)
Overall Number	n_____	n_____
Please indicate the level at which the above data are reported:  student-level  school-level
School Characteristics
Average school size	n_____/school	n_____/school
Title I eligible school-wide	___ schools	___ schools
Title I eligible targeted	___ schools	___ schools
Overall Number	___ schools	___ schools

NEi3 Data Collection Survey: Fidelity Study Section Instructions

Please complete a row for each of the key components of the i3 intervention that is included in your logic model and fidelity measurement system. The following instructions define the data that should be entered in each of the columns.

Column	Column Header	Instructions for completing corresponding cells
A	Key Component	Please enter the name of the key component. Key components are the activities and inputs that are under the direct control of the individual or organization responsible for program implementation (e.g., program developer, grant recipient), and are considered by the developer to be essential for implementing the intervention.  Key components may include financial resources, professional development for teachers, curricular materials, or technology products.
B	Indicators Characterizing Key Component	Using your study’s logic model, design summary, and fidelity matrix as a guide (whenever applicable), please provide the definition of each key component, i.e., a list of the indicators that represent each key component. Indicators should be quantified where possible. Depending upon how you defined each key component, indicators might encompass a combination of activities at the teacher, classroom, school, or sample level. Example: Fidelity of implementation for the key component “professional development” is based on four professional development indicators: a one-week summer institute, five webinars, two coaching sessions, and ongoing PD.
C	Year 1: Component Level Threshold for Fidelity of Implementation for the Unit that is the Basis for the Sample-Level Representation of Fidelity (i.e. definition of implementation with fidelity)	Please provide the definition of implementation with fidelity for the unit that is the basis for the sample-level representation of fidelity in Year 1 for each key component; i.e., the component level threshold that delineates implementation with fidelity from implementation without fidelity for a particular sample unit. Example of unit-level: If the sample-level fidelity threshold is defined as the percent of schools that have received an adequate score on a key component, the unit that is the basis for the sample-level representation of fidelity is “school”. The definition of implementation with fidelity could be: “a single school will be considered to have implemented the key component with fidelity if at least 90% of teachers at the school: attended the one-week summer institute, attended four of five webinars, attended two of two coaching sessions, and received ongoing PD as needed (as self-reported in teacher survey)."
D	Year 1: Sample Size (of Units Measured at the Sample Level)	Please enter the sample size (N) of units associated with the sample-level threshold and results. For example, if sample-level threshold is 80% or more schools are “high” implementers, please enter the number of schools.
E	Year 1: Representativeness of sample: Measured on all (A), some (S), or none (N) of the members representing the intervention group in the impact evaluation	Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.
F	Year 1: Component Level Threshold for Fidelity of Implementation at the Sample Level (i.e. definition of implementation with fidelity)	Please provide the definition of implementation with fidelity at the sample-level in Year 1 for each key component; i.e., the component level threshold that delineates implementation with fidelity from implementation without fidelity across sample units. Examples of sample-level: “At least 80% of schools implement with fidelity” or “60% or more of districts implement with fidelity”. The sample-level threshold (e.g., across schools) should be a summary measure of fidelity score ranges and thresholds established at level of the unit that is the basis for the sample-level representation of fidelity (e.g., teacher, classroom, school).
G	Year 1: Component Level Fidelity Score for the Entire Sample	Please report the component level fidelity scores achieved in Year 1 of the intervention for each key component. Examples: “85% of schools implemented with fidelity” or “70% of districts implemented with fidelity”
H	Year 1: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)	Note: This field is automatically calculated based on the two prior fields.
I	Year 2: Component Level Threshold for Fidelity of Implementation for the Unit that is the Basis for the Sample-Level Representation of Fidelity (i.e. definition of implementation with fidelity)	Same instructions as in column C will appear in survey instrument, replacing year one with year two.
J	Year 2: Sample Size (of Units Measured at the Sample Level	Same instructions as in column D will appear in survey instrument.
K	Year 2: Representativeness of sample: Measured on all (A), some (S), or none (N) of the members representing the intervention group in the impact evaluation	Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.
L	Year 2: Component Level Threshold for Fidelity of Implementation at the Sample Level (i.e. definition of implementation with fidelity)	Same instructions as in column E will appear in survey instrument, replacing year one with year two.
M	Year 2: Component Level Fidelity Score for the Entire Sample	Same instructions as in column F will appear in survey instrument, replacing year one with year two.
N	Year 2: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)	Same instructions as in column G will appear in survey instrument.
O	Year 3: Component Level Threshold for Fidelity of Implementation for the Unit that is the Basis for the Sample-Level Representation of Fidelity (i.e. definition of implementation with fidelity)	Same instructions as in column C will appear in survey instrument, replacing year one with year three.
P	Year 3: Sample Size (of Units Measured at the Sample Level	Same instructions as in column D will appear in survey instrument.
Q	Year 3:Representativeness of sample: Measured on all (A), some (S), or none (N) of the members representing the intervention group in the impact evaluation	Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.
R	Year 3: Component Level Threshold for Fidelity of Implementation at the Sample Level (i.e. definition of implementation with fidelity)	Same instructions as in column E will appear in survey instrument, replacing year one with year three.
S	Year 3: Component Level Fidelity Score for the Entire Sample	Same instructions as in column F will appear in survey instrument, replacing year one with year three.
T	Year 3: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)	Same instructions as in column G will appear in survey instrument.
U	Year 4: Component Level Threshold for Fidelity of Implementation for the Unit that is the Basis for the Sample-Level Representation of Fidelity (i.e. definition of implementation with fidelity)	Same instructions as in column C will appear in survey instrument, replacing year one with year four.
V	Year 4: Sample Size (of Units Measured at the Sample Level	Same instructions as in column D will appear in survey instrument.
W	Year 4: Representativeness of sample: Measured on all (A), some (S), or none (N) of the members representing the intervention group in the impact evaluation	Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.
X	Year 4: Component Level Threshold for Fidelity of Implementation at the Sample Level (i.e. definition of implementation with fidelity)	Same instructions as in column E will appear in survey instrument, replacing year one with year four.
Y	Year 4: Component Level Fidelity Score for the Entire Sample	Same instructions as in column F will appear in survey instrument, replacing year one with year four.
Z	Year 4: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)	Same instructions as in column G will appear in survey instrument.
AA	Year 5: Component Level Threshold for Fidelity of Implementation for the Unit that is the Basis for the Sample-Level Representation of Fidelity (i.e. definition of implementation with fidelity)	Same instructions as in column C will appear in survey instrument, replacing year one with year five.
AB	Year 5: Sample Size (of Units Measured at the Sample Level	Same instructions as in column D will appear in survey instrument.
AC	Year 5: Representativeness of sample: Measured on all (A), some (S), or none (N) of the members representing the intervention group in the impact evaluation	Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.
AD	Year 5: Component Level Threshold for Fidelity of Implementation at the Sample Level (i.e. definition of implementation with fidelity)	Same instructions as in column E will appear in survey instrument, replacing year one with year five.
AE	Year 5: Component Level Fidelity Score for the Entire Sample	Same instructions as in column F will appear in survey instrument, replacing year one with year five.
AF	Year 5: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)	Same instructions as in column G will appear in survey instrument.
Scale-Up Key Components (Scale-Up grants only)
AG	Key component / activities related to scale-up	Please enter each key component necessary to support scale-up efforts separately.
Scale-Up Goals (Scale-Up grants only)
AH	Goal	Please enter scale-up goals separately. Scale-up goals should be quantified and measurable, e.g., 500 additional teachers trained.
AI	Results at grant end	Please enter the results of each scale-up goal achieved by the end of the grant period, e.g., If 523 additional teachers were trained, enter “523” in the field
AJ	Goal met	Please enter a final determination (yes/no) indicating whether each scale-up goal was met by grant end.
AK	Results Year 1	Optional: if you choose, please enter scale-up results for each goal at the end of each or any year(s) of implementation. For example, if scale-up goals were measured in years 3 and 4, enter results for each component in those fields and enter N/A (or leave blank) fields associated with years 1, 2, and 5.
AL	Results Year 2
AM	Results Year 3
AN	Results Year 4
AO	Results Year 5

NEi3 Data Collection Survey: Impact Study Section Instructions

Please complete a row for each contrast measured by your impact study. The following instructions define the data that should be entered in each of the columns.

Column	Column Header	Instructions for completing corresponding cells (for each contrast)	INTERNAL USE ONLY: Programming notes about skip patterns
Contrast Description
A	Contrast Name	Please enter the names of each of the evaluation’s confirmatory and exploratory contrasts. This field does not affect the assessment or reporting of the contrast; the contrast name simply facilities communication between the evaluator and the NEi3 AR Team, and facilitates referencing contrasts between the data-entry fields of this survey. The contrast name should uniquely identify the contrast and differentiate among similar contrasts: e.g. “Math Achievement-confirmatory” and “Math Achievement-cohort 1” only might be the names of two similar contrasts.	Field is required.
Contrast Description: Key features that define a contrast
B	Confirmatory or Exploratory?	Indicate whether this contrast is confirmatory or exploratory. Please note that any pre-specified contrasts that were registered as confirmatory will be pre-populated as such and cannot be changed to be exploratory. All other contrasts will be listed as exploratory.	Field is required. Only AR team members can edit this field. Default value for new contrasts is "exploratory".
C	Contrast Status	For each contrast, please select from among the following options: - Contrast Confirmed if the effectiveness findings from this contrast are reported in this survey - Not Reported if the effectiveness findings from this contrast are not reported in this survey and will not be reported to the NEi3 in the future - Pending Additional Data if the effectiveness findings from this contrast will be reported in future waves of data collection. If you select this option, please provide the date when you plan to report effectiveness findings from this contrast in the next field.	Field is always applicable. Drop down menu: - Contrast confirmed - Pending Additional Data - Not reported
D	Expected Reporting Date	If effectiveness findings for this contrast are not yet available, please provide the date when you plan to report effectiveness findings. (Not applicable for contrasts for which data is provided in this survey or for contrasts for which findings will never be reported.)	If the contrast status is NOT "pending additional data", display "Not Applicable".
E	Contrast Notes	This field is not required, but is available to provide contextual information or notes about each specific contrast as needed.	Field is always applicable.
F	Outcome Domain	For confirmatory contrasts, this field is populated with the registered domain name of your pre-specified confirmatory contrast. As the domain is a defining feature of a contrast, it cannot be edited by the evaluator. Contact the helpdesk to correct typos or other minor errors. For exploratory contrasts, please enter the name of the domains measured. If the exploratory contrast measures the same construct as one of the confirmatory contrasts, please use the same domain name.	Field is always applicable. Field cannot be edited for confirmatory contrasts.
G	Intervention Condition	For confirmatory contrasts, this field is populated with the registered description of the intervention tested by the pre-specified confirmatory contrast. As the intervention condition is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please name the intervention(s) experienced by the treatment group in the study. Do not use acronyms. If the condition is the same as the one tested by a confirmatory contrast, please use the same name. If the contrast involves comparing one treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group and the comparison group.	Field is always applicable. Field cannot be edited for confirmatory contrasts.
H	Comparison Condition	For confirmatory contrasts, this field is populated with the registered description of the counterfactual condition in the pre-specified confirmatory contrast. As the intervention condition is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please name the intervention(s) experienced by the comparison group in the study. This may be the name of another intervention or Business as Usual, if no specific intervention. Do not use acronyms. If the condition is the same as the one tested by a confirmatory contrast, please use the same name. If the contrast involves comparing one treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group and the comparison group.	Field is always applicable. Field cannot be edited for confirmatory contrasts.
I	Educational Level	For confirmatory contrasts, this field is populated with the educational level that was registered in the pre-specified confirmatory contrast. As the educational level is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please report the educational level (e.g. elementary, middle or high school) of the students that experience the intervention. For teacher outcomes, report the educational level of the students in the teachers’ classrooms.	Field is always applicable. Field cannot be edited for confirmatory contrasts.
Contrast Description: Approach to addressing contrast
J	Design	Please select one of the following designs: RCT, RDD, QED, Comparison Group Design Without a Pretest, ITS, Pre-post. For confirmatory contrasts, this field is populated with the registered design for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. Definitions: Randomized controlled trial (RCT). Under this design, an evaluation uses a random process to assign study units to an intervention groups and to a control group, and measures the effect of the intervention by comparing the outcomes of units assigned to the two groups. Regression discontinuity design (RDD). Under this design, an evaluation would use a continuous “scoring” rule to assign study units to the intervention. Units with scores above a pre-set cutoff value are assigned to the treatment group and units with scores below the cutoff value are assigned to the comparison group, or vice versa. The evaluation then compares the outcomes of the two groups close to the cutoff to measure the effect of the intervention. Quasi-Experimental Design (QED). A QED compares outcomes for intervention participants with outcomes for a comparison group chosen through methods other than randomization (RCT) or a scoring rule with a well-defined threshold (RDD) and includes one or more pre-intervention measures. Comparative Interrupted Time Series designs are treated as QEDs by the NEi3. Comparison Group Design Without a Pretest. A comparison group design without a pre-test compares outcomes for intervention participants with outcomes for a comparison group chosen through methods other than randomization (RCT) or a scoring rule with a well-defined threshold (RDD) and does not include pre-intervention observations. Interrupted Time Series Design (ITS). An ITS design uses observations at multiple points in time both before and after the intervention was implemented, and compares the outcomes after the intervention with the trend established prior to the intervention. These designs do not use a comparison group. Pre-post Design. A pre-post design compares outcomes for intervention participants after receiving the intervention to their outcomes before receiving the intervention. These designs do not use a comparison group.	Field is always applicable. Drop down menu: -RCT -RDD -QED -Comparison Group Design without a Pretest -ITS -Pre-post For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
K	Unit of Assignment	For each contrast, please report the unit of assignment or the level at which assignment to the intervention and comparison conditions occur (e.g., student, teacher, classroom, or school). If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear.	Field is always applicable. Drop down menu: - District - School - Teacher - Classroom - Student - Other (contact helpdesk to enter) For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
L	Intervention Group [alternately, “Treatment Group”]: Description	For confirmatory contrasts, this cell is prepopulated with the registered description of the intervention group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe which schools, teachers and/or students will be included in the intervention (“treatment”) group for this analysis. Example 1 (Candy Valley): “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the intervention.” Example 2 (Apple Valley): “This contrast pools data across all treated cohorts and schools. All students in treated grades in Fuji Elementary (K-2), Granny Smith Elementary (K-1) and Red Delicious Elementary (K-2) who were present for data collection in Spring 2014 are included in the treatment group.”	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
M	Age/Grade during the Intervention	For confirmatory contrasts, this cell is prepopulated with the registered description of the age/grade of the students in the intervention group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe the age or grade-level of students in the intervention group during their exposure to the intervention. For teacher outcomes, please enter the grade(s) taught by the teachers. Example 1 (Candy Valley): “The intervention was fielded to first grade students.” Example 2 (Apple Valley): “Both Fuji and Red Delicious Elementary Schools introduced the intervention in the 2011-2012 school year to all kindergarten students, offered the intervention to all kindergarten and first graders in 2012-2013 and to kindergarten through second graders in 2013-2014. Granny Smith Elementary introduced the intervention in the 2012-2013 school year to all kindergarten students and offered the intervention to all kindergarten and first graders in 2013-2014.”	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
N	Length of Student Exposure to Intervention	For confirmatory contrasts, this cell is prepopulated with the registered description of the age/grade of the students in the intervention group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe how long (no. of semesters/years) the students in the intervention group were engaged in the intervention at the time the outcome was measured for this contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. Example 1 (Candy Valley): “Students received one year of treatment” Example 2 (Apple Valley): “When outcomes were measured in Spring 2014, both Fuji and Red Delicious Elementary Schools students had the following pattern of exposure to treatment: second grade students had three years of treatment, first grade students had two years of treatment and kindergarten students had one year of treatment. When outcomes were measured in Spring 2014, Granny Smith Elementary first graders had two years of treatment and kindergarten students had one year of treatment.”	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
O	Comparison Group: Description	For confirmatory contrasts, this cell is prepopulated with the registered description of the comparison group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe which schools, teachers and/or students will be included in the comparison group for this analysis. Example 1 (Candy Valley): “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the control group.” Example 2 (Apple Valley): “This contrast pools data across all cohorts and schools matched to treated cohorts and schools. K-2 students from Honey Crisp Elementary (comparison for Fuji Elementary), K-1 students from Ginger Gold Elementary serves (comparison for Granny Smith Elementary), and K-2 students Jonagold Elementary (comparison for Red Delicious Elementary) who were present for data collection in Spring 2014 are included in the treatment group.”	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Contrast Description: Outcome Measurement
P	Outcome or Instrument Name (full names, not acronyms)	For confirmatory contrasts, this cell is prepopulated with the registered outcome measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the full name of the outcome measure, including the name of the subtest or subscale if applicable. If the instrument for the outcome measure is not used in one of the confirmatory contrasts, please describe (1) what the instrument is designed to measure and (2) how the outcome measure is constructed.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Q	Construction of Outcome Measure (binary, ordinal, or continuous)	For confirmatory contrasts, this cell is prepopulated with the registered scale of the outcome measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please indicate whether the outcome measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
R	Unit of Observation for Outcome Measure	For confirmatory contrasts, this cell is prepopulated with the registered unit of observation for the outcome measure—i.e. the units for which outcomes are observed in the analysis data. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the unit for which outcomes are observed in your data (e.g. student, teacher, classroom, school, district). Example: If your data are school-level averages of student test scores, enter “School”.	Field is always applicable. Drop down menu: - District - School - Teacher - Classroom - Student - Other (contact helpdesk to enter) For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
S	Timing of Outcome Measure (year, Fall/Spring) for the Intervention Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report when during the calendar year the outcome was measured.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
T	Timing of Outcome Measure (age/grade) for the Intervention Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the age or grade level for which the outcome was measured.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
U	Timing of Outcome Measure (year, Fall/Spring) for the Comparison Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report when during the calendar year the outcome was measured.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
V	Timing of Outcome Measure (age/grade) for the Comparison Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the age or grade level for which the outcome was measured.	Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Contrast Description: Baseline Measures
W	Key Baseline Measures	For confirmatory contrasts, this cell is prepopulated with the registered baseline measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the baseline variable(s) on which you are establishing baseline equivalence. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
X	Construction of Baseline Measure (binary, ordinal, or continuous)	For confirmatory contrasts, this cell is prepopulated with the registered scale of the baseline measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please indicate whether the baseline measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Y	Unit of Observation for Baseline Measure	For confirmatory contrasts, this cell is prepopulated with the registered unit of observation for the baseline measure—i.e. the units for which outcomes are observed in the analysis data. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the unit for which baseline measures are observed in your data (e.g. student, teacher, classroom, school, district). Example: If your data are school-level averages of student test scores, enter “School”. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Z	Timing of Baseline Measure (year, Fall/Spring,) for the Intervention Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. For exploratory contrasts and for confirmatory contrasts you wish to change, please report when during the calendar year the data you will use to test baseline equivalence was measured. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
AA	Timing of Baseline Measure (age/grade) for the Intervention Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the age or grade level for which the data you will use to test baseline equivalence was measured. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
AB	Timing of Baseline Measure (year, Fall/Spring) for the Comparison Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report when during the calendar year the data you will use to test baseline equivalence was measured. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
AC	Timing of Baseline Measure (age/grade) for the Comparison Group	For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report when the data you will use to test baseline equivalence was measured. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
Confounds
AD	Intervention Condition: Other Activities	To your knowledge, were any other interventions (other than your project’s i3-funded intervention) targeting the outcome domain listed in [C] that were offered to the intervention group but not the comparison group? If so, please describe the other intervention.	Field is always applicable.
AE	Fundamental Confound at the District Level: Intervention Condition	Does the intervention condition include schools/teachers/students from a single school district which does not also include comparison group members?	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”.
AF	Fundamental Confound at the School Level: Intervention Condition	Does the intervention condition include teachers/students from a single school which does not also include comparison group members?	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”.
AG	Comparison Condition: Other Activities	To your knowledge, were any other interventions (other than your project’s i3-funded intervention) targeting the outcome domain listed in [C] that were offered to the comparison group but not the intervention group? If so, please describe the other intervention.	Field is always applicable.
AH	Fundamental Confound at the District Level: Comparison Condition	Does the comparison condition include schools/teachers/students from a single school district which does not also include intervention group members?	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”.
AI	Fundamental Confound at the School Level: Comparison Condition	Does the comparison condition include teachers/students from a single school which does not also include intervention group members?	If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”.
Outcomes
AJ	Outcome or Instrument Name (full names, not acronyms)	This field displays the outcome measure you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Outcome Measurement tab.	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Outcome Measurement"
AK	Who Provided the Outcome Data?	For each contrast, please report the name of the entity that provided the outcome data.	Field is always applicable.
AL	Value and Type of Reliability Measure of Instrument for Outcome Data	Please provide a narrative description of the outcome and discuss the reliability and validity of the measure, including the value and type of reliability measure. For example, you may report an internal consistency, test-retest reliability, or inter-rater reliability measure. For outcomes for which reliability statistics cannot be obtained, please describe why reliability statistics are not available. Example 1: “The Orange Gulch District writing assessment was scored by trained reviewers according to a holistic rubric. A random subset of essays was double scored and demonstrated an inter-rater reliability of 89%.” Example 2: “Attendance is the proportion of school days for which an individual student was present for at least one course at Orange Gulch High School. Reliability statistics cannot be calculated for attendance. However, it is a face valid measure of the pre-specified domain: High School Attendance.”	Field is always applicable.
AM	Consistency of Collection	Please describe any differences in the process for collecting data on the outcome measure between the groups. To be consistently collected, an outcome measure must be collected using the same rules or procedures in both groups. For example, data would not be consistently collected if intervention students take the post-test in January, but comparison students take the test in March – two months later.	Field is always applicable.
AN	Treatment of Missing Data	Describe how the study addressed missing values for the outcome measure.	Field is always applicable.
Assessing Attrition
AO	Unit of Assignment	This field displays the unit of assignment you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Design tab. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Design"
AP	Unit of Observation for Outcome Measure	This field displays the unit of observation for outcome entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Outcome Measurement tab. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Outcome Measurement"
AQ	Cluster Randomized Design?	Select “Yes” if the contrast is from a cluster randomized design. Select “No”, otherwise For NEi3, all cluster randomized designs are those in which clusters (of individuals) were assigned by a process that was functionally random. These clusters may be associated with, for instance, particular teachers, classrooms, schools, or districts. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Drop down menu: - Yes - No - Not Applicable
AR	Analytic Sample Includes Joiners?	Select “not applicable” if your design is not a cluster randomized design. Select “yes” if your analytic sample includes individuals who were not present in the cluster at the time of random assignment. Select “no” if your analytic sample is restricted to individuals who were present at the time of random assignment. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT a cluster randomized design, display "Not Applicable". Drop down menu: - Yes - No - Not Applicable
AS	Description of Joiners	Provide a narrative description of the joiners in your analytic sample. Include information on when they joined the sample relative to a) the point of random assignment and b) the point at which clusters were notified of their assignment status. Example 1: The analysis sample includes all students in study schools in Year 3. Some students may have been transferred into study schools after the point of random assignment (Year 1) and may not have two full years of exposure. Example 2: The student sample is comprised of all students in study teachers’ classrooms in Year 2 of the study. Because teacher rosters were not set at the point of random assignment (Year 1), all students in Year 2 are considered joiners. (Not applicable if your sample does not include joiners.)	If the contrast does NOT include joiners, display "Not Applicable".
AT	Intervention Group: Units of Assignment Randomized	Enter the number of clusters randomly assigned to the intervention group. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
AU	Intervention Group: Reasons for Loss of Units of Assignment	List reasons for sample loss separated by a carriage return. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
AV	Intervention Group: Units of Assignment Lost (for each reason)	For each reason for sample loss, list the number of units lost. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
AW	Intervention Group: Units of Assignment in Analytic Sample	Enter the number of clusters for the intervention group for the analytic sample. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
AX	Intervention Group: Units of Observation Randomized or in Randomized Cluster	Enter the sample size for the intervention group at the time of random assignment Complete this field the contrast is from: a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners any randomized design where the unit of analysis is the same as the unit of assignment (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
AY	Intervention Group: Units of Observation in Remaining Clusters	Enter the sample size for the intervention group at the time of random assignment in clusters that appear in the analytic sample. This excludes units of observation for which the entire cluster left the sample. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students at the time of random assignment in schools that did not attrit This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
AZ	Intervention Group: Reasons for Loss of Units of Observation	List reasons for sample loss separated by a carriage return. This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BA	Intervention Group: Units of Observation Lost (for each reason)	For each reason for sample loss, list the number of units lost. This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BB	Intervention Group: Units of Observation in Analytic Sample	Enter the sample size for the intervention group for the analytic sample This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BC	Comparison Group: Units of Assignment Randomized	Enter the number of clusters randomly assigned to the comparison group. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
BD	Comparison Group: Reasons for Loss of Units of Assignment	List reasons for sample loss separated by a carriage return. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
BE	Comparison Group: Units of Assignment Lost (for each reason)	For each reason for sample loss, list the number of units lost. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
BF	Comparison Group: Units of Assignment in Analytic Sample	Enter the number of clusters for the comparison group for the analytic sample. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
BG	Comparison Group: Units of Observation Randomized or in Randomized Cluster	Enter the sample size for the comparison group at the time of random assignment. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students in all schools at the time of random assignment This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BH	Comparison Group: Units of Observation in Remaining Clusters	Enter the sample size for the comparison group at the time of random assignment in clusters that appear in the analytic sample. This excludes units of observation for which the entire cluster left the sample. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students at the time of random assignment in schools that did not attrit This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BI	Comparison Group: Reasons for Loss of Units of Observation	List reasons for sample loss separated by a carriage return. This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BJ	Comparison Group: Units of Observation Lost (for each reason)	For each reason for sample loss, list the number of units lost. This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
BK	Comparison Group: Units of Observation in Analytic Sample	Enter the sample size for the comparison group for the analytic sample This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".
Baseline Measurement
BL	Key Baseline Measures	This field displays the key baseline measure(s) you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"
BM	Construction of Baseline Measure (binary, ordinal, or continuous)	This field displays the description of the construction of the baseline measure you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"
BN	Unit of Observation for Baseline Measure	This field displays the level of observation for the baseline measure you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"
BO	Treatment of Missing Data for Baseline Balance Testing	Describe how the study addressed missing values for the baseline measure for baseline balance testing.	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BP	Narrative Description of Sample for Baseline Balance Testing	Describe the eligibility criteria for inclusion in the sample for baseline balance testing. If balance is established using cluster-level averages (e.g. school-level average student test score), describe which individual units are included in the average.
BQ	Type of Information Used to Assess Baseline Equivalence	Select one option that best represents the information generated by your baseline balance testing: Unadjusted Mean and Standard Deviation T-statistic Dichotomous T/C Difference This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable” (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BR	Narrative Description of Analysis for Baseline Balance Testing	Provide a brief narrative description of the method used to assess baseline balance. Example 1: Baseline summary statistics for both groups in the analytic sample were calculated. Data are from unadjusted means and standard deviations. Example 2: Differences in baseline means were calculated using a statistical model with the same structural components as the statistical model that was used to estimate impacts on the outcome variable for the same contrast. This is a two level model with students nested in schools. The pretest variable is the independent variable. The only covariates included in the model are an intercept term and a school level treatment indicator.	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BS	Baseline Balance Testing Sample Size: Intervention Group	Enter the sample size for the intervention group for baseline balance testing (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BT	Mean of Baseline Measure: Intervention Group	Enter the pre-intervention mean for the intervention group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BU	Standard Deviation of Baseline Measure: Intervention Group	Enter the pre-intervention standard deviation for the intervention group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BV	Baseline Balance Testing Sample Size: Comparison Group	Enter the sample size for the comparison group for baseline balance testing (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BW	Mean of Baseline Measure: Comparison Group	Enter the pre-intervention mean for the comparison group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BX	Standard Deviation of Baseline Measure: Comparison Group	Enter the pre-intervention standard deviation for the comparison group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BY	Baseline Intervention-Comparison Difference	Enter the difference between the intervention and comparison group means.	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BZ	T-Statistic for Baseline Balance Test	Enter the t-statistic for the pre-intervention difference in means between the treatment and the comparison (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)	If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
Reported Findings
CA	Type of Findings Reported	Select one of the following types of data to use to report findings: Unadjusted means Adjusted means, e.g. ANOVA T-test Coefficient from a linear regression model, e.g. OLS, HLM, etc. Coefficient from a non-linear regression model, e.g. logit or probit This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable”	Field is always applicable. Drop down menu: - Adjusted means, e.g. ANOVA - T-test - Coefficient from a linear regression model, e.g. OLS, HLM, etc. - Coefficient from a non-linear regression model, e.g. logit or probit
CB	Narrative Description of Analysis for Impact Evaluation	Provide a brief narrative description of the method used to estimate impact. Include information that describes: - the nested or longitudinal structure of the model - any adjustments for clustering of standard errors - if the outcome is a binary or categorical measure and, if so, what kind of estimation is performed - whether the pre-test variable is included and the level of inclusion - the level at which the treatment variable enters the model Example 1: Impact estimates come from a two-level regression model with students nested in schools. The pretest variable is included as a student-level covariate. Treatment enters the model at the school level. Example 2: We use logistic regression to estimate the impact of the intervention on attendance: a binary measure defined for each student. Estimates are log-odds-ratios. Standard errors are clustered at the school-level.	Field is always applicable.
CC	Intervention Group: Sample Size	Please report the size of the intervention group sample for the analysis.	Field is always applicable.
CD	Intervention Group: Number of Clusters in Analytic Sample	Please report the number of clusters assigned to the intervention group that appear in the analysis sample. Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned.	If the unit of observation for outcome measure is the same as the unit of assignment, display "Not Required".
CE	Comparison Group: Sample Size	Please report the size of the comparison group sample for the analysis.	Field is always applicable.
CF	Comparison Group: Number of Clusters in Analytic Sample	Please report the number of clusters assigned to the comparison group that appear in the analysis sample. Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned.	If the unit of observation for outcome measure is the same as the unit of assignment, display "Not Required".
CG	Intraclass Correlation Coefficient	Please report the intraclass correlation coefficient to describe the similarity of units at the level of observation within the cluster at the level of assignment Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment, e.g. the correlation between outcomes of students within the same school, if schools were assigned.	If the unit of observation for outcome measure is the same as the unit of assignment, display "Not Required".
CH	Unadjusted Outcome Mean for the Intervention Group	Please provide the unadjusted mean of the outcome measure in the intervention group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.)	If the type of findings is NOT unadjusted means, display "Not Required".
CI	Adjusted Outcome Mean for the Intervention Group	Please provide the adjusted mean of the outcome measure in the intervention group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.)	If the type of findings is NOT adjusted means, display "Not Required".
CJ	Unadjusted Standard Deviation in the Outcome Measure of the Intervention Group	Please provide the unadjusted standard deviation of the outcome measure in the intervention group.	Field is always applicable.
CK	Unadjusted Outcome Mean for the Comparison Group	Please provide the unadjusted mean of the outcome measure in the comparison group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.)	If the type of findings is NOT unadjusted means, display "Not Required".
CL	Adjusted Outcome Mean for the Comparison Group	Please provide the unadjusted mean of the outcome measure in the comparison group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.)	If the type of findings is NOT adjusted means, display "Not Required".
CM	Unadjusted Standard Deviation in the Outcome Measure of the Comparison Group	Please provide the unadjusted standard deviation of the outcome measure in the comparison group.	Field is always applicable.
CN	Impact Estimate (e.g., coefficient from OLS or HLM)	Please report the regression coefficient that estimates of the effect of the intervention. If your impact estimate is the sum of multiple regression coefficients, please enter this sum here. (You do not need to complete this field if it is automatically populated with “Not applicable”.)	If the type of findings is NOT a coefficient from a linear or non-linear regression model, display "Not Required".
CO	Standard Error of Impact Estimate	Please insert the estimated standard error of the regression coefficient here. If your impact estimate is the sum of multiple coefficients, please enter the standard deviation of the summed variable. (You do not need to complete this field if it is automatically populated with “Not applicable”.)	If the type of findings is unadjusted means, display "Not Required".
CP	Two-tailed p-value	Please insert the associated p-value that has not been adjusted for multiple comparisons here. If your impact estimate is the sum of multiple coefficients, please enter the associated p-value here. (You do not need to complete this field if it is automatically populated with “Not applicable”.)	If the type of findings is unadjusted means, display "Not Required".
CQ	Degrees of Freedom	Please enter the degrees of freedom of your statistical analysis. (You do not need to complete this field if it is automatically populated with “Not applicable”.)	If the type of findings is unadjusted means, display "Not Required".