Description of Revisions

Date: September 2, 2015

To: OMB

From: CNCS

Subject: School Turnaround AmeriCorps National Evaluation: Update on Study Design and

Proposed Modifications to Year 2 Data Collection Instruments and Incentives

The Corporation for National and Community Service (CNCS) submits this memorandum to update OMB on the School Turnaround AmeriCorps National Evaluation including some recommended changes from Year 1 to Year 2. The changes include modifications to the:

• Evaluation design to address unanticipated difficulties in forming a valid comparison group;

• Research questions to enable the Year 2 evaluation to build on findings from Year 1; and

• Data collection instruments and incentive structure so that the instruments better align with the modified design and research questions and the incentives help ensure more successful school recruitment and improve data collection response rates.

This memo summarizes the reasons for the proposed changes, which are based on both the findings that emerged from the Year 1 Evaluation and the evaluation contractor’s experience in administering the Year 1 data collection instruments. The memo also details the activities from Year 1 that will be modified or removed in Year 2. The proposed changes described in this memo constitute a net reduction in the data collection burden and represent minimal changes to the substantive focus of the Year 1 data collection instruments that were approved under OMB control number 3045-0164 for the information collection that expires 11/30/2017. Consequently, CNCS requests from OMB an expedited review of the research design changes and the proposed modifications to the data collection instruments and incentives for Year 2 of the School Turnaround AmeriCorps National Evaluation.

Background

To review, the Corporation for National and Community Service (CNCS) and the U.S. Department of Education (ED) are collaborating on a new grant program to increase high school graduation, college readiness, and educational attainment for students in our nation’s persistently lowest-achieving schools. Since fall 2013, School Turnaround AmeriCorps has been providing AmeriCorps grants to eligible organizations that work with schools receiving School Improvement Grant (SIG) or Priority school funding.¹ The grants are designed to improve academic outcomes for disadvantaged children in SIG-funded schools as the schools implement improvement strategies. The SIG schools must implement one of four school intervention models—turnaround, transformation, restart, or school closure—or Priority schools can implement interventions aligned with the Elementary and Secondary Education Act (ESEA) flexibility turnaround principles. The School Turnaround AmeriCorps National Evaluation is designed to understand the value-added of AmeriCorps members who provide direct services in low-performing schools above and beyond the school turnaround resources already invested in these schools, and to describe the mechanisms by which this happens.

Modifications to Evaluation Design

Year 1 of the School Turnaround AmeriCorps National Evaluation yielded multiple inter-related insights about the program; it also generated a number of useful lessons about the evaluation itself. These lessons encompass elements of study design, program design and operations, and the practical realities of primary data collection in dynamic contexts. The study design changed over the course of Year 1 to accommodate updated information about both program and comparison schools.

Originally, the Year 1 evaluation featured a quasi-experimental design within which the study would compare the full sample of 57 School Turnaround AmeriCorps schools and a similar number of comparison schools without School Turnaround AmeriCorps, using surveys and interviews to compare outcomes across the two groups of schools.² The design called for comparison schools that met three eligibility conditions: 1) program and comparison schools were located within the same district and/or state, 2) the comparison school offers the program school’s relevant grades, and 3) comparison schools had fewer AmeriCorps members than School Turnaround AmeriCorps campuses.

To start, there were very few potential comparison schools available for the evaluation. Such schools first needed to come from the lowest-performing five percent in each state, which limited the pool. In the course of recruitment, the evaluation contractor next discovered that many of the eligible comparison schools had a significant AmeriCorps presence through other programs, thus making them unsuitable for inclusion in this evaluation. Furthermore, the remaining eligible schools (and their staff) were already committed to participating in a multitude of activities (both programmatic and research-focused) leading to either refusal to participate in the Year 1 Evaluation or the evaluation being a lesser priority relative to schools’ other mandates (which was reflected in the length of time required to obtain local IRB approvals, to recruit schools and individual study participants, and to schedule and complete data collection within the school calendar, with its multiple testing, vacation, and other scheduling constraints). These challenges resulted in a substantially smaller number of eligible comparison schools (only 11) that agreed to be part of the Year 1 Evaluation. Administering surveys to a reduced group of comparison schools would have resulted in a severely under-powered study unlikely to find any differences between AmeriCorps schools and comparison schools.

To address the Year 1 challenges, CNCS and Abt Associates, the evaluation contractor, shifted the focus of the evaluation to a comparative case study design, which involved case studies of six pairs of AmeriCorps and matched comparison schools. The case study design facilitated more detailed comparisons of turnaround models in SIG and Priority schools with AmeriCorps members to a similar group of schools with little or no AmeriCorps presence. The approved interview and focus group protocols were adapted and used for case study data collection during the spring of 2015; each case study protocol was used to collect information from less than nine respondents. Surveys were not administered to teachers and principals in comparison schools and instead these resources were reallocated to the case study data collection, resulting in a net reduction in the data collection burden during Year 1 compared to the estimated annual burden provided in Part A of the OMB justification for the study.

The Year 2 Evaluation will continue this focus. The design emphasizes primary data collection from diverse stakeholder groups, including comparative case studies and surveys of grantee staff and program school leaders, to examine implementation and perceptions from multiple perspectives. As in Year 1, primary data will be supplemented by secondary data on grantee performance and member activities. During Year 2, the contractor will recruit another six matched pairs of AmeriCorps and comparison schools for case studies (12 matched schools in total), and follow up with the sample of Year 1 case study schools. Further, the Year 2 Evaluation will also include case studies of four School Turnaround AmeriCorps schools that have successfully exited SIG/Priority status to understand the extent to which School Turnaround AmeriCorps resources contributed to their ability to improve and succeed in exiting.

Expanding the sample of case study schools will increase the representation of grantee organizations and ensure selection on key criteria to describe the full range of the portfolio of School Turnaround AmeriCorps program approaches. In addition, the Year 2 approach will expand what can be learned about the comparison condition and minimize burden on those schools already participating, thus facilitating recruitment.

Modifications to Research Questions

Findings from the Year 1 Evaluation have improved understanding of the mechanisms of action underlying the School Turnaround AmeriCorps program. For example, the findings have improved understanding of the strengths and challenges of program implementation, such as new program start-up issues and how grantees have mitigated them; they have also helped demonstrate the conditions that facilitate or hinder effective implementation, such as the importance of relationship building in grantee-school partnerships and of positive member and student interactions for effective service provision. Based on the findings from Year 1, the research questions have been adjusted and expanded in Year 2 (see Exhibit 1) to probe further into specific aspects of implementation, including the use of partnership agreements as a planning and accountability tool and strategies used by programs in schools that have exited SIG/Priority status since the beginning of the grant period. The Year 2 Evaluation also places a greater emphasis on comparing the strengths and challenges of AmeriCorps schools over time and across the different strategies and partnerships for engaging and helping schools to reach their turnaround goals. These refinements to the research questions will help to further improve understanding of the program theory of change. The Year 2 Evaluation will continue to assess AmeriCorps members’ contributions to low-performing schools’ progress in their turnaround efforts, and will seek to understand the mechanisms underlying those contributions.

Modifications to Data Collection Instruments and Incentives

Exhibit 2 displays the various data collection activities, sample sizes, and proposed incentives across multiple stakeholders along with the timing for collecting these data. As in Year 1, primary data collection will consist of surveys, semi-structured interviews, and focus groups with grantees, AmeriCorps members, school leaders, teachers, counselors, and parents/guardians of students who receive AmeriCorps services. The Year 2 evaluation will continue to use surveys to gather data from a broad number of program respondents, as well as interviews, observations, and focus groups to gather in-depth data from multiple perspectives at case study schools (both program and comparison schools). Secondary data analysis will be based primarily on updated partnership and data use agreements, annual grantee progress reports, and performance measure data, and AmeriCorps member activity tracking information that can be obtained from CNCS and grantees. The study purposefully does not include direct data collection from students so as not to detract from their instructional time.

The already discussed revisions to the design of the evaluation and research questions, along with the Year 1 findings, have prompted minor substantive modifications to some of the data collection instruments for the Year 2 evaluation and non-substantive modifications to improve the clarity of the instruments. The proposed revised instruments themselves, and detailed summaries of the changes made to each instrument and rationale for the proposed changes, are enclosed with this memo.

In light of the usefulness of the Year 1 findings for improving program implementation, CNCS recommends that the contractor continue to use case study interview and focus group protocols developed and tested with less than nine respondents each during Year 1. With the proposed modifications, the revised instruments will improve the research team’s ability to collect more precise data that directly address the Year 2 research questions. CNCS therefore seeks OMB expedited review and approval to administer the revised protocols to the full subsample of respondents listed in Exhibit 2.

Additionally, these instrument modifications result in substantially lower burden to respondents. The Year 1 design estimated a maximum data collection burden of 2,274 hours per year, whereas the Year 2 design estimated total burden is 172.5 hours, as shown in Exhibit 3. The data collection burden for the original Year 1 design included pre- and post-surveys of both teachers and principals in 62 treatment and 62 comparison schools, as well as a member survey. In the modified design for Year 2, we will no longer administer surveys to AmeriCorps program teachers or AmeriCorps members. Logistical and programmatic obstacles limited the usefulness of teachers’ survey responses and consumed a disproportionate amount of study resources. The overall response rate for teachers was 40 percent, and teachers’ survey responses were largely redundant with school leaders’ responses. AmeriCorps member response rates were similarly lower than expected. Furthermore, because the case study design involves a smaller sample size, conducting interviews and focus groups is a more appropriate methodology than surveying teachers and principals in both program and comparison schools.

Based on previous studies as well as the contractor’s recent experience with recruiting and conducting data collection during Year 1, CNCS also requests modifications to the incentive structure to improve school recruitment and response rates. A sub-study requested by OMB on the effect of incentives on survey response rates for school staff showed significant increases when an incentive of $15 or $30 was offered as opposed to no incentive (Gamse et al., 2008). In another study, Rodgers (2011) offered adult participants $20, $30, or $50 in one wave of a longitudinal study and found that offering the highest incentive of $50 showed the greatest improvement in response rates and also had a positive impact on response rates for the next four waves. In Year 1 of the School Turnaround AmeriCorps national evaluation, response rates were fairly low for some groups (teachers and members, as noted above) and we therefore recommend adding incentives for these groups to improve response rates. We also recommended increasing incentives for comparison schools (because many refused) and adding incentives for SIG exiters (since they may no longer have members). In sum, we request providing modest incentives for principals, teachers, and AmeriCorps members to increase survey and interview completion rates, as well as increases in stipends to comparison schools from $250 to $500 per school, and use of $500 stipends for program schools that have exited SIG status, to increase school participation in data collection activities. Exhibit 4 summarizes the proposed changes in incentives and the rationale for the proposed changes.

Exhibit 1: Year 2 Research Questions

1. How do School Turnaround AmeriCorps members help schools implement their turnaround plans and achieve key turnaround outcomes?

   1. How do AmeriCorps grantees work with teachers and other school personnel to identify and target students with whom their members will engage so that the school is more likely to achieve its turnaround goals?

   2. What are the specific service activities and school-level interventions that AmeriCorps members conducted at each school and how were those activities believed to support school turnaround?

   3. What are the specific capacity-building strategies that AmeriCorps members contributed to each school?

      1. How did school leaders and staff view the role and contributions of AmeriCorps members in building the school’s capacity to implement their turnaround effort?

      2. What are the areas in which schools believe AmeriCorps members have the most and least influence over the school’s ability to achieve its turnaround goals, and why?

      3. In what ways, if any, did the presence of AmeriCorps members allow school staff or volunteers to modify their activities in ways that might benefit students?

   4. Did the specific activities that AmeriCorps members conducted change over the course of the grant period? To what extent did grantees use data to inform continuous improvement efforts to meet changing needs and improve their interventions?

   5. How are School Turnaround AmeriCorps activities similar or different than activities provided by other partners in SIG/Priority schools?

2. Which aspects of grantee-school partnerships appear to be the most promising practices in terms of involvement and satisfaction of the school leadership and the participating AmeriCorps members?

   1. How have written partnerships agreements changed, if at all, over the course of the grant period?

   2. What improvements could be made to written partnership agreements and how they are used to improve desired outcomes for schools and for students?

   3. What lessons from the School Turnaround AmeriCorps partnerships could be applied to other education-focused programs in CNCS’s portfolio?

3. Are AmeriCorps members perceived by school leaders and other stakeholders to be more vital in supporting certain SIG/Priority strategies than others? Which activities pursued by AmeriCorps members are perceived as being more or less helpful in supporting schools’ turnaround efforts with respect to the following outcomes, and why?

   1. Overall success in school turnaround?

   2. Academic achievement?

   3. Students’ socio-emotional health?

   4. School climate?

   5. School capacity to implement its turnaround effort?

4. For School Turnaround AmeriCorps schools that have exited SIG/Priority status since the beginning of the grant period, what strategies have they used to improve?

   1. What strategies do school leaders and other stakeholders perceive as being most helpful?

   2. To what extent do school leaders and other stakeholders attribute improvement to the contributions of the School Turnaround AmeriCorps program?

Exhibit 2: Year 2 Data Collection Activities and Proposed Incentives

Respondent	Data Collection Activity	Total number of respondents	Proposed Incentive, Per Respondent for Each Activity	Minutes, Per Respondent	Fall 2015	Winter 2015	Spring 2016
Grantee	Phone interviews	13	None				
	Online surveys	13	None				
	Partnership & data use agreements	13	None				
	Grant activity information	13	None				
AmeriCorps Program Schools
Principal	Online surveys	60	$25	30			
	Focus groups (3 total)	9 (<10)	None	45	
	In-person interviews	3 (Year 2 case study)	None	30		
	Phone interviews	6 (Year 1 case study)	None	30		
		3 (Year 2 case study)	None	30		
Teacher	Phone interviews (3 per school)	18 (Year 1 case study) 9 (Year 2 case study)	$20	30		
	In-person interviews (2 per school)	6 (Year 2 case study)	$20	30		
	Focus groups (3 total)	9 (Year 2 case study)	$20	45		
AmeriCorps Member	Phone interviews (2-3 per school)	8 (Year 2 case study)	$25	30		
	In-person interviews (2-3 per school, if <3 members available for FG)	7 (Year 2 case study)	$25	30		
	Focus groups (3 total)	9 (Year 2 case study)	$25	45		
Parent	Focus groups (2 total)	6 (Year 2 case study)	$20	45		
Comparison Schools
Principal	In-person interviews	3 (Year 2 case study)	None	30		
	Phone interviews	3 (Year 2 case study) 6 (Year 1 case study)	None	30		
Teacher	In-person interviews (2 per school)	6 (Year 2 case study)	$20	30		
	Phone interviews (3 per school)	18 (Year 1 case study)	$20	30		
		9 (Year 2 case study)	$20	30		
	Focus groups (3 total)	9 (Year 2 case study)	$20	45		
Successful SIG Exiter Case Studies
Principal	Phone interviews	4 (Year 2 case study)	None	45		
Teacher	Phone interviews (3 per school)	12 (Year 2 case study)	$20	30		

Exhibit 3: Hour Burden of the Proposed Year 2 Data Collection

Average Time Per Response: 30 minutes per survey and most interviews; 45 minutes per 4 principal interviews; 45 minutes per focus group.

Estimated Maximum Total Burden Hours: 172.5 hours for year 2.³



Survey	AmeriCorps	Comparison		Pre/post?	Total
Principals	60	0		Yes	120
Grantee	13	NA		No	13
Total respondents					133
Unique respondents					73
	Minutes	3990
	Hours	66.5
Interviews	AmeriCorps	Comparison		Pre/post?	Total
Grantee staff	13	NA		Yes	26
AmeriCorps members	154	NA		No	15
Principals	16	12		No	28
Teachers	45	33		No	78
Total respondents					147
Unique respondents					134
	Minutes	4470
	Hours	74.5
Focus groups	AmeriCorps	Comparison	Pre/post?	Total
AmeriCorps members	95	NA	No	9
Principals	9	0	No	9
Teachers	9	9	No	18
Parents	6	0	No	6
Total respondents				42
Unique respondents				42
Minutes	1890
Hours	31.5
Total hours in Year 2	172.5



Exhibit 4: Incentive Payments in Year 1, Proposed Incentive Payments in Year 2

Respondent	Activity	Incentive Yr 1	Incentive Yr 2	Rationale for Change
AmeriCorps Principal	Survey	$0	$25	To increase response rate.
AmeriCorps Members	Interview	$0	$25	To increase response rate.
AmeriCorps Members	Focus Group	$0	$25, plus food/refreshments	To increase response rate
Teacher	Interview	$0	$20	To increase response rate. Last year teachers received a $10 gift card for completing the online survey. Year 2 does not include a teacher survey.
Teacher	Focus Group	$0	$20, plus food/refreshments	To increase response rate. Last year teachers received a $10 incentive for completing the online survey. Year 2 does not include a teacher survey.
Parents	Focus Groups	$20	$20, plus food/refreshments	Last year parents received a $20 incentive for completing telephone interviews. Year 2 includes in-person focus groups instead. Add food/refreshments and child care services given in-person format.
Matched Comparison School	School participation	$250 stipend	$500	To increase matched comparison school agreement/ participation rate.
Program schools that have exited SIG status	Principal and teacher interviews	$0	$500	To increase program school agreement/ participation rate since schools may no longer be participating in the initiative.

1 School Improvement Grants (SIG), authorized under section 1003(g) of Title I of the Elementary and Secondary Education Act of 1965 (Title I or ESEA), are grants to State educational agencies (SEAs) that SEAs use to make competitive sub-grants to local educational agencies (LEAs) that demonstrate the greatest need for the funds and the strongest commitment to use the funds to provide adequate resources in order to raise substantially the achievement of students in their lowest-performing schools. Source: http://www2.ed.gov/programs/sif/index.html. Accessed October 9, 2014.

2 The Evaluation Design Plan for Year 1 of the national evaluation assumed a sample size of 62 AmeriCorps program schools. The sample size was adjusted to 57 schools after a handful of schools was determined ineligible to participate in the evaluation, because they were no longer hosting AmeriCorps members or they were not in their second year of implementation.

3 Since this exhibit assumes both the maximum number of in-person member interviews and the maximum number of member focus groups, some members are double counted. In actuality, we will conduct either in-person interviews or focus groups with members at each site visit school.

4 This is a conservative estimate that includes estimated response times for phone and in-person interviews with members. In-person interviews with members will only occur at if <3 members at a school are available for a focus group.

5 This estimate assumes member focus groups will be possible at all 3 site visit schools.

4