Supporting Statement Part A_REVISED 2021-02-08

P art A: Supporting Statement for Paperwork Reduction Act Submission

Evaluating the DC Opportunity Scholarship Program After the 2017 Reauthorization

February 2021

Prepared for:

Meredith Bachman

U.S. Department of Education

555 New Jersey Ave, NW

Room 502I

Washington, DC 20208-5500

Submitted by:

Abt Associates Inc.

10 Fawcett Street

Cambridge, MA 02138

Part A: Supporting Statement for Paperwork Reduction Act Submission

Table of Contents

Part A: Supporting Statement for Paperwork Reduction Act Submission i

A. Justification 1

Introduction 1

A.1 Circumstances Making the Collection of Information Necessary 1

A.2 Purposes and Use of the Information Collection 1

A.3 Use of Information Technology and Burden Reduction 8

A.3.1 Administrative Data 8

A.3.2 Parent, Student, and School Administrator Surveys 8

A.3.3 Program Operator Interview 9

A.4 Efforts to Identify Duplication 9

A.5 Efforts to Minimize Burden in Small Businesses 9

A.6 Consequences of Not Collecting the Information 10

A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6 10

A.8 Consultation Outside the Agency 10

A.8.1 Federal Registrar Announcement 10

A.8.2 Consultations Outside the Agency 10

A.8.3 Unresolved Issues 11

A.9 Payments or Gifts to Respondents 11

A.10 Assurance of Confidentiality 12

A.11 Questions of a Sensitive Nature 13

A.12 Estimate of Response Burden 14

A.13 Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record-Keepers 20

A.14 Estimates of Costs to the Federal Government 20

A.15 Changes in Burden 20

A.16 Plans for Analysis, Publication, and Schedule 20

A.16.1 Analysis Plans 20

A.16.2 Publication and Schedule 21

A.17 Approval to Not Display Expiration Date 22

A.18 Exceptions to Item 19 of OMB Form 83-1 22

References 23

1. Justification

Introduction

The U.S. Department of Education (ED)’s Institute of Education Sciences (IES) requests clearance for data collection activities to support a congressionally mandated study of the District of Columbia (DC) Opportunity Scholarship Program (OSP). Specifically, this request covers administrative data as well as surveys of the OSP program operator, administrators of participating and non-participating OSP schools, OSP applicants, and OSP users.

In 2004, Congress established the DC Opportunity Scholarship Program (OSP), the only federally funded private school choice program. The OSP awards scholarships to low-income DC residents to allow their children to attend participating DC private schools.

1. Circumstances Making the Collection of Information Necessary

Collecting information about the OSP is critical given ED’s interest in private school choice as a way to improve students’ educational outcomes and Congress’s focus on the program. Proposed legislation supports both expanding the OSP to serve more students in DC and new tax credits that would make up to $5 billion available to fund similar programs nationwide.

The importance of the OSP to Congress is reflected in its requirement that IES conduct a third evaluation of the program, following those completed in 2011 and 2019. The two previous evaluations relied on lotteries to award private school scholarships to provide the conditions for a rigorous assessment of the program’s effectiveness. Together, those earlier evaluations raised some questions about how well the program was improving student achievement and parent and student satisfaction with their schools. Additionally, the most recent evaluation found declining rates of participation in the OSP program.

In the 2017 Scholarship for Opportunity and Results (SOAR) Reauthorization Act, Congress mandated that this third program evaluation not rely on a lottery-based allocation of scholarships. Therefore, the data collection for the new evaluation will address how the OSP is being implemented and what changes might be made to better fulfill the mission of increasing access to high-quality education for all students (SOAR Act § 38-1853.02 (4)). ED plans to use the study results to inform program improvements aimed to increase families’ participation and satisfaction with the OSP and students’ academic success. Results may also inform a future reauthorization of the program.

2. Purposes and Use of the Information Collection

IES has contracted with Abt Associates Inc. and its partners, the Center on Reinventing Public Education (CRPE) and Dr. Anna Egalite (together, the “study team”), to conduct the evaluation, including all data collection. The new OSP evaluation will collect data to answer and report on the questions shown in Exhibit A1 below.

Exhibit A1: Key Research Questions

1. Who is the OSP serving, and who is it not serving?

2. What are the top challenges families face participating in the OSP?

3. What does the OSP program operator do or not do to help families overcome challenges in applying for, using, and continuing to use an OSP scholarship to enroll and stay enrolled in a private school?

4. What do schools do or not do to help families overcome challenges to enrolling and staying enrolled?

5. How are OSP students progressing academically?

The study will result in a report on the implementation of the OSP, including identification of challenges encountered by OSP-eligible applicants,¹ participating schools, and the program operator; and potential program or policy changes to help address these challenges. A subsequent issue brief will focus on challenges related to families’ ongoing participation in the OSP, since about 20% of students stop using scholarships after one year of participation.

The study will also use the collected data to disseminate up to four issue briefs. The first brief will examine who participates in the OSP, describing how many students are likely eligible for the OSP in DC, who applies and who uses the scholarship to enroll in a private school.

A second brief will examine the academic success of recent OSP users (those enrolled in OSP private schools in 2017-2018, 2018-2019, and 2019-2020) and past OSP applicants (those included in the most recent random assignment evaluation of the OSP). The study team will examine OSP users’ academic achievement in the across concurrent school years. The brief will allow ED to meet the congressional requirement to examine the academic progress of participating students (SOAR Act § 38-1853.09).

Another brief will examine families’ schooling decisions across the DC educational landscape. The brief would explore which school characteristics DC families value most when deciding on a school to attend and how these preferences vary by characteristics of the family. Focusing on schooling decisions of all DC families, this brief could help ED to understand whether there are program or policy changes that would induce more parents to participate in OSP.

The final brief would report on long-term educational outcomes such as high graduation, college enrollment, and degree attainment of the three cohorts of OSP applicants from the most recent prior OSP evaluation (Webber et al., 2019a, 2019b). For OSP applicants randomly assigned in the prior OSP evaluation to receive an OSP scholarship or not, the study team plans to compare long-term outcomes (including high school graduation, college enrollment, persistence, and graduation) for students offered an OSP scholarship to those of students who were not offered (or were offered but did not use) an OSP scholarship. Clearance is not being requested for any of the data that will be used in these analyses.²

Exhibit A2 lists the type of administrative and primary data to be collected and describes the purpose of each. The planned data collection and written products – the implementation report, the brief on challenges related to families’ ongoing participation in the OSP, and the additional issue briefs – are designed to fulfill the requirements of the SOAR Act (§ 38-1853.09(a)(4)) and provide actionable information for program improvement.

Exhibit A2. Data Sources

Data Source	Respondent	Mode and timing of data collection	Use(s) in Study
Administrative data
OSP application data 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022OSP scholarship applicants	OSP Program Operator	Data extraction Spring 2021 Fall 2021, and Fall 2022	To identify: Eligible applicants for an OSP scholarship How applicants learned about the OSP, reasons for applying Applicants’ top three private school choices Students renewing an OSP scholarship
OSP participant data 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022, 2022-2023 OSP scholarship users	OSP Program Operator	Data extraction Spring 2021, Fall 2021, and Fall 2022	To identify: Who uses an OSP scholarship In which private schools OSP users enrolled Scholarship amount received
My School DC data Students entering lottery for DC public (traditional, charter) schools for enrollment in 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022	Office of the State Superintendent of Education (OSSE)	Data extraction Spring 2021 and Fall 2022	To identify: Who participates in DC public school lottery, including for PK3 and PK4 Schools requested by lottery users Schools offered by lottery (or waitlist status)
Public school enrollment data Students enrolled in DC public (traditional, charter) schools in 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022, 2022-2023	Office of the State Superintendent of Education (OSSE)	Data extraction Spring 2021, Fall 2021, Fall 2022, and Fall 2023	To identify characteristics of students and schools: Demographics of students who are eligible for, apply, use, and do not use OSP scholarship Prior academic achievement and disciplinary records Identify schools in which OSP-eligible students who do not use OSP scholarship enroll
Public school characteristics data DC public (traditional, charter) schools in 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022, 2022-2023	Office of the State Superintendent of Education (OSSE)	Data extraction Spring 2021, Fall 2021, Fall 2022, and Fall 2023	To identify characteristics of schools that may affect families’ schooling preferences and school enrollment decisions
OSP student achievement data OSP students who take norm-referenced tests in mathematics and English/language arts in 2017-2018, 2018-2019, 2019-2020, 2020-2021, 2021-2022, 2022-2023	OSP-participating private schools	Data extraction Spring 2021, Fall 2022, and Fall 2023	To examine OSP students’ academic progress
High school graduation dataa For OSP applicants from 2012-2013, 2013-2014, 2014-2015 randomly assigned to receive or not receive an offered OSP scholarship (see Webber et al., 2019a)	OSSE and DC private schools that participated in OSP in in 2012-2013 to 2014-2015	Data extraction Fall 2024	Estimate impact of an offered OSP scholarship on high school graduation
College enrollment datab For OSP applicants from 2012-2013, 2013-2014, 2014-2015 randomly assigned to receive or not receive an offered OSP scholarship (see Webber et al., 2019a)	National Student Clearinghouse	Paid data extraction Fall 2024	Estimate impact of an offered OSP scholarship on college enrollment, degree completion
OSP private school website contentc Schools participating in OSP in 2021-2022	Study team will collect data directly OSP-participating private schools’ public websites	Data extraction Spring/Summer 2021	Identify information about the OSP, school admissions process, financial information that is readily accessible to parents Assess alignment between information schools provide and families’ reported challenges in applying Identify what parents interested in private schools might (or might not) learn about OSP from private school web sites
Primary data
Program operator interview	Executive director, Assistant director, Enrollment coordinator	In-Person Interview Fall 2021	Assess program operator’s perception of challenges OSP families encounter applying for, using OSP scholarship to enroll in private school, and alignment with families’ reported challenges Identify what information and supports the program operator provides OSP families Identify challenges the program operator faces (for example, with families, schools) Identify potential improvements to information or supports provided to families
Parent applicant survey	Parent of each eligible 2021-2022 OSP scholarship applicant	Web Survey Spring 2022	Understand families’ reasons for applying for OSP scholarship Identify other schooling options families considered and reasons for schooling decisions Identify challenges families face with OSP scholarship application process Identify challenges families face applying for admission to OSP private school(s)
Parent user survey	Parent of each eligible 2021-22 OSP scholarship applicant who used an OSP scholarship to enroll in an OSP-participating private school for 2021-22 school year	Web Survey Spring 2023	Understand families’ reasons for using or not using OSP scholarship Identify challenges families face enrolling, remaining enrolled in a private school Understand families’ reasons for staying enrolled or leaving an OSP school Measure parents’ satisfaction and perceived safety of student’s current school
Student applicant survey	Eligible 2021-22 OSP scholarship applicants in grades 4-12	Web and Paper Survey Spring 2022	Measure students’ experiences with learning environment, teachers, peers, and staff in current school Measure students’ satisfaction with, perceived safety of current school Measure percentage of OSP applicants who received tutoring services
Student user survey	Eligible 2021-22 OSP scholarship applicants in grades 4-12 who used an OSP scholarship to enroll in an OSP-participating private school for 2021-2022 school year	Web and Paper Survey Spring 2023	Measure students’ experiences with learning environment, teachers, peers, and staff in current school Measure students’ satisfaction with, perceived safety of current school Measure percentage of OSP applicants who received tutoring services
OSP school administrator survey	School administrators in private schools participating in OSP	Web Survey Spring 2022	Assess OSP schools’ perception of challenges OSP families encounter applying for admission to, enrolling in, staying enrolled in private school and alignment with families’ reported challenges Identify what information and support OSP schools provide to OSP scholarship students and their parents Identify challenges that schools face in participating in the OSP Identify characteristics of schooling environments that may mediate family schooling decisionsd
Non-OSP school administrator survey	One administrator at each DC private school not participating in OSP	Web Survey Spring 2022	Identify reasons that DC private schools choose not to participate in the OSP Identify characteristics of schooling environments that may mediate family schooling decisionsd
Public school administrator survey	One administrator at each DC public (traditional or charter) school	Web Survey Spring 2022	Identify characteristics of schooling environments that may inform family schooling decisionsd

^a These data will be extracted in the fall of 2024 and are not covered in this clearance request.

^b These data will be extracted in the fall of 2024 and are not covered in this clearance request but will have no associated burden, as the team would pay NSC to extract the data.

^c These data will be extracted by the study team and are not covered in this clearance request.

^d While some of these characteristics for public traditional and charter schools are available from public sources, many are inconsistently measured across school sectors; for other characteristics the level of detail in the public measures is limited. Therefore, a survey that asks consistent and specific questions about all schools private, traditional public, and charter public is necessary.

3. Use of Information Technology and Burden Reduction

To minimize burden on study participants, the study team will use strategies that have proven successful in the prior DC OSP evaluations as well as past studies the team has conducted with similar populations of school administrators, students, and parents (e.g., the Study of Enhanced College Advising in Upward Bound, the Study of Student Messaging in GEAR UP, and the Impact Evaluation of Academic Language Interventions). General strategies to minimize burden using technology are described below for each type of data collection.

1. Administrative Data

To minimize burden on organizations providing administrative data to the study team, administrative data will be collected in electronic form using a secure file sharing site with FedRAMP security authorization and web-based interface with drag-and-drop capability and user-friendly administrative controls to efficiently manage internal and external user permissions and levels of access. Organizations will have the flexibility to submit electronic data in a wide range of file formats (Excel, csv, SAS, etc.).

2. Parent, Student, and School Administrator Surveys

The study team will ensure that surveys are as low burden and secure as possible for respondents. The study team will administer the parent, student, and school administrator surveys electronically using an integrated survey system that combines computer-assisted web interviewing (CAWI) and computer-assisted personal interviewing (CAPI) implementation. This will allow respondents to take the survey at any time within the window of the data collection period and to submit their responses electronically. Features of the online surveys include:

• Secure personalized access. Each survey respondent will receive a customized link to the survey. The survey software allows respondents to save responses and return to the survey later to finish at their convenience.

• Automated skip patterns. Skip logic embedded in the survey will minimize respondent burden by omitting non-applicable questions. This type of programming also reduces entry errors that may require follow-up contacts to gather correct information.

• Automated validation checks. The software will check for allowable ranges for numeric questions, minimizing out-of-range or unallowable values. This type of programing also reduces entry errors that may require following up with contacts to gather correct information.

• Closed-ended questions. These types of questions reduce burden on respondents and facilitate data analysis. Some questions will include “other, specify” options to ensure respondents have an opportunity to enter information that does not fit pre-existing options.

To make accommodations for students with no access to computers or phones and to improve response rates, the study team will also administer the student surveys in paper form. For students in OSP schools, the study team will ask the school’s permission to administer a paper version of the student survey to OSP students on site.³ For students in a non-OSP private school or a DC public school (or OSP students absent when the study team administers the survey on site), the study team will send parents an email and letter containing web links to the student and parent surveys. For students whose parents receive an email link, parents will provide electronic consent before allowing their child to complete the online student survey.

3. Program Operator Interview

The program operator interview will be conducted in person with trained interviewers from the study team. For these interviewers, the use of technology will not reduce burden on the respondent, but interviewers may use laptop computers to record responses and notes. In addition, to ensure that interviewers capture responses accurately and can conduct efficient content analysis of the interviews, the study team may digitally record the interview with respondent permission and transcribe.

4. Efforts to Identify Duplication

Whenever possible, the study team will utilize administrative data to gather the information needed to address the study’s research questions (see Exhibit A1 for the research questions and Exhibit A2 for the types and sources of administrative data to address each question). The study team has sought to build on the data⁴ and findings of the prior two evaluations of OSP (Wolf et al., 2010; Webber et al., 2019a, 2019b) to prevent unnecessary collection of data and design surveys that are clearly written and concise. The information to be collected in the program operator interview and the parent, student, and school administrator surveys is not available elsewhere.

5. Efforts to Minimize Burden in Small Businesses

No information for this study will be collected from small businesses. The primary small entities for this study are (a) the OSP program operator; (b) OSP-participating private schools and DC public schools; and (c) OSSE. The data collection procedures have been designed to minimize burden on these entities. First, the study team will limit requests for administrative data to the minimum set of variables and records needed to conduct the study. The study team will schedule a single in-person interview with the OSP staff in the spring of 2022 at a time that minimizes any disruption to the program operator’s regular responsibilities. The study team has minimized the burden of on-site data collection from OSP-participating schools by using study staff to administer the 15-minute student survey during regular school hours.

All other entities are individual students and parents.

6. Consequences of Not Collecting the Information

The proposed evaluation is congressionally mandated as part of the 2017 Reauthorization of the SOAR Act (SOAR Act § 38-1853.09). The evaluation is intended to inform Congress about student and parent satisfaction with the OSP and student progress, as well as to identify potential levers for program improvement. Without collecting the proposed data, ED could not meet its obligations to Congress.

7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

There are no special circumstances concerning the collection of information in this study.

8. Consultation Outside the Agency

   1. Federal Registrar Announcement

On November 5, 2020, a 60-day Federal Register Notice was published at 85 FR 70596.

2. Consultations Outside the Agency

The study team has formed an external Technical Working Group (TWG) in partnership with ED to provide guidance on the evaluation design, instrumentation, and data collection for the study. The following eight members serve as TWG members:

• Jeanne Century, Director, Outlier Research and Evaluation, University of Chicago (implementation methodology)

• Jennifer Comey, Director of Planning, Data, and Analysis, DC Office of the Deputy Mayor for Education (DC choice system)

• Joshua Cowen, Associate Professor and Co-Director of the Education Policy Innovation Collaborative (EPIC), Michigan State University (parent school choice decision-making)

• David Figlio, Professor and Dean, School of Education and Social Policy, Northwestern University (evaluation of voucher programs)

• Elizabeth Grant, Visiting Associate Professor, Graduate School of Education and Human Development; Program Lead, Education Consortium on Research and Evaluation (EdCORE) DC, The George Washington University (research in DC schools)

• Jeffrey Henig, Professor, Teachers College, Columbia University (school and family responses to choice)

• Brian Kisida, Assistant Professor, Truman School of Public Affairs, University of Missouri (implementation of voucher programs)

• Martin West, Professor, Harvard Graduate School of Education, Harvard University (evaluation of school choice programs, measuring non-cognitive skills)

The study team convened the TWG in October 2019 to review the study design and ensure the study provides actionable information that policymakers and the program operator can use to improve the program.⁵ The study team plans to convene the TWG again prior to the public release of the report.

3. Unresolved Issues

There are no unresolved issues.

9. Payments or Gifts to Respondents

To maximize the success of our data collection effort, we propose to provide incentives to offset students’, parents’, and school administrators’ time and effort with completing the data collection activities. Incentives are also proposed because high response rates are needed to make the study findings reliable, and prior evidence suggests the importance of providing an incentive. Monetary incentives, particularly for low-income populations similar to OSP applicant families, reduce non-response bias and improve survey representativeness (James, 1997; Goritz, 2006; Groves et al., 2006; Messer & Dillman, 2011; Singer & Kulka, 2002; Singer & Ye, 2013). The prior OSP evaluation used incentives to encourage respondents to complete data collection activities, with varying degrees of success (see Exhibit A3). We propose some changes to the incentives and burden from the prior evaluation to increase response rates. These proposed amounts are consistent with guidelines in the March 22, 2005, memo “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB, including the standard practice of linking the dollar amounts to the extent of burden.

Exhibit A3. Incentives, Burden, and Response Rates of Data Collection Activities: Prior and Proposed

Data collection activity	Prior Incentive Amount OMB (#1850-0800)	Prior burden (hours)	Prior Response Rate OMB (#1850-0800)	Proposed Incentive	Proposed Burden Estimate (hours)	Proposed Target Response Rate
Parent applicant survey	$20 ($5 in advance, $15 on completion)	0.33	77% a	$25 gift card ($5 in advance, $20 on completion)	0.25	80%
Parent user survey	$20 ($5 in advance, $15 on completion)	0.33	72% b	$25 gift card ($5 in advance, $20 on completion)	0.25	80%
Student applicant survey	$0	0.25	68% c	$10 gift card (on completion)	0.25	80%
Student user survey	$0	0.25	68% b	$10 gift card (on completion)	0.25	80%
OSP private school administrator survey	$10 (in advance)	0.33	100% d	$25 gift card (on completion)	0.25	100%
Non-OSP private school administrator survey	$10 (in advance)	0.33	61% d	$25 gift card (on completion)	0.20	80%
Public school administrator survey	$10 (in advance)	0.33	91% d	$25 gift card (on completion)	0.20	80%

Note: The Prior Response Rates shown for the parent and student surveys reflect the prior evaluation’s treatment group response rates only. The study team expects that in the current evaluation, all eligible OSP applicants will be offered OSP scholarship, as recent data shows that OSP is not oversubscribed.

^a Treatment group only, Dynarski et al. (2017, p. B-10).

^b Treatment group only, Dynarski et al. (2018, p. B-9).

^c Treatment group only, Dynarski et al. (2017, p. B-11).

^d Betts et al. (2016, Table A-2, p. 21).

10. Assurance of Confidentiality

Abt Associates and its research partners will conduct all data collection activities for this study in accordance with relevant regulations and requirements, which are:

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a)

• The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99)

• The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98)

• The Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183

The study team and ED will protect the confidentiality of all data collected for the study and will use it for research purposes only. All data will be kept in secured locations. As soon as it is feasible, the study team will replace personally identifiable information (PII) with study-specific identifiers, retaining this PII separately from responses to data collections. PII will be destroyed as soon as it is no longer required. All members of the study team having access to the data will be trained and certified on confidentiality and data security procedures. Reports will present data only in aggregate form, such that reports do not identify individuals and schools. The study team will release to ED personally identifiable information (PII) in separate files that cannot be linked to study-specific identifiers or responses to data collections. ED may provide its contractor(s) access to this PII for future evaluations of this program.

The following is an example of the type of statement that will be included on requests for data:

“Study reports will not identify you or your child or your child’s school. We will give data collected as part of this research to the U.S. Department of Education for possible future use by other researchers. The U.S. Department of Education and all researchers they hire are required by law to follow all federal laws regarding privacy and confidentiality.”

The following safeguards are routinely employed by Abt Associates to ensure confidentiality, and they will be consistently applied to this study:

• All employees sign a confidentiality pledge that emphasizes the importance of confidentiality and describes employees’ obligations to maintain it.

• Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification numbers.

• Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

To ensure the security of administrative data requested from the program operator, OSSE, and OSP schools, the study team uses a FIPS 140-2 compliant data sharing tool (MOVEit) that encrypts data in transit and at rest. This data sharing tool allows the study team to manage user permissions for uploading, viewing, editing, or downloading data and restrict unauthorized access to data.

The protocol for the program operator interview will include the following statement:

“Although the study team will not release personally identifiable information about individual respondents, data from this interview will be attributed to the OSP program operator.”

The study will take several steps to safeguard respondent information:

1. All contractor staff will comply with the security investigation requirements governed by their risk/sensitivity level as detailed in the ED Contractor Vetting Security Requirements (11-1-2019).

2. All contractor staff will receive instruction in the privacy requirements of the study.

3. Access to any data with identifying information will be limited to contractor staff directly working on the study. Access to electronic data will require individual usernames and passwords.

4. Names and other identifying information for survey respondents will be replaced with numerical identifiers after the data are collected and prior to analysis. A key linking the names to the identifiers will be kept in a separate location, with access for Abt staff on a need-only basis.

5. Any quotations from responses used in public reporting will be edited to ensure that the identity of the respondent cannot be ascertained, with the caveat that responses from program operator staff will be attributable to the program operator (but not an individual staff member).

The study team will house survey data on the Confirmit platform, which is FISMA Moderate compliant. For analyses, the study team will store de-identified data within Abt’s Analytic Computing Environment (ACE), which complies with HIPAA, FERPA, and FISMA Moderate standards implemented in FedRAMP-certified Amazon Web Services (AWS) environments. Additional security features include encrypted storage, intrusion detection, and audit log aggregation. ACE is monitored seven days a week, 24 hours a day using advanced monitoring and alerting tools. Expert security and IT staff continuously review audit logs, conduct regular scans, and apply patches. Redundant backups are stored both on AWS and at a remote data center.

To ensure that study participants are properly protected, the Abt Institutional Review Board (IRB) will review the study design protocols, informed consent process, data security plan, and all data collection instruments and procedures.

11. Questions of a Sensitive Nature

The study does not include questions of a sensitive nature.

12. Estimate of Response Burden

The total annual respondent burden for the data collection effort covered by this clearance request is 303 hours. Exhibit A4a presents the estimated time burden to respondents. Exhibit A4b presents the estimated cost burden to respondents. The following assumptions informed these burden estimates: ⁶

• Program Operator:

  • One designated point of contact at the OSP program operator will compile and submit application and participant data. Collecting and submitting the applicant and participant data will take approximately two hours per request for each of the data types (six data requests total). The cost to the program operator is based on an average hourly wage of $48.73 per hour in 2018 for Social and Community Service Managers in the District of Columbia (BLS, 2020a).

  • Up to three staff will participate in the program operator interview one time. The interview will take approximately one hour. The cost to the program operator is based on an average hourly wage of $48.73 per hour in 2018 for Social and Community Service Managers in the District of Columbia (BLS, 2020a).

• OSSE:

  • One point of contact at OSSE will compile and submit DC public school enrollment data and school-level characteristics for all DC public schools four times as well as My School DC data two times. Collecting and submitting these data will take an average of five hours per request for each of data type (ten data requests in total). The cost to OSSE is based on an hourly wage of $46.01 per hour in 2018 for Database Administrators in the District of Columbia (BLS, 2020b).

• School administrators:

  • One administrator at each OSP, non-OSP private, and DC public school will complete a school administrator survey. The OSP administrator survey will take approximately 15 minutes to complete. The non-OSP private and DC public school administrator surveys will take approximately 12 minutes to complete. The hourly cost estimate to each school is based on an annual salary of $91,700 per year in 2018 for Education Administrators, Elementary and Secondary Schools in private industry ($44.09 per hour) (BLS, 2020d) and $111,760 per year in 2018 ($53.73 per hour annually) for Education Administrators, Elementary and Secondary Schools in the District of Columbia (BLS, 2020e).

• Study liaisons:

  • The study team will communicate with a designated school-based study liaison at each of the 43 OSP-participating private schools to schedule visits to administer student surveys. Scheduling the visits and communicating with teachers in classrooms with OSP students will take approximately 15 minutes per visit. The study liaison will be asked to schedule visits to administer surveys twice: once for the student applicant surveys and once for the student user surveys. The cost to each school is based on an average hourly wage of $32.24 per hour in 2018 for Educational, Guidance, School and Career Counselors in the District of Columbia (BLS, 2020f).

  • The study point of contact at each OSP-participating private school will compile and submit OSP student achievement data. Collecting and submitting these data will take approximately two hours per request (three requests total). The cost to the schools is based on an hourly wage of $32.24 per hour in 2018 for Educational, Guidance, School and Career Counselors in the District of Columbia (BLS, 2020c).

• Parents:

  • One parent from each family with at least one student applicant will complete the parent applicant survey. The parent applicant survey will take approximately 15 minutes to complete. The burden estimate accounts for the fact that parents in families with multiple applicants will only be asked to do the survey once for a single applicant, chosen at random by the study team. The average hourly cost of $20.09 is based on study team calculations using 185% of the federal poverty level for households ranging in size from two to eight persons and the prior evaluation’s reported percentage of families with one to six OSP applicants.

  • The number of responses to the parent user survey assumes that an estimated 60% of households with an eligible OSP applicant will use an offered OSP scholarship. Data from the prior evaluation showed that 42% of eligible students offered an OSP scholarship used it to attend a participating school in 2018-2019 (Webber et al., 2019a, Figure 11). We have conservatively estimated that up to 60% of those offered an OSP scholarship for the 2021-22 school year will use it to attend a private school. For each household with at least one OSP scholarship user, one parent will complete the parent user survey that will take approximately 15 minutes to complete. Parents of more than one scholarship user will be asked to take the survey once for a randomly selected user.

• Students:

  • One eligible student OSP applicant entering grades 4-12 will complete the student applicant survey once. Completing the survey will take approximately 15 minutes.

  • One student using an OSP scholarship in grades 4-12 will complete the student user survey once. Completing the survey will take approximately 15 minutes.

Exhibit A4a. Estimate of Respondent Time Burden by Year

Respondent Type	Time per Response (Hours)	Maximum Number of Responses	Number of Respondents	Total Time Burden (hours)
2021 CALENDAR YEAR (January 1, 2021 – December 31, 2021)
Program Operator
OSP applicant data (spring 2021, fall 2021)	2	2	1	4
OSP participant data (spring 2021, fall 201)	2	2	1	4
Program operator interview (fall 2021)	1	1	3	3
OSSE
DC public school enrollment data (spring 2021, fall 2021)	6	2	1	12
DC public school characteristics (spring 2021, fall 2021)	4	2	1	8
My School DC data (spring 2021, fall 2021)	6	2	1	12
OSP private schools
Student achievement data (spring 2021)	2	1	43	86
2021 total hours				129
2022 CALENDAR YEAR (January 1, 2022 – December 31, 2022)
Program Operator
OSP applicant data (fall 2022)	2	1	1	2
OSP participant data (fall 2022)	2	1	1	2
OSSE
DC public school enrollment data (fall 2022)	6	1	1	6
DC public school characteristics (fall 2022)	4	1	1	4
OSP private schools
School administrator survey (spring 2022)	0.25	1	43	11
School-based study liaison for coordinating student surveys (spring 2022)	0.25	1	43	11
Student achievement data (fall 2022)	2	1	43	86
Non-OSP private schools
School administrator survey (spring 2022)	0.20	1	23	5
DC public schools
School administrator survey (spring 2022)	0.20	1	225	45
Parents of eligible OSP applicants
Parent applicant survey (spring 2022)	0.25	1	750	188
Eligible OSP student applicants in grades 4-12
Student applicant survey (spring 2022)	0.25	1	500	125
2022 total hours				485
2023 CALENDAR YEAR (January 1, 2023 – December 31, 2023)
OSSE
DC public school enrollment data (fall 2023)	6	1	1	6
DC public school characteristics (fall 2023)	4	1	1	4
OSP private schools
School-based study liaison for coordinating student surveys (spring 2023)	0.25	1	43	11
Student achievement data (fall 2023)	2	1	43	86
Parents of OSP scholarship users
Parent user survey (spring 2023)	0.25	1	450	113
OSP student scholarship users in grades 4-12
Student user survey (spring 2023)	0.25	1	300	75
2023 total hours				295
Total burden across all years				909

Exhibit A4b. Estimate of Respondent Cost Burden by Year

Data collection	Annual Salary Estimate	Average Hourly Wage	Time per Response (Hours)	Maximum Number of Responses	Cost per Response	Number of Respondents	Total Cost for Responses
2021 CALENDAR YEAR (January 1, 2021 – December 31, 2021)
Program Operator
OSP applicant data (spring 2021, fall 2021)	$101,360	$48.73 a	2	2	$97.46	1	$194.92
OSP participant data (spring 2021, fall 2021)	$101,360	$48.73 a	2	2	$97.46	1	$194.92
Program operator interview (fall 2021)	$101,360	$48.73 a	1	1	$48.73	3	$146.19
OSSE
DC public school enrollment data (spring 2021, fall 2021)	$95,710	$46.01 b	6	2	$276.06	1	$552.12
My School DC data (spring 2021, fall 2021)	$95,710	$46.01 b	6	2	$276.06	1	$552.12
DC public school characteristics (spring 2021, fall 2021)	$95,710	$46.01 b	4	2	$184.04	1	$368.08
OSP private schools
Student achievement data (spring 2021)	$67,050	$32.24 c	2	1	$64.48	43	$2,772.64
2021 total cost							$4,780.99
2022 CALENDAR YEAR (January 1, 2022 – December 31, 2022)
Program Operator
OSP applicant data (fall 2022)	$101,360	$48.73 a	2	1	$97.46	1	$97.46
OSP participant data (fall 2022)	$101,360	$48.73 a	2	1	$97.46	1	$97.46
OSSE
DC public school enrollment data (fall 2022)	$95,710	$46.01 b	6	1	$276.06	1	$276.06
DC public school characteristics (fall 2022)	$95,710	$46.01 b	4	1	$184.04	1	$184.04
OSP private schools
School administrator survey (spring 2022)	$91,700	$44.09 d	0.25	1	$11.02	43	$473.93
School-based study liaison for coordinating student surveys (spring 2022)	$67,050	$32.24 c	0.25	1	$8.06	43	$346.58
Student achievement data (fall 2022)	$67,050	$32.24 c	2	1	$64.48	43	$2,772.64
Non-OSP private schools
School administrator survey (spring 2022)	$91,700	$44.09 d	0.2	1	$8.82	23	$202.80
DC public schools
School administrator survey (spring 2022)	$111,760	$53.73 e	0.2	1	$10.75	225	$2,417.85
Parents of eligible OSP applicants
Parent applicant survey (spring 2022)	$41,783.24	$20.09 f	0.25	1	$5.02	750	$3,766.88
Eligible OSP student applicants in grades 4-12
Student applicant survey (spring 2022)	NA	NA	0.25	1	$0.00	500	$0.00
2022 total cost							$10,635.70
2023 CALENDAR YEAR (January 1, 2023 – December 31, 2023)
OSSE
DC public school enrollment data (fall 2023)	$95,710	$46.01 b	6	1	$276.06	1	$276.06
DC public school characteristics (fall 2023)	$95,710	$46.01 b	4	1	$184.04	1	$184.04
OSP private schools
School-based study liaison for coordinating student surveys (spring 2023)	$67,050	$32.24 c	0.25	1	$8.06	43	$346.58
Student achievement data (fall 2023)	$67,050	$32.24 c	2	1	$64.48	43	$2,772.64
Parents of OSP scholarship users
Parent user survey (spring 2023)	$41,783.24	$20.09 f	0.25	1	$5.02	450	$2,260.13
OSP student scholarship users in grades 4-12
Student user survey (spring 2023)	NA	NA	0.25	1	$0.00	300	$0.00
2023 total cost							$5,839.45
Total cost across all years							$21,256.14
Average cost per year							$7,085.38

^a The cost to the program operator is based on an average hourly wage of $48.73 in 2018 for Social and Community Service Managers in the District of Columbia (BLS, 2020a).

^b The cost to OSSE is based on an hourly wage of $46.01 in 2018 for Database Administrators in the District of Columbia (BLS, 2020b).

^c The cost to each school-based study liaison is based on an average hourly wage of $32.24 in 2018 for Educational, Guidance, School and Career Counselors in the District of Columbia (BLS, 2020f).

^d The hourly cost estimate to each private school administrator of $44.09 is based on an annual salary of $91,700 per year in 2018 for Education Administrators, Elementary and Secondary Schools in private industry (BLS, 2020d).

^e The hourly cost estimate to each public school administrator of $53.73 is based on an annual salary of $111,760 per year in 2018 for Education Administrators, Elementary and Secondary Schools in the District of Columbia (BLS, 2020e).

^f The average hourly cost to each parent of $20.09 is based on study team calculations where the annual income at 185% of the federal poverty level for households of two to eight persons was weighted based on the prior evaluation’s data on the family sizes of OSP applicants.

13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record-Keepers

There are no annualized capital/startup or ongoing operation and maintenance costs involved in the collection of the proposed data.

14. Estimates of Costs to the Federal Government

The estimated cost to the federal government of the evaluation, including its data collection activities as described above, is $3,286,584.⁷ The data collection activities are planned to occur from spring 2021 to fall 2023 and once again in fall 2024. Thus, over the four-year study period, the average annual cost to the federal government is $821,646.

15. Changes in Burden

This is a request for a new collection of information.

16. Plans for Analysis, Publication, and Schedule

    1. Analysis Plans

The study team will use descriptive, comparative, and regression analyses to address the study’s research questions:

• Descriptive analyses will be used to summarize administrative and survey data. For measures using continuous scales, the study team will calculate means and standard deviations to describe central tendency and variation. For categorical scales, the study team will use frequency distributions and percentages.

• For comparative analyses, the study team will use cross-tabulations to illustrate differences between groups or the distribution of measures across subgroups of interest (for example, demographic characteristics of OSP applicants and non-applicants; the percentage of OSP users and non-users who report difficulty completing private school admissions applications) and will calculate the F-test or chi-square statistic to test for statistical significance of observed differences.

• Regression analyses will be used for two purposes: (1) to understand which school characteristics are most important to families when deciding on school; and (2) to estimate the impact of the OSP on high school graduation, college enrollment, and college completion for the students in the most recent prior evaluation. Exhibit A5 summarizes the analyses planned for each of the study’s research questions.

Exhibit A5. Analysis Methods Planned for Each Research Question

Research Question	Descriptive analyses	Comparative analyses	Regression analyses
RQ 1. Who is the OSP serving, and who is it not serving? Who… applies for an OSP scholarship, and how do applicant and non-applicant families differ? uses an OSP scholarship, and how do OSP users differ from non-users? continues to use an OSP scholarship to stay enrolled in a private school (into a second school year), and how do “stayers” differ from “leavers”?	X	X
RQ 2. What are the top challenges families face participating in the OSP? Why do families… apply for an OSP scholarship, and what challenges do they face? use (or not use) an OSP scholarship to enroll initially in a private school, and what challenges do they face? continue (or not) to use an OSP scholarship to stay enrolled in a private school, and what challenges do they face?	X	X
RQ3. What does the OSP program operator do, and not do, to help families overcome challenges in applying for, using, and continuing to use an OSP scholarship to enroll and stay enrolled in a private school? Is the program operator aware of families’ challenges at each stage of the program? How does the program operator support families at each stage of the program and engage OSP schools to support families? What information or supports could the program operator expand or add? What challenges does the program operator face?	X	X
RQ4. What do schools do and not do to help families overcome challenges to enrolling and staying enrolled? Are OSP schools aware of families’ challenges? What information and supports do OSP schools provide to help families enroll and stay enrolled? How do the schooling environments in OSP schools, non-OSP schools, and public schools relate to families’ schooling decisions?	X	X	X
RQ5. How are OSP students progressing academically? What is the average national percentile ranking for OSP participants, and what is the average gain in national percentile ranking for OSP participants between two school years? What is the effect of receiving and using an OSP scholarship on long-term outcomes of students compared to those of students not offered an OSP scholarship?	X		X

2. Publication and Schedule

The evaluation plans call for one report on first-year findings, to be released in February 2024, with a follow-up issue brief on second-year findings, to be released in June 2024. Four additional issue briefs are planned for release, one in approximately February 2022, a second in approximately July 2022, a third in approximately February 2024, and a fourth in approximately June 2025. The planned publications are:

• An issue brief on academic progress, expected in Winter 2022;

• An issue brief on OSP participation, expected in Summer 2022;

• A report on the first-year findings, expected in Winter 2024;

• An issue brief on schooling decisions, expected in Winter 2024;

• An issue brief on the second-year findings, expected in Summer 2024; and

• An issue brief on long-term outcomes, expected in Summer 2025.

17. Approval to Not Display Expiration Date

No exemption is requested. The data collection instruments will display the expiration date.

18. Exceptions to Item 19 of OMB Form 83-1

The submission describing data collection requires no exemptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

References

Betts, J., Dynarski, M., Feldman, J. (2016). Evaluation of the DC Opportunity Scholarship Program: Features of Schools in DC, NCEE Evaluation Brief, completed under contract number ED-12-CO-0018.

Bureau of Labor Statistics, U.S. Department of Labor (BLS). (2020a). Occupational Employment and Wages. Accessed online at https://www.bls.gov/oes/current/oes119151.htm (March 11, 2020).

Bureau of Labor Statistics U.S. Department of Labor (BLS). (2020b). Occupational Employment and Wages. Accessed online at https://www.bls.gov/oes/current/oes151141.htm#st (March 11, 2020).

Bureau of Labor Statistics. U.S. Department of Labor (BLS). (2020c). Occupational Employment and Wages. Accessed online at https://www.bls.gov/oes/current/oes211012.htm#st (March 11, 2020).

Bureau of Labor Statistics, U.S. Department of Labor (BLS). (2020d). Occupational Employment Statistics Query System, one occupation, multiple industries, for Education Administrators, Elementary and Secondary Schools in private industry. Accessed online at https://data.bls.gov/oes/#/occInd/One%20occupation%20for%20multiple%20industries (March 11, 2020).

Bureau of Labor Statistics, U.S. Department of Labor (BLS). (2020e). Occupational Employment and Wages. Accessed online at https://www.bls.gov/oes/current/oes119032.htm#st (March 11, 2020).

Bureau of Labor Statistics, U.S. Department of Labor (BLS). (2020f). Occupational Outlook Handbook. Accessed online at https://www.bls.gov/ooh/community-and-social-service/school-and-career-counselors.htm (December 19, 2019).

Dynarski, M., Rui, N., Webber, A., Gutmann, B. (2017). Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year (NCEE 2017-4022). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Dynarski, M., Rui, N., Webber, A., and Gutmann, B. (2018). Evaluation of the DC Opportunity Scholarship Program: Impacts Two Years After Students Applied (NCEE 2018-4010). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Goritz, A. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1): 58-70.

Groves, R.M., Couper, M.P., Presser, S., Singer, E., Tourangeau, R., Acosta, G., & Nelson, L. (2006) Experiments in Producing Nonresponse bias. Public Opinion Quarterly. 70(5): 720-736.

James T. (1997). Results of wave 1 incentive experiment in the 1996 survey of income and program participation. Proceedings of the Survey Research Section, American Statistical Association, 834-83.

Messer, B. and Dillman, D. (2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75:429-457.

Singer, E., and R.A. Kulka. (2002). “Paying Respondents for Survey Participation.” In Studies of Welfare Populations: Data Collection and Research Issues. Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, edited by Michele Ver Ploeg, Robert A. Moffitt, and Constance F. Citro. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press, pp. 105–128.

Singer, E. and Ye, C. (2013). The use and effectives of incentives in surveys. Annals of the American Academy of Political and Social Science, 645(1):112-141.

Webber, A., Rui, N., Garrison-Mogren, R., Olsen, R., & Gutmann, B. (2019a). Evaluation of the DC Opportunity Scholarship Program. Impacts Three Years After Students Applied. (NCEE 2019-4006). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Webber, A., Rui, N., Garrison-Mogren, R., Olsen, R., & Gutmann, B. (2019b). Evaluation of the DC Opportunity Scholarship Program. Impacts Three Years After Students Applied. Technical Appendix (NCEE 2019-4006). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Wolf, P., Gutmann, B. Puma, M., Kisida, B., Rizzo, L., Eissa, N. and Carr, M. Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education

1 Eligible students must be current DC residents, 5 years old or entering Kindergarten through 12th grade for the upcoming school year, and be either recipients of SNAP benefits (food stamps) OR meet income guidelines: at or below 185% of the income threshold for first-time applicants or 300% of the income threshold for renewing families. Source: https://servingourchildrendc.org/our-program/apply/

2 High school graduation data will be extracted from OSSE and OSP private schools in the fall of 2024 and will be covered in a future clearance request. There is no respondent burden for collection of college enrollment data from the National Student Clearinghouse (NSC), which provides these data in exchange for payment.

3 To administer a web-based survey to children younger than age 13, the Children’s Online Privacy & Protection Act (COPPA) requires active, written parental consent. By administering a paper survey to students in OSP schools, the study team need only notify parents and provide them an opportunity to opt their child out of the student survey. For students not surveyed in school, parents will receive the web link for their child’s survey and may provide (or withhold) active, written consent.

4 The study team will use prior data from the second evaluation of the OSP (Webber et al., 2019a, 2019b) to identify OSP applicants in prior cohorts in order to obtain administrative data on high school graduation and college enrollment and completion.

5 All TWG members except Martin West were consulted either during the group meeting or individually.

6 High school graduation data will be extracted from OSSE and OSP private schools in the fall of 2024 and are not covered in this data request. There is no respondent burden for collection of college enrollment data from the National Student Clearinghouse (NSC), which provides these data in exchange for payment. There is no respondent burden for collection of content from OSP-participating private schools’ websites.

7 This is the cost for the full study, but two data collection activities were not included in Exhibits A4a and A4b (that is, OSP private school website content and college enrollment data) because there is no burden associated with them. See Exhibit A2 for further detail.)