OMB package part B 07 10 08

U.S. Department of Education

Impact Evaluation of the DC Opportunity Scholarship Program

Office of Management and Budget

Statement for Paperwork Reduction Act Submission

Part B: Collection of Information Employing Statistical Methods

Contract ED-04-CO-0126

July 10, 2008

TABLE OF CONTENTS

PART B Collection of Information Employing Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Information Collection Procedures 1

B.3 Methods to Maximize Response Rates 2

B.4 Tests of Procedures 3

B.5 Individuals Consulted on Statistical Aspects of Design 3

PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This package represents a request for approval to collect one additional round of data – in spring 2009 – for the Impact Evaluation of the DC Opportunity Scholarship Program (OSP), using the same evaluation design, instruments, and data collection and analysis procedures previously approved by OMB as OMB No.1850-0800 (initial approval notice dated 4/15/05; extension beyond the original three-year expiration date approval notice dated 1/25/08). The only substantive deviation from the earlier submissions (aside from the new burden) is a request to increase respondent payments for this final round of data collection beyond what was last approved by OMB (NOC 1/10/06 and NOA 4/24/07).

This additional data collection was not anticipated when ED submitted the package to extend data collection by 3 months (to complete the already approved data collection) or when ED acted to discontinue approval for this evaluation’s collection beyond the new 7/31/08 expiration date (NOA 5/22/08). It is made possible by the recent House committee vote (June 25) to appropriate an extra year of funds (FY 2009) for the DC School Choice Incentive Act, providing money for both scholarships and the evaluation.¹ This request for OMB approval to administer an additional round of student testing and student, parent, and principal surveys reflects policymakers’ interest in collecting longer-term information about OSP participants, providing the evaluation the ability to estimate fourth year program impacts.

Since we are submitting an already-approved design and set of instruments and procedures, this package is identical to the package approved by OMB on 1/25/08 with the exceptions described below. We have proceeded in this manner so that this package can function as a stand-alone document reflecting the full evaluation. Throughout this document, any difference between the recently approved package and the current one are shown in bold, to assist OMB in its review. The differences are:

• For every occurrence of the number of years of data collection, we have updated the language to say “4 years of follow up data collection for each cohort.” (See pages 1 and 2.)

• We added an additional method to maximize response rates to Section B3 (see page 3).

B.1. Respondent Universe and Sampling Procedures

The Impact Evaluation of the DC Opportunity Scholarship Program will be based on administering assessments and surveys to the universe of program applicants and their parents as well as school principals. The surveys that will provide information on schools’ competitive response to the program will also include the universe of DCPS principals and of private school principals. Therefore, there is no sampling proposed for this study.

B.2. Information Collection Procedures

Student Assessments

We plan to administer the SAT-9 math and reading assessments when the treatment and control group families come in to renew their eligibility for the Program, so that the test administration will be similar across all types of evaluation members. The scholarship users will clearly be the most motivated to attend and we will be conscious of the need to take steps to encourage the scholarship non-users (decliners) and control group members to fulfill the requirements to participate in the evaluation’s data collection. These assessments will be administered in early April of each year, for the four years of follow up data collection for each cohort.

School Records

Administrative records will be collected from DCPS and charter school authorizers to obtain data on attendance, persistence, disciplinary actions, and grades for members of the treatment and control groups at baseline. In addition, Westat will seek to obtain these data for all public school students, including those in charter schools, so that the program applicants can be compared to other students in the relevant grade levels, as required by the DC Choice Act.

Parent Surveys

The legislation requires the evaluation to examine the impact of the program on parents. These surveys will be administered to the parents when they come in to renew their child’s program eligibility, with telephone follow up as necessary, for the four years of follow up data collection for each cohort.

Student Surveys

The study will conduct surveys of treatment and control group students who are in grades four and above. The surveys will be administered at the same time (and place) as the student assessments – the family events where the parents come in to renew program eligibility, for the four years of follow up data collection for each cohort.

Principal Surveys

Surveys will be mailed to principals of all private schools in DC and principals of all public and charter schools in DCPS. The surveys will be mailed in the spring, with telephone follow up, for the four years of follow up data collection for each cohort.

B.3. Methods to Maximize Response Rates

We will maximize the response rate for this portion of the study both by distributing survey instruments that are fairly easy for respondents to complete and by following up with non-responders by mail, fax, and telephone. Although the primary respondents are quite disadvantaged (eligibility requirements include family income less than or equal to 185 percent of poverty), this study is striving for a response rate of 80 percent.

Obtaining high response rates in the Impact Evaluation of the DC Opportunity Scholarship Program will be critical to the success of the study. It will be particularly important to obtain response rates that are not only high overall, but that are approximately equal in the treatment and control groups. This will be challenging due to the fact that while most of the treatment group will presumably be in a relative small set of participating private schools, control group students will likely attend a large number of different DCPS schools, and the identities of these schools will not be known in advance.

We have several strategies for ensuring a high rate of response. First, we have planned to conduct most of the data collection—student assessments, student and parent surveys—at events the program operator will hold for the treatment and control groups to re-establish eligibility for the program. Second, because of a key provision in the law, in our communications with parents we can stress that participation in the evaluation’s data collection is required for students to keep their scholarship or remain eligible to receive a scholarship in the future. We believe these requirements will be a formidable incentive to respond to the surveys and assessments. We will employ a sophisticated tracking system to ensure that we follow up with non-response in a timely and comprehensive way. Finally, we use respondent payments to help make participation in the data collection events on Saturdays and evenings sufficiently attractive to the treatment and control group.

B.4. Test of Procedures

We pre-tested each of the surveys with nine or fewer people who were similar demographically to respondents in the study. We asked the pretest respondents to first complete the relevant survey and then participate in a focus group about it. In the focus group discussions we tested for completion times and feelings of burden, salience of language, concept recognition, and understanding of terms. After the pretests, we revised the surveys as needed based on the pretest results.

B.5. Individuals Consulted on Statistical Aspects of Design

The statistical aspects of the design have been reviewed thoroughly by staff at the Institute of Education Sciences, as well as by members of the study’s expert panel (listed in Part A, section A8). Table 1 shows the individuals most closely involved in developing the statistical procedures.

Table 1. Individuals Involved in this Project
Name	Affiliation	Role	Phone Number
Babette Gutmann	Westat	Project Director	(301) 738-3626
Patrick Wolf	University of Arkansas	Principal Investigator	(479) 575-2084
Mike Puma	CRA	Senior Analyst	(410) 897-4968
Juanita Lucas-McLean	Westat	Director of Data Collection	(301) 294-2866
Marsha Silverberg	ED/IES	Economist, COR	(202) 208-7178

1 If, after OMB grants approval for this new round of data collection, the funds to conduct the collection ultimately fail to be appropriated by the full Congress, ED will formally terminate the approval.

2