RFIS 2008-09 Draft OMB Section A

U.S. Department of Education

Reading First Implementation Study:

2008-2009

Section A

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

Revised August 8, 2008

Contents

Introduction A-1

Overview A-1

Data Sources to Address Evaluation Questions A-2

Part A. Justification A-3

A.1 State Personnel Telephone Interviews A-3

A.2 Federal Register Announcement A-5

A.3 Consultations Outside the Agency A-5

Part B. Collection of Information Employing Statistical Methods B-1

B.1 Respondent Universe and Sample Size B-1

B.2 Procedures for Collection of Information B-1

B.3 Methods for Maximizing Response Rate and Dealing with Nonresponse B-1

B.4 Test of Procedures and Methods B-1

B.5 Consultations on Statistical Aspects of the Design B-2

Appendix A: Federal Register Notice

Appendix B: State Interview Protocol

Appendix C: Reading First Legislation

Introduction

This document presents the first Supporting Statement for the Reading First Implementation Study: 2008-09, which is being conducted by Abt Associates, in collaboration with Hezel Associates, for the U.S. Department of Education’s Policy and Program Studies Service.

The purpose of this OMB package is to address the state personnel interviews that comprise the only direct data collection activity of the Reading First Implementation Study: 2008-09. These interviews will be conducted with the Reading First Directors, Title I Directors, and Reading/Literacy Directors in each of the 54 states and jurisdictions that have received Reading First funding. This OMB package addresses a data collection activity that is time-sensitive because the recent budget cuts to the Reading First program necessitate a timely turn-around of OMB approval for these interviews. Conducting interviews prior to changes in funding is critical in order to document states’ plans for addressing budget cuts so that ED can provide timely technical assistance to states. The interviews will also collect data on states’ overall literacy policies and programs, pre-budget cut process for selecting instructional materials and assessments, professional development programs, “spill-over” to non-RF districts and schools, and integration with other reading programs.

This OMB package also provides a brief introduction to the entire study and additional extant data collection elements.

The Reading First Implementation Study: 2008-09 is designed to advance our understanding of the implementation and long term sustainability of this signature program. This OMB Package focuses only on Question 1 (Question 2 will be addressed using extant data); the study’s two evaluation questions are listed below:

1. What are states’ planned responses to the Reading First budget reduction, and which RF program elements do state-level staff believe can be sustained beyond Reading First?

2. How does Reading First students’ reading achievement compare to other Title I students when they complete first through sixth grades?

Overview

The No Child Left Behind Act of 2001 (PL 107-110) established the Reading First Program (Title I, Part B, Subpart 1), a major federal initiative designed to help ensure that all children can read at or above grade level by the end of third grade. With the exception of almost $12 billion in Title I funding, which supports general reading activities for children in low-income schools, RF is substantially larger, and both more ambitious and explicit in its guidance than any previous school-based early literacy initiative undertaken in the U.S.

The Reading First Program has been in place for five years, and at present, there are nearly 6,000 Reading First schools in approximately 1900 school districts charged with operationalizing the underlying principles and strategies described above. To date, five major evaluation reports on Reading First have been released.¹ Findings from these studies provide evidence that reading instruction is changing in ways consistent with key program goals and strategies. In particular, The Reading Implementation Evaluation: Interim Report found that the overall time spent teaching reading has increased, reported instructional practice aligns with scientifically based reading practices, teachers report using results of reading assessments to guide instruction, and reading intervention services are being targeted towards struggling readers.

Data Sources to Address Evaluation Questions

Interviews with state personnel are primarily focused on answering the first research question. These phone interviews will be conducted in all 54 states and jurisdictions that have received Reading First funding, and will include a phone interview with each of the following individuals in every state/jurisdiction: State Reading First Director, State Title I Director, and State Reading/Literacy Directors. In some states, one individual occupies more than one of these positions. Interviews with state personnel will be conducted in the fall of 2008; ED may exercise an option to conduct a second round of interviews in fall 2009.

Below, we present a list of topics and sample questions for the state personnel interviews. Some will be asked of all three respondents, while others will be asked only of the respondent who is likely to be responsible for the topic area.

• Management and Leadership. Background and responsibilities of the state Reading First Coordinator/Title 1 Director/Reading-Literacy Director.

• RF Funding at the State Level. The number of rounds of RF funding competition; changes in RF school or district participation; plans for future RF awards; staff stability or turnover.

• Reading First Student Achievement Data. Procedures for collecting, maintaining, managing, analyzing, using and disseminating RF student achievement data.

• Technical Assistance from State Reading First Program. State RF Office assistance provided to RF district and school staff before budget cuts were announced. Role of RF coaches in statewide reading program.

• Outreach and Communication with Schools and Districts. Structure of outreach to RF and non-RF schools and districts; topics in which non-RF school are most interested.

• Long Term Sustainability of Reading First. Aspects of the RF program that are likely to persist post-RF funding.

Part A. Justification

1. Explain the circumstances that make data collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information

Findings from earlier studies provide evidence that reading instruction is changing in ways consistent with key Reading First program goals and strategies.² However, information has not been collected from state-level Reading First directors since the program’s funding has been reduced. In order to document the planned responses to this budget cut, it is critical to conduct state personnel interviews that will document states’ pre-budget cut processes for selecting instructional materials and assessments, professional development, “spill-over” to non-RF districts and schools, and RF influence on overall state literacy policies and programs. Information from these interviews will allow ED to plan appropriate technical assistance to states.

The Reading First legislation (PL 107-110, Title I, Part B, Subpart 1, Section 1205) outlines the requirements for national evaluation activities.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection

Information on the implementation of the Reading First program will be collected by Abt Associates, Inc. (Abt) and by its subcontractor, Hezel Associates (Hezel) under contract number ED-04-CO-0015 with ED. The data will be analyzed by Abt and Hezel. The specific data to be collected will be obtained from telephone interviews with RF state coordinators, Title I coordinators, and state Reading/Literacy Directors.

These data will be maintained in a secure manner (described below, in response to Item 10), and neither respondents nor states will be publicly identified. In reports and publications, data will be aggregated to a group level in statistical models (interviews with the three state personnel will be aggregated for each state). The purpose for sharing information through publications will be to inform the public about the effectiveness of the approach being tested in this study. Information will not be used to identify states, districts, schools, or individuals for any purpose.

A.1 State Personnel Telephone Interviews

State administration of the RF program and other state reading initiatives may influence the level and effectiveness of implementation at the district and local levels. Collecting data on states’ RF program administrative structures and processes as well as their relationship with other reading initiatives at the state level, particularly those funded by Title I, will provide more systematic information about how states are continuing to implement Reading First in the face of substantial reductions in funding.

We will conduct telephone interviews with state personnel in the fall of 2008. These interviews will help us obtain information about planned response to the RF program budget cuts, the criteria and processes the states used to select the RF district sub-grantees, as well as about key district and state policies that affect reading instruction—in both RF and comparison schools. Additionally, interviews will provide information on the relationship between Reading First and other state reading initiatives (including Title I).

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. Wherever possible, information will be gathered from existing data sources rather than imposing additional burden by collecting primary data. State personnel interviews will be conducted by telephone at a time convenient for the respondent. No online survey tools will be used.

4. Describe efforts to identify duplication. Show specifically why any similar information available cannot be used or modified for use for the purposes described in item 2 above

In preparation for the state personnel interviews, we will review state websites, state assessment information (the SEDL database), and other sources of available information, to prepare a state summary profile. This will prevent collecting any duplicate information, and will allow us to use interviews to verify and update the information we already have. Specifically, state interviews will provide information on states’ planned responses to the reduction of Reading First funds; how Reading First (RF) funds have been targeted to schools and districts; how SEAs have used Reading First data; the relationship between Reading First and other state reading initiatives (including Title I); and the sustainability of state- and district-level RF implementation.

These phone interviews will be conducted in all 54 states and jurisdictions that have received Reading First funding, and will include a phone interview of the following individuals (noting that in some states, one individual occupies more than one of these positions): State Reading First Director, State Title I Director, and State Reading/Literacy Directors.

The information to be collected as part of the state personnel interviews will not be available elsewhere as it will represent the individual experiences of each of state/jurisdictions’ Reading First Director, Title I Director, and Reading/Literacy Director—information that is not currently available elsewhere.

5. If the collection of information impacts small businesses or other small entities (item 5 of OMB Form 83-1), describe any methods used to minimize burden

The primary entities for this study are state-level personnel. Burden is reduced for all respondents by requesting only the minimum information required to meet the study objectives.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

Budget cuts are anticipated to become effective for Reading First State grantees in Fiscal 2009, and in order to learn about states’ plans for their programs, the interviews must be conducted as soon as possible so that ED can plan appropriate technical assistance.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner: *requiring respondents to report information to the agency more often that quarterly; *requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it; *requiring respondents to submit more than an original and two copies of any document; *requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years; *in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study; *requiring the use of a statistical data classification that has not been reviewed and approved by OMB; *that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; *or requiring respondent to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

There are no special circumstances that would cause collection of information to be conducted in any manner listed under section seven of the OMB guidelines.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden. Consultation with representatives of those from who compile records should occur at least once every 3 years—even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained

A.2 Federal Register Announcement

In accordance with the Paperwork Reduction Act of 1995, a 60-day notice to solicit public comments was published in the Federal Register on Month XX 2008 (p. XXXX). No comments have been made to date. A copy of the Federal Register Notice is attached in Appendix A.

A.3 Consultations Outside the Agency

Consultations on the research design, sample design, data sources and needs, and study reports have occurred during the study’s design phase and will continue to take place throughout the study. The purpose of such consultations is to ensure the technical soundness of the study and the relevance of its findings, and to verify the importance, relevance, and accessibility of the information sought in the study.

Abt Associates Inc. and its subcontractor Hezel Associates have provided substantial input to ED for the study. Senior technical staff from these organizations who are conducting the study are listed below:

Abt Associates Inc. Beth Boulay 617-520-2903

Beth Gamse 617-349-2808

Alyssa Rulf Fountain 617-520-2657

Fatih Unlu 617-520-2528

Hezel Associates Fran Hurley 315-422-3512

Kathe Simons 315-422-3512

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payments or gifts will be provided to state coordinators for participating in interviews.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy

Abt will follow procedures for ensuring and maintaining confidentiality, consistent with the Privacy Act Section 552 of Title 5 of the United States Code, the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, which requires all collection to conform to the requirements of the Privacy Act, as well as the confidentiality standards of the Protection of Pupil Rights Amendment, subsection (c) of this section and sections 444 and 445 of the General Education Provision Act (20 USC 1232h). We will also follow procedures consistent with the Federal common rule or

Department final regulations on protection of human research subjects.³ As detailed below, the system of records that will be established will be in accordance with Privacy Act stipulations, such that no individual will be identifiable by his/her data, as described in greater detail below.

Data to be collected will not be released with individual identifiers. Data will be presented in aggregate statistical form only (data from the interviewed state personnel aggregated by state). Additionally, individual state data will only be reported with an anonymous identification code so that states are not individually identifiable. All Abt and Hezel data collectors will be knowledgeable about confidentiality procedures and will be prepared to describe them in full detail, if needed, or to answer questions raised by participants. Participants will be assured that all information identifying them or their agency will be kept confidential in compliance with the legislation (P.L. 103-382).⁴ Prior to consenting to participation, subjects will be informed that information about the study will ultimately be disseminated in reports and publications but that such information will not contain individual identifiers (such as state, agency, or individuals’ names) and that information will be presented in a way that will not make individual identification possible.

The following safeguards are routinely employed by Abt to carry out confidentiality assurances:

• Access to sample selection data is limited to those who have direct responsibility for providing the sample and maintaining sample locating information. At the conclusion of the research, these data are destroyed.

• Unique study identification numbers are assigned to subjects (in this case, interviewed state personnel) that enable the study team to link a subject’s data to his/her identification number rather than his/her name and state. Identifying information is maintained on separate forms and files, which are linked only by sample identification number.

• Access to the file linking sample identification numbers with participants’ identification and contact information is limited to a small number of individuals who have a need to know this information.

• Access to the hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded material is shredded.

• Individual state names and state respondents’ names do not appear in reports produced for the study, and no reports will describe states such that they could be identified.

• No information reported to the Federal Government or submitted for publication will contain information that could be used to identify individual states or state-level respondents.

• Computer data files are protected with passwords, and access is limited to specific users. With especially sensitive data, the data are maintained on removable storage devices that are kept physically secure when not in use.

11. Provide additional justification for any questions of a sensitive nature such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

No questions of a sensitive nature will be asked. Please see State Interview Protocol (Appendix B).

12. Provide estimates of the hour burden of the collection of information. The statement should: *Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices. *If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-1. *Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriated wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

Exhibit 1.1 presents estimates of the reporting burden for conducting the state interview data collection. Time estimates are based on experience with similar instruments in similar studies. There are no direct monetary costs to respondents other than their time to participate in the study.

Exhibit 1.1 Respondent Burden—Telephone Interviews with State Personnel
Respondent	Number of Respondents (fall 2008)*	Number of Respondents (Optional ,fall 2009)*	Time per Response	Total Hours
RF Coordinator	54	54	1.0 hours	54
Title I Coordinator	54	54	1.0 hours	54
State Reading/Literacy Director	54	54	1.0 hours	54
Total	162	162		162
Note: The 54 respondents include the 50 states, the District of Columbia, the Virgin Islands, American Samoa, and the Bureau of Indian Affairs. Should Puerto Rico be awarded a Reading First grant, interviews with relevant state personnel will be incorporated into the study’s data collection. * At present, ED plans to collect interview data only once, in the fall of 2008, with an option to collect data in fall 2009.

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.) The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services components. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life or capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling, and testing equipment; and record storage facilities. *If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10). Utilize the 60-day pre-OMB submission public comment process and use existing economic and regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate. Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices

There are no additional respondent costs associated with this data collection other than the hour and cost burden estimated in item 12.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

Exhibit 1.2 displays the estimated total annual cost to the Federal Government, as well as the costs for the state personnel interview data collection activities.

Exhibit 1.2 Study Year (Dates)
	Total Study Costs	State Personnel Telephone Interviews
Year 1 (10-01 2007---09-30-2008)	$584, 978	$193,628
Year 2 (10-01 2008---09-30-2009)	$832,289	$167,058
Year 3 (10-01 2009---06-30-2010)	$356,017	$0
Total	$1,773,292	$360,686

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1

This is a new study.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Findings from the state-level interviews described in this OMB package will be summarized in an interim report. Findings from state personnel interviews will be summarized to address: 1) Reading First funding patterns across states; 2) the uses of Reading First data; 3) outreach to non-RF districts and schools; 4) technical assistance from the state Reading First program; 5) Outreach and communication with schools and districts; 6) Diversity of reading programs within states; and 7) planned responses to the 61 percent budget reduction in the program, including program sustainability.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate

The OMB approval expiration date will appear in all materials distributed to participants in the study.

18. Explain each exception to the certification statement identified in Item 19, “certification for Paperwork Reduction Act Submissions,” of OMB Form 83-1

There are no exceptions to the certification statement.

1 The five reports include Keeping Watch on Reading First; and Reading First: Locally Appreciated, Nationally Troubled, both released by the Center on Education Policy; Reading First Implementation Evaluation: Interim Report; Reading First Impact Study: Interim Report, and The Analysis of State K-3 Reading Standards and Assessments, all released by the U.S. Department of Education.

2 The four reports include Keeping Watch on Reading First; and Reading First: Locally Appreciated, Nationally Troubled, both released by the Center on Education Policy; Reading First Implementation Evaluation: Interim Report, and The Analysis of State K-3 Reading Standards and Assessments, both released by the U.S. Department of Education.

3 REL-SE follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). REL-SE will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team. All institution-level identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. REL-SE obtains signed NCEE Affidavits of Nondisclosure from all employees, subcontractors, and consultants that may have access to this data and submits them to our NCEE COR.

4 The legislation provides that no person may use “individually identifiable information furnished under this title for any purpose other than a statistical purpose, make any publication whereby the data furnished by any particular person under this title can be identified, or permit anyone other than the individuals authorized by the Commissioner to examine the individual reports.”

Table of Contents 2