Final REL OMB Part A 062111

Initially formed in the 1960s, the primary mission of the Regional Educational Laboratories (RELs) is to serve the educational needs of 10 designated geographical regions, using applied research, development, dissemination, and training and technical assistance to bring up-to-date, rigorous research and proven practices into educational improvement efforts. Since their founding, the RELs have conducted applied research and development designed to serve their regions. Under the Education Sciences Reform Act of 2002 (ESRA), the RELs were specifically funded to provide expert advice, training, and technical assistance pertaining to the goals of the Elementary and Secondary Education Act (ESEA), including measuring academic performance in reading and mathematics in grades three through eight, identifying weaknesses, and making appropriate changes to improve student achievement. To meet the needs of state education agencies (SEAs), local education agencies (LEAs), and schools related to ESEA, the RELs are charged with using the highest quality evidence in providing technical assistance. Where such evidence is not available and schools need valid and reliable information on strategies to improve learning, the RELs are expected to conduct relevant research and development activities. The long-standing RELs program is currently authorized by ESRA and administered by the National Center for Education Evaluation and Regional Assistance (NCEE).

Overview of the Evaluation

While the U.S. Department of Education (ED) supported several qualitative descriptions and evaluations of the RELs in the 1990s (see Kober et al. 1996; Spencer and Stonehill 1999; Turnbull et al. 1994; Vinovskis 1993), few independent data are available on the performance of the laboratories. The last time REL products were systematically reviewed by outside experts was in the Interim Evaluation completed in 1999. This evaluation will fill an important gap in information on the quality of the products and services provided by the RELs and the extent to which they are meeting the needs of the education agencies in their regions.

The evaluation involves several components. First, to assess the technical quality and relevance of one of the RELs’ technical assistance products, Fast Response Projects (FRPs), specifically Technical Briefs and Issues and Answers proposals and reports that were completed in each of the 10 RELs between the start of the grant period and December 1, 2009, we downloaded FRP reports from the web and obtained FRP proposals submitted to ED by the RELs from the NCEE project officer. We distributed the final reports and their corresponding proposals to members of an expert panel for review. The expert panels rated the technical quality and relevance of the FRPs. The panels comprised individuals with relevant content and/or methodological expertise. A parallel process will be used in winter/spring 2011 to rate the Rigorous Applied Research and Development (RARD) Studies. We are also conducting a review of other REL documents to describe the ways in which the RELs have met the 10 missions outlined by Congress. To further address issues of relevance and also explore usefulness, we will administer a survey of REL customers, including users and potential users.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

Research Questions

As one component of a multi-part evaluation, NCEE plans to fund a REL customer survey, which will address the following key questions through a web-based survey of educators and policymakers in state and local education agencies:

How relevant are the REL technical assistance products and activities to the needs of the states, localities, and policymakers in their regions?
How useful have the REL technical assistance products and activities been to the states, localities, and policymakers in the regions?
How aware are state and local educational agency officials of the products and activities of the RELs?

Respondents

We will collect data to address the three study questions listed above from employees of SEAs and LEAs in specific policymaking or administrative positions. These may include both users and potential users of REL services and products.

Instrument

The survey will include items on (1) the respondent’s research and technical assistance needs, (2) the respondent’s familiarity and satisfaction with the REL’s products and services, and (3) a respondent profile.

Respondents to the survey will include both actual users and potential users of REL services and products. Surveying potential users instead of only actual users will allow us to glean valuable information about the target audiences’ awareness of REL services and products, their needs, the reasons for nonuse, and intended future use. Skip patterns within the web-based survey will navigate potential users to relevant items only. A draft of the survey is found in Appendix A. The survey will occur only once.

A.3 Use of Improved Information Technology to Reduce Burden

We will administer the survey on the web, so it is easily accessible to respondents. Burden will be reduced with the use of skip patterns and prefilled information based on responses to previous items when appropriate. When needed, paper and phone survey options will be offered to respondents as part of the nonresponse follow-up effort.

A.4 Efforts to Identify and Avoid Duplication

A survey of this nature has never been administered to SEA and LEA employees. SEA and LEA employees are being surveyed about REL reports as part of a separate analytic and technical support contract (ATS). However, the survey under the ATS contract primarily includes items on respondents’ familiarity with specific reports produced by the RELs. The survey included in this current submission covers different topics in an attempt to glean respondents’ level of familiarity with the RELs, their technical assistance and research needs, and overall satisfaction with the RELs. In addition, because Congress requires an evaluation with results reported by REL, the sample for this evaluation will be stratified by REL, which the ATS survey was not. Therefore, while some of the same types of respondents are being surveyed under both contracts, the surveys cover different topic areas and use different sample designs.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. Every effort has and will be made to minimize the burden on SEA and LEA employees. As noted below, the survey will take an average of 10 minutes.

A.6 Consequences of Less-Frequent Data Collection

The data collection will occur only once. If the data collection is not completed, OMB, administrators, policymakers, and the public will not know whether the REL program is performing effectively; that is, if potential users are aware of the RELs and if products and services of the RELs are useful and relevant to the needs of state and local administrators and policymakers.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

The 60-day Federal Register notice was published on November 24, 2010 and the 30-day notice was published on March 25, 2011.

A Technical Work Group (TWG) met in March and November 2010 to discuss the evaluation design and data collection. Their input led to changes to the sample design and survey. Participating members included:

Judy Arter, ETS;
Gregg Jackson, George Washington University;
Conrad Katzenmeyer, University of Central Florida;
Larry Ludlow, Boston College;
Colleen Serement, Maryland State Department of Education; and
Deb Sigman, California Department of Education.

A.9 Payments to Respondents

There will be no payments made to survey respondents.

A.10 Assurance of Confidentiality

The collection of information in this evaluation is authorized by Public Law 107-279 Education Sciences Reform Act of 2002, Title 1, Part D, Section 174. Participation is voluntary.

Other than the names and contact information for the respondents, which is information typically already available in the public domain (i.e., state and district websites) no data collected for this survey will contain personally identifiable information. No names and contact information will be released.

ED, in the conduct of the study, will follow procedures for ensuring and maintaining participant privacy, consistent with the Education Sciences Reform Act of 2002. Title I, Part E, Section 183 of this Act requires, “All collection, maintenance, use, and wise dissemination of data by the Institute” to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. Respondents were assured that confidentiality was maintained, except as required by law. Specific steps to guarantee confidentiality include the following:

Identifying information about respondents (e.g., respondent name, address, and telephone number) will not be entered into the analysis data file, but will be kept separate from other data and will be password protected. A unique identification number for each respondent will be used for building raw data and analysis files.
In emails, participants will be referred to by unique identification number. School districts will be referred to by identification number. Files containing more information will be password protected.
A fax machine used to send or receive documents that contain confidential information will be kept in a locked field room, accessible only to study team members.
Confidential materials will be printed on a printer located in a limited access field room. When printing documents that contain confidential information from shared network printers, authorized study staff will be present and retrieve the documents as soon as printing is complete.
In public reports, findings will be presented in aggregate by type of respondent (e.g., SEA personnel) or for subgroups of interest (e.g., individuals with certain years of work experience). No reports will identify individual respondents or local agencies.
Access to the sample files will be limited to authorized study staff only; no others will be authorized such access.
All members of the study team will be briefed regarding confidentiality of the data.
A control system will be in place, beginning at sample selection, to monitor the status and whereabouts of all data collection instruments during transfer, processing, coding, and data entry. This includes sign-in/sign-out sheets and the hand-carrying of documents by authorized project staff only.
All data will be stored in secure areas accessible only to authorized staff members. Computer-generated output containing identifiable information will be maintained under the same conditions.
When any hard copies containing confidential information are no longer needed, they will be shredded.

A.11 Questions of a Sensitive Nature

The questions included on the data collection instruments for this study do not involve sensitive topics.

A.12 Estimates of Respondent Burden

In all, responses will be required one time from 11,760 respondents (240 SEA officials and 11,520 LEA officials). We estimate that it will take respondents an average of 10 minutes to complete the survey, so total burden is 117,600 minutes or 1,960 hours (see Exhibit A-1 below).

Exhibit A-1. Estimates of Respondent Burden

Respondent	Anticipated number completed	Minutes per completion	Burden in minutes	Burden in hours	Burden in Dollars
	(a)	(b)	(c) a x b	c/60
SEA official	240	10	2,400	40	$2,080
LEA official	11,520	10	115,200	1,920	$99,840
Total burden			117,600	1,960	$ 101,920

NOTE: Assumes an hourly rate of $52 per hour. This hourly rate is based on average daily rates for administrators obtained from the survey “Salaries and Wages Paid Professional and Support Personnel in Public Schools” (http://www.edweek.org/media/43ers-data.pdf).

A.13 Estimates of the Cost Burden to Respondents

There are no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the information.

A.14 Estimates of Annualized Government Costs

The amount for the design, completion, analysis, and reporting on the survey of REL customers is $496,118, and the annual cost is $172,927 in FY 2010, $209,030 in FY 2011, and $114,161 in FY 2012. The annualized cost is $165,373.

A.15 Changes in Hour Burden

This is a first-time submission.

A.16 Time Schedule, Publication, and Analysis Plan

Time Schedule

The schedule shown below in Exhibit A-2 displays the sequence of activities required to conduct this information collection activity and includes key dates for activities related to instrument design, data collection, analysis, and reporting.

Exhibit A-2. Time Schedule

Activities and deliverables	Date
Instrument design	Summer 2010
Conduct survey	March -May 2011
Analyze survey data	June-September 2011
Draft report	2011
Final report	2012

Analysis Plan

The survey data will be used to address the research questions posed by ED. This will be accomplished through descriptive statistics, inferential subgroup comparisons, as well as factor analysis. Specifically, once the data from the web-based survey have been collected, coded, and cleaned and sampling weights applied, relevant descriptive statistics (e.g., mean, medians, ranges, percentages) will be generated for each item in the survey along with corresponding variance estimates. Subgroup comparisons, for example, ANOVAs with post-hocs, will be used to examine differences across regions. In addition, factor analysis will be used to determine whether latent usefulness and relevance constructs were captured in the survey and, if so, to generate continuous relevance and usefulness scores. An average of the relevance and usefulness ratings will be generated to arrive at an overall rating of relevance and an overall rating of usefulness, respectively, for each REL. The project team will include a description of research methods as well as results in the evaluation report.

A.17 Display of Expiration Date for OMB Approval

The Institute of Education Sciences is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instrument. The data collection instrument will display the expiration date for OMB approval.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-02-01