Generic Clearance Template

Appendix D

Generic Clearances

DOCUMENTATION FOR THE GENERIC CLEARANCE

OF CUSTOMER SERVICE SATISFACTION COLLECTIONS

TITLE OF INFORMATION COLLECTION: Evaluation of the Relevance and Utility of National Center for Education Evaluation (NCEE) Products: Regional Educational Laboratories Fast Response Research reports (10-day review request)

[X ] SURVEY [ ] FOCUS GROUP [ ] SOFTWARE USABILITY TESTING

DESCRIPTION OF THIS SPECIFIC COLLECTION

1. Intended Purpose

This submission is a request for approval of data collection activities that will support an evaluation of research reports produced by Regional Educational Laboratories (RELs) under contract to the U.S. Department of Education (ED), through the Institute of Education Sciences (IES) and its National Center for Education Evaluation and Regional Assistance (NCEE). The overall intent of the reports is to provide educators and policymakers with credible scientifically based research on a regular basis. Fast Response reports are REL research reports disseminated as two ongoing web-based series—Issues & Answers and Technical Briefs. These reports can be accessed on the REL home page on the IES website.

The objective of the evaluation is to estimate the extent to which the research reports are perceived as relevant and useful to educators and policymakers who access the reports.

IES is conducting the evaluation as part of a larger contract entitled the Analytic and Technical Support for Advancing Education Evaluations, hereafter referred to as ATS. Mathematica Policy Research is conducting the evaluation.

In this package, we are requesting approval to conduct the following data collection:

A web survey of educators and policymakers who are potential Fast Response Issues & Answers and Technical Briefs report users to ascertain the extent to which the reports address their intended goals.

2. Need for the Collection

Under the IES authorizing legislation, the Education Sciences Reform Act of 2002, Section 171 (b) states that NCEE’s mission shall be “To provide technical assistance; To conduct evaluations of Federal education programs administered by the Secretary (and as time and resources allow, other education programs) to determine the impact of such programs (especially on student academic achievement in the core academic areas of reading, mathematics, and science); To support synthesis and wide dissemination of results of evaluation, research, and products developed; and to encourage the use of scientifically valid education research and evaluation throughout the Unites States.”

The evaluation is essential to identify the extent to which RELs are disseminating syntheses of findings from NCEE-supported research and determining that the information is perceived as relevant and useful to its intended audience, specifically education practitioners, researchers, and policymakers. The goal of the Survey of the REL Research Reports is to collect information from potential report consumers to determine their familiarity with the reports and the extent to which they consider the information presented in the reports relevant and useful to their work (Attachment A).

The survey is designed to address the following primary research questions:

• To what extent are Fast Response reports reaching the education practitioners and policymakers for whom the reports are intended?

• To what extent are targeted audiences of Fast Response reports using the information presented in the reports, and how do they perceive the reports as relevant to their information needs and useful for their decision making or practice?

• For professed users of the reports, what kinds of information presented in the reports are perceived as relevant to sample members’ work, and for what purposes have respondents used the information in their work?

The evaluation also will address the following secondary research questions:

• To what extent do report users differ in the types of work that they do, their use of education research, and their reported knowledge of the topics covered in the reports?

• How does the perceived relevance and reported use of information vary by attendees’ job type and prior knowledge?

3. Planned Use of the Data

The ATS study’s data collection will give ED and IES useful information for effectively targeting and meeting the needs of NCEE stakeholders. The survey data will enable IES and NCEE to determine the types of reports or research findings that users find most useful and relevant, learn how NCEE-supported research and products are used, and identify research and products needed in the future. The data will enable NCEE to better serve the informational needs of its target audiences by bringing the latest and best research and proven practices into school improvement efforts, especially in reading, mathematics, and science.

4. Date(s) and Location(s)

Data will be collected during the first quarter of 2010. The following table shows the time line for this study.

Schedule of Activities

Activity	Schedule
Prepare sample of potential report users	December 2009
Conduct Survey	January–March 2010
Analysis and report of findings	April 2010

5. Collection Procedures

To answer the primary and secondary research questions, Mathematica will first develop a sampling frame of potential users of Fast Response reports (described more fully below). Sample members will receive an email invitation to participate in a study being conducted by ED. The invitation will include a link to a short, closed-ended, web-based questionnaire. To improve the response rate, Mathematica will send a follow-up email request to non-respondents two weeks after the initial email invitation; we will send a second reminder three weeks after the initial email. Approximately one month after the first email Mathematica will conduct telephone follow-up interviews with web-survey non-respondents.

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. To conduct the Survey of the REL Research Reports, we will use a web-based data collection method programmed to accept only valid responses and to check for logical consistency across answers. Respondents will thus be able to correct any errors as they complete the survey, minimizing the need for later contacts to obtain missing data or clarify inconsistent data. An added advantage of web-based data collection is that respondents may complete the survey at their convenience. An email invitation sent to sample members will contain a URL link to the web-based survey and a unique user ID and password (Attachment B).

Individuals who choose not to respond to the web-based survey will be able to print a portable document format (PDF) version from the web for faxing or mailing to Mathematica. Respondents may also request participation through two other modes: (1) by standard mail and (2) by telephone. It is important to offer these other modes of response to make the survey as convenient as possible, thus increasing the response rate. Sample members who have not completed the survey will receive two email reminders encouraging them to respond; the names of subsequent non-responders will be sent to Mathematica’s Survey Operations Center for telephone followup. For sample members with questions about the study, all email communications will include access to Frequently Asked Questions (FAQs) (Attachment C) along with a project-specific email address and a toll-free telephone number.

This is a voluntary survey and sample members may choose whether or not to participate. Also, they may choose to not answer specific questions included on the questionnaire. All of the responses from the survey will be kept strictly confidential. The project will adhere to the guidelines outlined in the Mathematica Security Manual to protect survey participants to the extent the law allows. Sample members will be assigned unique identification numbers to ensure confidentiality, and no names will be attached to any responses. Mathematica has rigorous requirements in place to protect the security of the electronic and paper survey responses. The following statement will be used to describe data confidentiality to sample members:

Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific individual. We will not provide information that identifies you to anyone outside the study team, except as required by law.

a. Number of Focus Groups, Surveys, Usability Testing Sessions

This request is for one data collection—a web survey. Up to nine people will be contacted to pre-test the web questionnaire.

b. Description of Respondents/Participants

The target population of this survey includes four groups, each of which will be sampled from different frames. The groups are:

1. Local public school policymakers

2. Those who have attended REL events

3. Those who have attended other IES events

4. Chief State School Officers (or members of their staffs)

The sample of local public school policymakers will be identified by their district superintendents as the people whose positions makes them most likely to have knowledge of Fast Response reports. The sampling frame for selecting districts will be the Common Core of Data (CCD), a file maintained by ED. We will exclude districts

• Located outside the 10 REL regions includes the 50 contiguous states, the District of Columbia, Puerto Rico, and several US territories and possessions

• That are federally or state-operated districts charged with providing services to special needs populations

• That have ceased to operate or have no schools or students

The number of potentially eligible school districts is approximately 15,650.

Attendees of REL events will include all attendants at REL-sponsored events during the period from April 2009 to October 2009. The frame for this group will consist of approximately 2,450 individuals. Lists of registrants will be obtained from the REL sponsoring each event; the lists will provide the sampling frame.

IES will provide lists of attendees of other IES events. Those who are employed by ED, any REL staff, and any contractor staff working on this project will be excluded from sampling. The number of survey-eligible attendees is estimated to be 825.

For the final group, the Chief State School Officers, Mathematica will attempt to conduct a census. The frame for the Chief State School Officers will be the list on the website of the Council of Chief State School Officers. There are 57 members, including representatives from the District of Columbia, Puerto Rico, and territories. The table below summarizes the population and sample sizes for the four groups.

The questionnaire is designed to answer two key questions: (1) Do the sample populations know about the Fast Response series of reports? and (2) For the reports they do know about, what is their assessment of the reports’ relevance and usefulness? We expect a 70 percent response rate for two reasons.

The first reason is that people who “screen out” because they are not aware of any Fast Response reports will be considered as completed cases. There is minimal burden for these sample members since they will be asked few questions. The second reason is we have been averaging about a 60 percent response rate on the Bridge Event Survey. The sample for that survey is attendees at REL events, which is one of the populations to be sampled for the Survey of the REL Research Reports. We have obtained this response rate with a data collection design in which the amount of time available for conducting follow-up calls with nonrespondents is limited, in order to minimize recall error. For the Survey of the REL Research Reports we will not have these time constraints and will therefore have more time to recontact nonrespondents and achieve a higher response rate.

The objective of the survey is to collect data from an adequate number of users of Fast Response reports to obtain a 95 percent confidence interval of no more than plus-or-minus 5 percentage points for each of the four groups that are being sampled. All Chief State School Officers will be asked to complete the survey or nominate knowledgeable members of their staffs to do so (that is, a census will be conducted), so no precision criteria were established.

POPULATION AND SAMPLE SIZES FOR SURVEY OF THE REL RESEARCH REPORTS

	Population Size	Initial Sample	Responding Sample	Use Ratea	Sample of Usersb
Local school district policy-makers	15,650	1,295	907	0.40	363
Attendees of REL events	2,450	690	483	0.65	314
Attendees of other IES events	825	545	382	0.55	314
Chief State School Officers	57	57	40	0.55	22
Total	18,982	2,587	1,812		918

^a Proportion expected to have used a Fast Response report.

^b In calculating the sample sizes we estimated the maximum 95 percent confidence interval applying the finite population correction factor.

6. Procedures for Sampling Methods and Analysis

The sample of the local school policymakers will be stratified by size and REL region. Because larger districts are more likely to employ such policymakers, districts will be sampled with probability proportional to total student enrollment.

The samples of event attendees will employ stratification but not oversampling; event attendees will be stratified by the event attended. The survey of Chief State School Officers is a census and does not require stratification.

Selected samples will be checked for duplicates across samples and if any are detected, one record will be randomly selected to be retained.

Analysis will be conducted with data that have been weighted to reflect differences in probabilities of selection and nonresponse (nonresponse adjustments will be made within sampling strata). For the Chief State School Officers, adjustments will be made by REL region.

To develop the sample frame we identified four groups as the target population. The information available to conduct the non-response bias analysis differs for these four groups and most, if not all, of that information will be used in stratification. Therefore, for the non-response bias analysis will examine response rates by sampling strata, and if REL is not used in sampling, by REL. For the Chief State School Officers, we will examine non-response by REL. For each of the four groups, we will make weighting adjustments to correct for differences. This information will be presented on a table with a memo describing the non-response characteristics.

AMOUNT OF ANY PROPOSED STIPEND OR INCENTIVE

No financial incentives or gifts will be offered to respondents.

BURDEN HOUR COMPUTATION (Number of respondents (X) estimated response or participation time in minutes (/60)=annual burden hours)

Category of Respondent	No. of Respondents	Participation Time	Burden
Web Survey Respondent	1,812 potential users	0.214 hour*	392.34 hours
Total	1,812 respondents		392.34 hours

*Average time estimated 918 x .33 hour (estimated number of users; full questionnaire) and 894 x .10 hour (estimated number of non-users; screening questions).

BURDEN COST COMPUTATION

There are no cost burdens to respondents.

REQUESTED APPROVAL DATE:

NAME OF CONTACT PERSON:

TELEPHONE NUMBER:

MAILING LOCATION:

ED DEPARTMENT, OFFICE:

PAPERWORK REDUCTION ACT

CHANGE WORKSHEET

Agency/Subagency U.S. Department of Education/ office name		OMB Control Number 1800-0011 v. #
	Enter only items that change Current Record	New Record
Agency form number(s)	NA	NA
Annual reporting and record keeping hour burden
Number of respondent	70,000	70,000
Total annual responses	70,000	70,000
Percent of these responses collected electronically	80%	80%
Total annual hours	25,000	25,000
Difference		0
Explanation of difference
Program change		0
Adjustment		0
Annual reporting and record keeping cost burden (in thousands of dollars)
Total annualized capital/startup costs	0	0
Total annual costs (O&M)	0	0
Total annualized cost requested	0	0
Difference		0
Explanation of difference
Program change adjustment		0
Other: ED is requesting approval of the “title” under the Customer Satisfaction Survey Master Plan, 1800-0011. The burden hours for this individual survey fall within the annual cap for 1800-0011.
Signature of Senior Official or designee:	Date:	For OIRA Use _________________________________ _________________________________

**This form cannot be used to extend an expiration date

OMB 83-C

8