OMB C Memo

u. s. dEPARTMENT OF eDUCATION

institute of education sciences

National Center for Education Evaluation and regional assistance

to: Sharon Mar, OMB / OIRA

from: Jonathan Jacobson, IES

THROUGH: Kathy Axt, OM / Information Collections Clearance Division

subject: Request for Modification to OMB-Approved Plan for Survey of Customers, Evaluation of the Regional Educational Laboratories

(OMB# 1850-0885)

date: 1/19/2012

In this memo, we update OMB on progress in data collection for the customer survey authorized by OMB (#1850-0885) and being conducted for the congressionally-mandated evaluation of the Regional Educational Laboratories (OMB# 1850-0885). This web-based survey is being administered by Westat under contract to IES.

The original plan for this data collection was to sample 300 SEA staff and 14,400 LEA staff, obtaining 11,760 completions, for a total burden of 1,960 hours (see Exhibit A-1, p. 6 of OMB Justification Part A). To gather information on response rates by region in order to achieve the overall target of 80 percent (cf. p. 5 of OMB Justification Part B), Westat began data collection in summer 2011 from the universe of eligible SEA officials in the ten REL regions and a sample of 1,027 LEA officials drawn randomly from a nationally representative sample of 1,271 LEAs, stratified by REL region.

As of January 17, 2012, Westat has obtained, using the strategies described in Exhibit B-3 of OMB Justification Part B (p. 5), response rates of 75 percent for both eligible SEA officials and eligible LEA officials sampled for the web survey. Response rates have varied as follows since November 11, 2011:

Date Sample of SEA Officials Sample of LEA Officials

11/11/2011* 52 percent 48 percent

12/2/2011* 71 percent 69 percent

12/9/2011* 74 percent 72 percent

1/17/2012 75 percent 75 percent

* reminder calls/emails/letters to non-respondents were employed repeatedly through 12/9/2011

Response rates have varied considerably by REL region, from a low of 61 percent and a high of 81 percent for SEA officials in the ten REL regions, and from a low of 66 percent to a high of 85 percent for LEA officials in the nine REL regions including LEAs. Low response rates suggest that a wider set of tools is needed if the evaluation is to be based on valid and statistically representative data. Increasing region-specific response rates is of particular importance for this evaluation, since the Education Sciences Reform Act of 2002 (P.L. 107-279, Section 174) requires that REL evaluation findings be reported separately for each REL and its region.

Because the target response rate for the survey is 80 percent, and because of the ESRA requirement to report evaluation findings separately for each REL region, we are requesting approval to use a $10/respondent financial incentive with the goal of raising the response rate to at least 80 percent nationally for each type of respondent (LEA official or SEA official), and to at least 70 percent for each REL region.

Several studies have found various levels of increased response rates for web surveys for when incentives are used compared to no incentives.   These have included a web survey with a response rate of 23.9% for no incentive compared to 31.4% with an incentive (Cobanoglu and Cobanoglu 2003), and a difference of a 38% response rate for a control group on a web survey compared to 53% for an incentive of prepaid and postpaid bonus points and 54% for an immediate notification of prize draw results (Bosnjak & Wenzel 2005).  Yet another web survey had a response rate of 42% with no incentive and 54% with a pre-incentive (Sánchez-Fernández, Muñoz-Leiva, Montoro-Ríos and Ibáñez-Zapata 2008).

Prepaid monetary incentives have been shown to be more effective than promised incentives in raising response rates (Couper 2008), so we propose including $10 in a mailing to sampled SEA and LEA officials in agencies that have not yet provided responses to the survey, with a letter from Westat describing the study and providing the survey website and login information. The mailing will also include a letter from IES encouraging participation. Westat will use phone calls to districts to verify the eligibility of non-respondents, e.g., to ensure that they are still employed in the same capacity in the same district. For any regions that do not meet a 70% response rate, Westat will engage in more rigorous non-response follow-up, including additional phone calls, emails, and letters.

In contrast with their status for many other IES evaluations, SEA and LEA officials are not grantees under the REL Program. For this reason, we think a modest incentive, consistent with the estimated burden, is justified as a means of increasing survey response to acceptable levels.

If OMB approves this request, our estimates of respondent burden would be revised to read:

Revised Exhibit A-1. Estimates of Respondent Burden

Respondent	Anticipated number completed	Minutes per completion	Burden in minutes	Burden in hours	Burden in Dollars
	(a)	(b)	(c) a x b	c/60
SEA official	307	12	3,684	61	$3,172
LEA official	3,017	12	36,204	603	$31,356
Total burden	3,324	12	39,888	664	$ 34,528

NOTE: Assumes an hourly rate of $52 per hour. This hourly rate is based on average daily rates for administrators obtained from the survey “Salaries and Wages Paid Professional and Support Personnel in Public Schools” (http://www.edweek.org/media/43ers-data.pdf).

The revised estimate of 12 minutes per completion (versus 10 minutes previously) accounts for experience to date and in particular the possibility that some officials will complete the survey by phone rather than through the Web. The costs of obtaining sufficiently high response rates in a budget-neutral manner are reflected in a decrease in the anticipated total number of respondents from 11,760 to 3,324, of whom 307 would be SEA officials, and 3,017 would be LEA officials. The corresponding respondent burden would decrease from 1,960 to 664 hours. Precision levels for the LEA survey would be 3.1 percentage points for potential REL users in the average region, versus 1.9 percentage points in the original justification (Part B, p. 5).

3