Response Memo 3

M EMORANDUM

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone (609) 799-3535

Fax (609) 799-0005

www.mathematica-mpr.com

TO: Michel Smyth

FROM: Pat Nemeth and Karen Needels DATE: 9/14/2012

UCP-092

SUBJECT: Response to the OMB Question about Prior Research Regarding Differential Incentives for Surveys, for the OMB Supporting Statement for the Evaluation of the Unemployment Compensation Provisions of the American Recovery and Reinvestment Act of 2009

During a September 5, 2012, discussion, OMB staff requested additional information about four topics related to the OMB Supporting Statement for the Evaluation of the Unemployment Compensation Provisions of the American Recovery and Reinvestment Act of 2009 (“the UCP study”). The four topics are as follows:

1. Additional information about the policies for which estimates of quasi-experimental impacts will be generated.

2. Comparison between the UCP study and the Rothstein (2011) study.

3. Examples of reports including the use of legislators and lobbyists as sources of data.

4. Information about the use in prior studies of a differential in the incentives provided to respondents based on the mode of survey completion.

As was agreed during the discussion, the information that was requested has been provided as it has become available. This memo includes information pertaining to the fourth topic—about prior research on the use of differential incentives.¹

The data collection plan for the UI recipient survey seeks to maximize both the response rate and the number of questionnaires completed online. The first goal will help provide high quality data for the analyses that follow, and the second goal will help control data collection costs. To accomplish both of these goals, the project is requesting clearance to offer a two-tiered incentive that will motivate sample members to complete the survey and to favor web completions (questionnaires completed online) over questionnaires via telephone. The proposed plan is to offer a $50 incentive for a web complete, and $40 for a phone complete. These incentive amounts would be offered initially, and throughout the letter, postcard and email (if available) reminders.

To provide the documentation from other studies where a differential or two-tiered incentive has been used, this memo presents the findings from the National Survey of Recent College Graduates (NSRCG), which had surveys for the 2008 and 2010 cohorts of graduates. Slides for a recent presentation about the study at an American Association for Public Opinion Research (AAPOR) conference also accompany this memo. While the UCP project will not use the exact same design as NSRCG, many aspects will be similar.

A summary of the 2010 NSRCG survey and the proposed UCP survey data collection plan is presented below.

Survey Characteristic	2010 NSRCG Survey	UCP Survey
Mode	Web and mail	Web and phone
Incentive	$30 for web, $20 for mail	$50 for web, $40 for phone
Reminders	Email, letters, postcards, and phone calls	Letters, postcards, phone calls and emails (if available)
Questionnaire length	25 minutes	30 minutes
Study population	Recent college graduates	Unemployment insurance (UI) benefit recipients

Mathematica has successfully completed two surveys for the National Science Foundation that used this two-tiered incentive approach, and others are currently under way. The 2008 NSRCG study embedded the two-tiered incentive in an experiment defined by three factors; (1) the impact of the incentive amount, (2) the timing of the incentive, and (3) the initial response mode. The entire 2008 NSRCG sample of 18,000 recent graduates was randomized into one of eight treatment groups defined by these three factors. Results looked at (1) response rates and (2) the proportion of survey completes conducted via the web. In the 2008 experiment, the two-tiered incentive groups achieved the highest proportion of web completes and were two of the three groups with the highest response rates. The groups that were offered no incentives had response rates 10 to nearly 15 percentage points lower than groups that were offered incentives and groups that were offered the differential incentive had the highest response rates.

The table below illustrates the eight treatment groups and the outcomes of web completion proportions and response rates:

2008 NSRCG Treatment Groups

Group	Initial Response Mode	First Mailing Incentive	Second Mailing Incentive	Percent Web Completes	Response Rate*
1	Web only	0	0	71.6	40.0
2	Web only	$20	$20	78.7	48.2
3	Web only	0	$20 paper $30 web	79.9	49.5
4	Web/Mail	0	0	53.6	39.8
5	Web/Mail	$20	$20	59.8	49.9
6	Web/Mail	0	$20 paper $30 web	64.7	47.7
7	Web/Mail	$20 paper $30 web	$20 paper $30 web	81.7	52.8
8	Web/Mail	0	$20	56.3	46.4

*At the time the experiment ended (about half way through data collection).

The information above can be summarized as follows:

Highest percentage of web completes:
Group 7	two-tier incentive	81.7
Group 3	two-tier incentive	79.9
Group 2	single level incentive	78.7

Highest response rates:
Group 7	two-tier incentive	52.8
Group 5	single level incentive	49.9
Group 3	two-tier incentive	49.5

The 2010 NSRCG data collection plan received OMB clearance (OMB control number 3145-0077) to incorporate the best practices identified from the 2008 experiment, with an emphasis on cost savings. Group 7 had both the highest response rate and the highest percentage of web completes in 2008 (see above). However, in 2010, the project chose to go with a Group 3 model, which also included a two-tiered incentive. This is because Group 3, while promising to produce results very close to a Group 7 design, could do so at a substantially lower cost. The 2010 NSRCG surpassed the 2008 response rate and proportion of web completions.

The UCP study is proposing to use a modified version of the 2010 NSRCG incentive structure, adapted to meet the study’s needs given the UI recipient survey length and target survey respondents. Although the sample composition and completion modes for the NSRCG and UCP projects are not exactly the same, it is anticipated that the use of the proposed incentive structure for UCP will lead to a high response rate achieved in a very cost-efficient manner, maximizing the proportion of web completes. As shown in the 2008 NSRCG experiment results, offering a larger incentive to encourage web completes, combined with frequent contact with the sample member’s household, led to higher completion rates.

The abstract for the AAPOR presentation, which contains more information on the 2008 NSRCG experiment, can be found at

http://www.websm.org/db/12/15355/Web%20Survey%20Bibliography/Influencing_Mode_Choice_in_a_Mixed_Mode_Survey/?&p1=1123&p2=82&p3=85&page=1

The full AAPOR presentation is attached to the email that accompanies this memo. For questions or further clarifications, the main author of the presentation, Geraldine Mooney Ph.D., Mathematica Vice President and Managing Director, Methodology and Development, can be contacted at [email protected] or at (609) 275-2359.

cc: Stefanie Schmidt, Jonathan Simonetta

1 Two previous memos, dated September 10th and September 11th, provided information on the first three topics.

An Affirmative Action/Equal Opportunity Employer