Request for Approval to Use Monetary Incentives in the 2010 Survey of Doctorate Recipients

Request for Approval to Use Monetary Incentives in the 2010 Survey of Doctorate Recipients

Overview

The Survey of Doctorate Recipients (SDR, OMB No. 3145-0020) is conducted by NSF biennially and is comprised of a national (NSDR) and international (ISDR) sample. Data collection for the 2010 SDR began on 11 October 2010 and is currently ongoing. For the 2008 SDR, NSF implemented a late-stage incentive plan to ensure an adequate response rate for each primary analytical domain within the NSDR. Based on the current state of 2010 SDR response rates, NSF requests approval to offer monetary incentives to NSDR cases according to a strategy similar to that used for the 2008 SDR.

Background

As of 15 March 2011 (5 months into data collection), the overall response rate for the 2010 SDR was 65.4%. NSF has considered three options for moving forward with this data collection:

1. Offer an incentive to attempt to achieve an 80% response rate and finish data collection on time,

2. Do not offer an incentive, finish data collection on time with a response rate less than 80%, and conduct a nonresponse bias study, and

3. Do not offer an incentive and continue data collection until an 80% response rate is reached, delaying delivery of the resulting data.

Because an 80% response rate is considered critical to this project (as fewer completed cases will result in more suppressed data), option #2 is not acceptable. Similarly, option #3 would require extending the data collection period for an estimated four months and attempting additional follow-up contacts with sample members. Not only would this delay the reporting of the data, but past experience has shown that excessive contacts have the potential to alienate sample members such that they request permanent removal from the project. Consequently, option #3 is not considered viable. Hence, implementing incentives (option #1) is considered the most effective approach.

However, NSF has been concerned about an additional issue: namely, the potential for incentives to negatively condition sample members to not respond in future survey rounds without an incentive. To respond to this concern, we analyzed data based on incentive experiments that were included in the 2003 and 2006 SDRs. In contrast to negative conditioning, the reverse effect seems to have resulted. Sample members who received an incentive in 2003 and/or 2006 responded to the 2008 SDR at a high rate (93%) without an incentive, suggesting the existence of residual goodwill among these sample members.

Given the potential for an increased response rate and a positive effect on SDR sample members, NSF wishes to implement incentives in the 2010 SDR.

Late-Stage Incentive Results from the 2008 SDR

The 2008 SDR offered a late-stage incentive ($30 prepaid) after 5.5 months of data collection. Nonrespondents in low responding sampling strata were offered the incentive with certainty, and 20% of the nonrespondents in better performing strata were offered the incentive.

Of the 11,163 sample members eligible for a late-stage incentive, 7,499 were selected for the incentive. However, only 4,717 of these cases were not in locating (i.e., had current contacting information).

In the end 4,215 sample members were offered the late-stage incentive and 2,108 (50%) cashed the check. The survey response and cashing outcomes for the 2008 SDR late-stage sample are displayed in Table 1.

Table 1: 2008 SDR Results from Offering Late-Stage Incentives

Completed Survey	Cashed Check	Cases	Percent
Yes	No	884	21.0%
	Yes	2,057	48.8%
No	No	1,223	29.0%
	Yes	51	1.2%
Overall		4,215	100.0%

Proposed Plan for the 2010 SDR

The overall strategy for the late stage incentive is to ensure that all sample members who have been subject to the standard survey data collection protocols and still remain as survey nonrespondents will have a probability of receiving a monetary incentive. In the plan used for the 2008 SDR and again proposed here, a greater probability of selection for the incentive will be given to cases in those sampling cells where there are relatively lower response rates, in order to improve the accuracy of survey estimates (given that the sampling cells are aligned with the domains of interest for analysis).

The sample parameters for the SDR are summarized below. There are 150 sampling cells, with three major stratification variables:

1) Demographic group (9 values) – this variable describes citizenship, disability status, and race/ethnicity,

2) Sex (2 values), and

3) Field of degree (7 values) – this variable describes the field of doctorate.

To determine which groups would have a greater or lesser probability of being offered an incentive, NSF prepared a table displaying for each 2010 sampling cell the 2008 final weighted response rate and the 2010 preliminary response rate as of 15 March 2011 (5 months into the data collection period). The sampling cells are used because it is at this level that NSF will make decisions about the incentive treatment. The 2008 final survey response rates are used as a reference because the target rate of 80% was achieved overall and because the response rates for the individual sampling cells indicate what might be expected at the end of the 2010 data collection. The 2010 preliminary response rates are shown so that we can make a determination of data collection progress to date.

To allocate its available limited resources for the monetary incentive to late-stage survey nonrespondents, NSF will divide the SDR’s sampling cells into two groups. One group will contain sampling cells from which all pending nonresponse cases will be offered an incentive, referred to as the “certainty” incentive treatment. The second group will be comprised of sampling cells from which a random sample of the pending nonresponse cases will be selected for the incentive offer, referred to as the “sample” incentive treatment. Whereas 100% of the pending nonresponse cases will be offered an incentive in the certainty incentive treatment, 20% of the cases in the sample incentive treatment will receive the same offer.

To define which sampling cells would fit into the certainty and sample incentive treatments , NSF reviewed the 2008 final response rates, the 2010 preliminary response rates, and the difference between the 2010 preliminary response rates and the target rate at the current stage in the 2010 data collection. Based on these data points, NSF developed the following broad parameters to use in making decisions for the incentive treatment.

1. Parameter 1: High vs. Low Response Rate in 2008
   A “low” response rate is any rate that is 105% or less of the overall 2008 final weighted response rate (i.e., 80.5%). A “high” response rate is any rate that is higher than 105% of the overall 2008 response rate.

2. Parameter 2: Fast vs. Slow Response Rate in 2010
   A “slow” response rate is any response rate that is 110% or less of the overall 2010 preliminary response rate. A “fast” response rate is any rate higher than 110% of the overall 2010 preliminary response rate.

3. Parameter 3: Big vs. Small Response Rate Difference between the Target Rate and the 2010 Preliminary Rates
   A “big” difference is any response rate difference that is 10% lower than the overall difference between the target rate and the 2010 preliminary response rate. A “small” difference is any response rate difference that is less than 10% lower than the overall difference between the target rate and 2010 preliminary response rate. For example, the target response rate for the 2010 SDR is 80.0%, while it was 65.4% as of 15 March 2011, for an overall difference of 14.6%. Therefore, any cells that have a target-2010 difference of 13.1% or more (10% lower than 14.6%) will be labeled as cells with a “big” difference in current versus target response rate, while those with a difference of less than 13.1% will be labeled as cells with a “small” difference in current versus target response rate.

To stay within the budgetary restraints of the 2010 SDR, given the response rates at the time when the incentives will be offered, it may be necessary to adjust the cut points for the three parameters. However, the three parameters will remain the same as will the 20% of the sample incentive treatment group receiving the incentive.

Taking these three parameters together, eight scenarios are possible. Table 2 provides the composition of possible incentive treatment groups and the rationale for the incentive treatment for each scenario. Table 3 provides SDR response rate information.

Each 2010 SDR sampling cell has been assigned a “certainty” or “sample” incentive treatment status based on this plan. Table 4 shows the distribution of cases in the sampling cells by each group.

Table 2: Response Rate Parameters and Assignment of Incentive Treatment

2008: High or Low Response Rate	2010: Fast or Slow Response Rate	Target-2010: Big or Small Difference	Group Acronym	Late Stage Incentive Treatment	Rationale for Incentive Treatment
Low	Slow	Small	LSS	Certainty	Cells that did not perform well in 2008, and are still not performing well relative to other cells in 2010.
Low	Slow	Big	LSB	Certainty	Cells that did not perform well in 2008, and are still not performing well relative to other cells in 2010.
Low	Fast	Small	LFS	Sample	Cells that did not perform well in 2008, but are performing well relative to other cells in 2010.
Low	Fast	Big	LFB	Certainty	Cells that did not perform well in 2008, and while they are performing better in 2010, still need to overcome a large difference to perform as well as the target rate.
High	Slow	Small	HSS	Sample	Cells that performed well in 2008, and although they are slow-performing in 2010, the difference to overcome is small in order to perform as well as the target rate.
High	Slow	Big	HSB	Certainty	Cells that performed well in 2008, but are not performing well in 2010, and the difference to overcome is large in order to perform as well as the target rate.
High	Fast	Small	HFS	Sample	Cells that performed well in 2008 and continue to do so in 2010.
High	Fast	Big	HFB	Sample	Cells that performed well in 2008 and 2010, even though the difference rate is large relative to the target.

Table 3: SDR Response Rates

2008 Final Response Rate	2010 Preliminary Response Rate (as of 03/15/11)	2010 SDR Target Response Rate	Difference between 2010 Target Rate and Preliminary Response Rate
80.5%	65.4%	80.0%	14.6%

Table 4: 2010 SDR Sampling Cells, Number of Cases, and
Response Rates by Incentive Treatment (as of 03/15/11)

Group Acronym	Late Stage Incentive Treatment	2010 SDR
		No. of sampling cells	No. of non-respondent cases	2010 response rate
HFS	Sample	19	1,138	75.9%
LFS	Sample	12	1651	73.4%
HFB	Sample	0	0	--
HSS	Sample	7	504	69.7%
LSS	Certainty	17	2117	70.7%
HSB	Certainty	7	269	62.3%
LSB	Certainty	88	7,565	57.2%
LFB	Certainty	0	0	--
Subtotal	Sample	38	3,293	73.9%
Subtotal	Certainty	112	9,951	61.2%
TOTAL		150	13,244	65.4%

Incentive Costs

According to this plan, NSF requests approval to offer a $30 prepaid incentive for the 2010 NSDR, as was done for the 2008 NSDR.

Based on the status of the surveys as of 15 March 2011, the following proportions of sampling cells would receive the certainty or sample late-stage incentive treatments:

• 112 of the sampling cells would receive the certainty incentive treatment (i.e., all nonresponse cases in the sampling cells would be selected for the incentive with certainty) – 100%*9,951 = 9,951 cases

• 38 of the sampling cells would receive the sample incentive treatment (i.e., 20% of the nonresponse cases in the sampling cells would be selected for an incentive) – 20%*3,293 = 659 cases

• Maximum possible incentive cost: $324,480

• Total estimated incentive cost based on 2008 cashing level: $93,018

• 94% of incentive cost to cases in certainty incentive treatment cells

• 6% of incentive cost to cases in sample incentive treatment cells

OMB submission 30 March 2011 Page 4 of 4