Incentive Experiment NSFG Interim Results [Q3Q4] 2023

NSFG OMB Attachment D4 OMB No. 0920-0314

Incentive Experiment for NSFG

Interim Results from Quarters 3 and 4 of 2022 and Quarters 1 and 2 of 2023

(updated for 30-Day OMB Revision Package Submitted July 2023)

As approved by OMB, a 3-condition incentive experiment was implemented in Quarters 3&4 of 2022 and is continued as a 2-condition experiment in 2023. The results presented here are from 2022 Quarters 3&4, 2023 Quarter 1, and Phases 1&2 of Quarter 2¹. Since both the 3-condition and 2-condition experiments have interventions focused on Phases 1&2, these interim results can be used to inform design changes for Year 3 (2024) and beyond. Counter to results from prior incentive experiments that have found increasing response rates with additional incentive amounts, albeit with diminishing benefit, the $80 incentive tested in 2022 Quarters 3&4 has led to similar increases in the response rates as the $60 incentive. Therefore, the experiment included only the $40 and $60 conditions in 2023, and the sample households assigned to the $80 condition in 2022 are grouped together with households assigned to the $60 condition in analyses.

1. Response rates. For these interim findings during implementation of the incentive experiment, results are restricted to response rates; as Quarters 1 and 2 of 2023 are completed, the data will allow for examination of the impact on demographic distributions.

The interim AAPOR RR4² unweighted response rates for the screener, conditional male and female, and overall male and female response rates for the three incentive conditions are shown in Table 1. The experimental Phase 1&2 incentive conditions ($60 and $80) lead to substantially higher response rates compared to the current Phase 1&2 incentive amount ($40). The $60 condition has led to a 11% increase in the male survey response rate and a 6% increase in the female survey response rate .

Table 1. Unweighted Response Rates by Incentive Condition, 2022 Quarter 3 and 4, 2023 Quarter 1, and Phases 1&2 of Quarter 2.

	Phase 1&2 Incentive Condition
	$40 n=17,882	$60 & $80 n=35,776	Difference
Screener	44.9%	44.7%	-0.2%
Male Survey	58.0%	64.3%	6.4%**
Female Survey	58.0%	61.7%	3.7%**
Overall Male Response Rate	26.0%	28.7%	2.7%*
Overall Female Response Rate	26.0%	27.6%	1.5%

* Statistically significant at =0.10

** Statistically significant at =0.05

2. Nonresponse bias. Respondent demographic characteristics and substantive estimates such as recent contraceptive use or ever had a child were used to evaluate sample balance and potential nonresponse bias reduction under the hypothesis that higher response rates through increased incentives should reduce rather than increase nonresponse bias (Groves, Presser and Dipko, 2004; Groves et al., 2006). Based on the similar response rate increases from the $60 and $80 conditions seen in Quarters 3 and 4 of 2022, the $60 and $80 conditions were combined in the nonresponse bias analyses, and all experimental condition cases were assigned to $60 in 2023.

The nonresponse bias analysis has three subcomponents: (a) sample balance for variables available from the screener, (b) differences in other socio-demographic variables available in the main survey, and (c) differences in selected key statistics from NSFG.

Completion rates by race/ethnicity, sex, and age group based on data from Quarters 1&2 of 2022 are shown in Table 2. Most noteworthy are the lower participation rates for individuals of Hispanic origin (42%) and for teenagers (43%).

Table 2. Main Interview Completion Rates by Demographic Subgroups Prior to the Incentive Experiment, Quarters 1&2 of 2022 (prior to the incentive experiment).

	Number Eligible	Percent Complete
Race/Ethnicity
Hispanic	375	42.4%
Black	217	52.5%
White & Other	1,288	50.8%
Sex
Female	993	49.2%
Male	895	49.4%
Age
15-19	305	43.0%
20-29	515	48.2%
30-39	597	53.6%
40-49	471	49.3%

These survey completion rates are among screened households. Some demographic groups are underrepresented at the screener stage. The samples are designed to oversample non-Hispanic Blacks to yield 20% of the completed surveys, yet only 12.3% in Quarters 1 and 2 of 2022 were classified in this group. Similarly, those who are 15-19 years old are oversampled to yield 20% of the completed surveys but are only 16% of the respondents.

Sample imbalances are also observed for socio-demographic groups such as “ever married.” Among respondents in Quarters 1 and 2 of 2022, 44% were ever married, yet the proportion in the population 15 to 54 years of age who have ever married based on the 2022 Current Population Survey is 52%. In Quarters 1 and 2 approximately 45% reported 4-year college education or higher, while the 2022 CPS estimated the proportion with a bachelor’s degree or higher to be 36% for those 15 to 49 years of age.

a) Sample balance for variables from the screener.

The higher incentive amounts led to significantly different completion rates for all but one of the race/ethnicity, sex, and age groups, shown in Table 3. It was not significantly different for the 30-39 age group, which already had the highest response rate. Of particular importance is that the higher incentive amounts not only increased completion among some groups, but the increase was largest for groups that had the lowest completion rates, improving sample balance. The completion rate for the White & Other group increased by 5 percentage points, while for the Black and Hispanic groups it increased by 7 percentage points. The same was true for sex and age. Most noteworthy, the completion rate for those 15-19 years old was 17 percentage points lower than those 30-39 years old (45.8% vs. 62.8%). This difference in completion rates was decreased to 10 percentage points in the higher incentive conditions (54.2% vs. 63.9%). The higher incentive amount increased completion in the 15-19 group by 8 percentage points, compared to the not statistically significant increase of 1 percentage point in the 30-39 group.

Table 3. Main Interview Completion Rates by Demographic Subgroups and Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.

	$40		$60 & $80		Rate Diff	p-value
	Number Eligible	Percent Complete	Number Eligible	Percent Complete	$60&80-$40
Race/Ethnicity
Hispanic	620	50.3%	1,311	57.6%	7.3%	0.0028
Black	405	52.1%	872	58.8%	6.7%	0.0245
White & Other	1,934	58.3%	4,110	62.9%	4.6%	0.0006
Sex
Female	1,694	56.7%	3,481	60.9%	4.2%	0.0043
Male	1,265	54.5%	2,812	61.7%	7.2%	0.0000
Age
15-19	506	45.8%	1,058	54.2%	8.3%	0.0021
20-29	747	52.2%	1,580	59.7%	7.5%	0.0007
30-39	952	62.8%	2,050	63.9%	1.1%	0.5656
40-49	754	57.0%	1,605	64.0%	7.0%	0.0013

b) Differences in other socio-demographic variables.

Detecting differences by socio-demographic characteristics among respondents requires larger sample sizes than detecting differences in completion rates by demographic characteristics. Although none of the differences—for having ever been married, having a 4-year college degree, and having a household income over $100,000—is statistically significant, the overrepresentation of those with a college degree (relative to their percentage in the household population of 36%, discussed earlier) was reduced from 42.2% in the lower incentive condition to 40.8% in the higher incentive conditions, shown in Table 4.

Table 4. Socio-demographic Characteristics of Main Survey Respondents by Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.

	$40		$60 & $80		Rate Diff
	Count	Percent	Count	Percent	$60&80-$40	p-value
Ever married	725	43.8%	1,642	42.5%	-1.3%	0.3757
4-year college degree	698	42.2%	1,574	40.8%	-1.4%	0.3379
Income of $100,000 or more	568	38.8%	1,313	38.7%	-0.1%	0.9529

c) Differences in selected key NSFG statistics.

Higher response rates reduce the risk of nonresponse bias. Estimates do not have to be different for the risk to be reduced, as the variance in nonresponse bias is reduced. That is, higher response rates have been found to reduce the likelihood of nonresponse bias (Brick and Tourangeau, 2017). Nonetheless, differences in unweighted NSFG key survey estimates were compared across the incentive conditions, presented in Table 5, as they would demonstrate an instance of potential reduction in nonresponse bias. Of the 15 key survey estimates that were examined, 2 were significantly different in the higher incentive conditions. The percentage of respondents with age at first sex between 15 and 17 years increased from 35.1% to 39.0%. The percentage of respondents intending to have a birth or another birth increased from 48.3% to 51.6%.

Table 5. Select NSFG Key Statistics by Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.

	$40		$60 & $80		Rate Diff	p-value
	Count	Percent	Count	Percent	$60&80-$40
Age at first sex (<15)	134	12.6%	271	11.0%	1.6%	0.1730
Age at first sex (15-17)	373	35.1%	964	39.0%	-3.9%	0.0254
Age at first sex (18+)	557	52.4%	1,237	50.0%	2.3%	0.2076
Ever cohabited	691	41.8%	1,585	41.3%	0.6%	0.6893
No biological children	788	52.7%	1,870	54.3%	-1.5%	0.3224
One biological child	245	16.4%	543	15.8%	0.6%	0.5751
Two or more biological children	461	30.9%	1,029	29.9%	1.0%	0.4836
Intend a/another birth	774	48.3%	1,924	51.6%	-3.2%	0.0316
Used contraception at first sex	737	70.3%	1,730	71.0%	-0.7%	0.6689
Had sex in the last 12 months	1,052	81.2%	2,505	82.8%	-1.6%	0.2103
Ever smoked at least 100 cigarettes	385	23.5%	931	24.4%	-0.9%	0.4893
Ever had an HIV test outside of blood donation	687	42.2%	1,573	41.3%	0.9%	0.5517
Health care coverage in last 12 months	1,429	87.2%	3,365	88.8%	-1.6%	0.1059
Received public assistance in the last 12 months	82	5.3%	182	5.1%	0.2%	0.7449
Females only
Ever pregnant	529	55.0%	1,141	53.7%	1.3%	0.4953

In sum, three conclusions can be made from the data so far. First, response rates for the $60 and $80 incentive conditions are consistently higher than the $40 incentive condition (across incentive amounts and quarters). Second, the higher incentive amounts are showing improved participation rates, particularly among previously underrepresented groups. Relatedly, completion rates across the groups were more similar in the higher incentive conditions. Third, some key NSFG estimates showed sensitivity to the increased main survey participation under the higher incentive amount. There was a 3.9 percentage point increase in respondents with first sex between 15 and 17 years of age. There was also a 3.2 percentage point increase in respondents intending to have a birth or another birth.

In addition to the results on improved response rates, representation, and impact on key survey estimates, there is a pragmatic aspect to the incentive experiment. With post-COVID field interviewer recruitment and retention challenges, it is of critical importance in the new multimode NSFG data collection design to increase participation through increased web completion. The higher incentive amount achieves increased participation almost exclusively through web completion.

References

Brick, J. M. and R. Tourangeau (2017). "Responsive Survey Designs for Reducing Nonresponse Bias." 33(3): 735.

Groves, R. M., M. P. Couper, S. Presser, E. Singer, R. Tourangeau, G. P. Acosta and L. Nelson (2006). "Experiments in Producing Nonresponse Bias." Public Opinion Quarterly 70(5): 720-736.

Groves, R. M., S. Presser and S. Dipko (2004). "The role of topic interest in survey participation decisions." Public Opinion Quarterly 68(1): 2-31.



1 Interim results from Quarter 2 of 2023 may be used based on the current schedule for submission of the OMB 30-day package.

2 The American Association for Public Opinion Research. 2023. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 10th edition. AAPOR.

Page 8 of 8