Incentive Experiment NSFG Interim Results [Q3Q4] 2023

ATT-D4-NSFG-IncentiveExperimentResults-July2023.docx

[NCHS] National Survey of Family Growth

Incentive Experiment NSFG Interim Results [Q3Q4] 2023

OMB: 0920-0314

Document [docx]
Download: docx | pdf

NSFG OMB Attachment D4 OMB No. 0920-0314


Incentive Experiment for NSFG

Interim Results from Quarters 3 and 4 of 2022 and Quarters 1 and 2 of 2023

(updated for 30-Day OMB Revision Package Submitted July 2023)



As approved by OMB, a 3-condition incentive experiment was implemented in Quarters 3&4 of 2022 and is continued as a 2-condition experiment in 2023. The results presented here are from 2022 Quarters 3&4, 2023 Quarter 1, and Phases 1&2 of Quarter 21. Since both the 3-condition and 2-condition experiments have interventions focused on Phases 1&2, these interim results can be used to inform design changes for Year 3 (2024) and beyond. Counter to results from prior incentive experiments that have found increasing response rates with additional incentive amounts, albeit with diminishing benefit, the $80 incentive tested in 2022 Quarters 3&4 has led to similar increases in the response rates as the $60 incentive. Therefore, the experiment included only the $40 and $60 conditions in 2023, and the sample households assigned to the $80 condition in 2022 are grouped together with households assigned to the $60 condition in analyses.


1. Response rates. For these interim findings during implementation of the incentive experiment, results are restricted to response rates; as Quarters 1 and 2 of 2023 are completed, the data will allow for examination of the impact on demographic distributions.


The interim AAPOR RR42 unweighted response rates for the screener, conditional male and female, and overall male and female response rates for the three incentive conditions are shown in Table 1. The experimental Phase 1&2 incentive conditions ($60 and $80) lead to substantially higher response rates compared to the current Phase 1&2 incentive amount ($40). The $60 condition has led to a 11% increase in the male survey response rate and a 6% increase in the female survey response rate .


Table 1. Unweighted Response Rates by Incentive Condition, 2022 Quarter 3 and 4, 2023 Quarter 1, and Phases 1&2 of Quarter 2.


Phase 1&2 Incentive Condition

$40

n=17,882

$60 & $80

n=35,776

Difference

Screener

44.9%

44.7%

-0.2%

Male Survey

58.0%

64.3%

6.4%**

Female Survey

58.0%

61.7%

3.7%**

Overall Male Response Rate

26.0%

28.7%

2.7%*

Overall Female Response Rate

26.0%

27.6%

1.5%

* Statistically significant at =0.10

** Statistically significant at =0.05


2. Nonresponse bias. Respondent demographic characteristics and substantive estimates such as recent contraceptive use or ever had a child were used to evaluate sample balance and potential nonresponse bias reduction under the hypothesis that higher response rates through increased incentives should reduce rather than increase nonresponse bias (Groves, Presser and Dipko, 2004; Groves et al., 2006). Based on the similar response rate increases from the $60 and $80 conditions seen in Quarters 3 and 4 of 2022, the $60 and $80 conditions were combined in the nonresponse bias analyses, and all experimental condition cases were assigned to $60 in 2023.


The nonresponse bias analysis has three subcomponents: (a) sample balance for variables available from the screener, (b) differences in other socio-demographic variables available in the main survey, and (c) differences in selected key statistics from NSFG.


Completion rates by race/ethnicity, sex, and age group based on data from Quarters 1&2 of 2022 are shown in Table 2. Most noteworthy are the lower participation rates for individuals of Hispanic origin (42%) and for teenagers (43%).


Table 2. Main Interview Completion Rates by Demographic Subgroups Prior to the Incentive Experiment, Quarters 1&2 of 2022 (prior to the incentive experiment).


Number Eligible

Percent Complete

Race/Ethnicity



Hispanic

375

42.4%

Black

217

52.5%

White & Other

1,288

50.8%

Sex



Female

993

49.2%

Male

895

49.4%

Age



15-19

305

43.0%

20-29

515

48.2%

30-39

597

53.6%

40-49

471

49.3%


These survey completion rates are among screened households. Some demographic groups are underrepresented at the screener stage. The samples are designed to oversample non-Hispanic Blacks to yield 20% of the completed surveys, yet only 12.3% in Quarters 1 and 2 of 2022 were classified in this group. Similarly, those who are 15-19 years old are oversampled to yield 20% of the completed surveys but are only 16% of the respondents.


Sample imbalances are also observed for socio-demographic groups such as “ever married.” Among respondents in Quarters 1 and 2 of 2022, 44% were ever married, yet the proportion in the population 15 to 54 years of age who have ever married based on the 2022 Current Population Survey is 52%. In Quarters 1 and 2 approximately 45% reported 4-year college education or higher, while the 2022 CPS estimated the proportion with a bachelor’s degree or higher to be 36% for those 15 to 49 years of age.



a) Sample balance for variables from the screener.


The higher incentive amounts led to significantly different completion rates for all but one of the race/ethnicity, sex, and age groups, shown in Table 3. It was not significantly different for the 30-39 age group, which already had the highest response rate. Of particular importance is that the higher incentive amounts not only increased completion among some groups, but the increase was largest for groups that had the lowest completion rates, improving sample balance. The completion rate for the White & Other group increased by 5 percentage points, while for the Black and Hispanic groups it increased by 7 percentage points. The same was true for sex and age. Most noteworthy, the completion rate for those 15-19 years old was 17 percentage points lower than those 30-39 years old (45.8% vs. 62.8%). This difference in completion rates was decreased to 10 percentage points in the higher incentive conditions (54.2% vs. 63.9%). The higher incentive amount increased completion in the 15-19 group by 8 percentage points, compared to the not statistically significant increase of 1 percentage point in the 30-39 group.


Table 3. Main Interview Completion Rates by Demographic Subgroups and Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.

 

$40

$60 & $80

Rate Diff

p-value

 

Number Eligible

Percent Complete

Number Eligible

Percent Complete

$60&80-$40

Race/Ethnicity

Hispanic

620

50.3%

1,311

57.6%

7.3%

0.0028

Black

405

52.1%

872

58.8%

6.7%

0.0245

White & Other

1,934

58.3%

4,110

62.9%

4.6%

0.0006

Sex

Female

1,694

56.7%

3,481

60.9%

4.2%

0.0043

Male

1,265

54.5%

2,812

61.7%

7.2%

0.0000

Age

15-19

506

45.8%

1,058

54.2%

8.3%

0.0021

20-29

747

52.2%

1,580

59.7%

7.5%

0.0007

30-39

952

62.8%

2,050

63.9%

1.1%

0.5656

40-49

754

57.0%

1,605

64.0%

7.0%

0.0013



b) Differences in other socio-demographic variables.


Detecting differences by socio-demographic characteristics among respondents requires larger sample sizes than detecting differences in completion rates by demographic characteristics. Although none of the differences—for having ever been married, having a 4-year college degree, and having a household income over $100,000—is statistically significant, the overrepresentation of those with a college degree (relative to their percentage in the household population of 36%, discussed earlier) was reduced from 42.2% in the lower incentive condition to 40.8% in the higher incentive conditions, shown in Table 4.


Table 4. Socio-demographic Characteristics of Main Survey Respondents by Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.

 

$40

$60 & $80

Rate Diff


 

Count

Percent

Count

Percent

$60&80-$40

p-value

Ever married

725

43.8%

1,642

42.5%

-1.3%

0.3757

4-year college degree

698

42.2%

1,574

40.8%

-1.4%

0.3379

Income of $100,000 or more

568

38.8%

1,313

38.7%

-0.1%

0.9529



c) Differences in selected key NSFG statistics.


Higher response rates reduce the risk of nonresponse bias. Estimates do not have to be different for the risk to be reduced, as the variance in nonresponse bias is reduced. That is, higher response rates have been found to reduce the likelihood of nonresponse bias (Brick and Tourangeau, 2017). Nonetheless, differences in unweighted NSFG key survey estimates were compared across the incentive conditions, presented in Table 5, as they would demonstrate an instance of potential reduction in nonresponse bias. Of the 15 key survey estimates that were examined, 2 were significantly different in the higher incentive conditions. The percentage of respondents with age at first sex between 15 and 17 years increased from 35.1% to 39.0%. The percentage of respondents intending to have a birth or another birth increased from 48.3% to 51.6%.



Table 5. Select NSFG Key Statistics by Incentive Condition, 2022 Quarters 3 and 4, 2023 Quarter 1, and Phases 1&2 of 2023 Quarter 2.


$40

$60 & $80

Rate Diff

p-value

 

Count

Percent

Count

Percent

$60&80-$40

Age at first sex (<15)

134

12.6%

271

11.0%

1.6%

0.1730

Age at first sex (15-17)

373

35.1%

964

39.0%

-3.9%

0.0254

Age at first sex (18+)

557

52.4%

1,237

50.0%

2.3%

0.2076

Ever cohabited

691

41.8%

1,585

41.3%

0.6%

0.6893

No biological children

788

52.7%

1,870

54.3%

-1.5%

0.3224

One biological child

245

16.4%

543

15.8%

0.6%

0.5751

Two or more biological children

461

30.9%

1,029

29.9%

1.0%

0.4836

Intend a/another birth

774

48.3%

1,924

51.6%

-3.2%

0.0316

Used contraception at first sex

737

70.3%

1,730

71.0%

-0.7%

0.6689

Had sex in the last 12 months

1,052

81.2%

2,505

82.8%

-1.6%

0.2103

Ever smoked at least 100 cigarettes

385

23.5%

931

24.4%

-0.9%

0.4893

Ever had an HIV test outside of blood donation

687

42.2%

1,573

41.3%

0.9%

0.5517

Health care coverage in last 12 months

1,429

87.2%

3,365

88.8%

-1.6%

0.1059

Received public assistance in the last 12 months

82

5.3%

182

5.1%

0.2%

0.7449

Females only

Ever pregnant

529

55.0%

1,141

53.7%

1.3%

0.4953


In sum, three conclusions can be made from the data so far. First, response rates for the $60 and $80 incentive conditions are consistently higher than the $40 incentive condition (across incentive amounts and quarters). Second, the higher incentive amounts are showing improved participation rates, particularly among previously underrepresented groups. Relatedly, completion rates across the groups were more similar in the higher incentive conditions. Third, some key NSFG estimates showed sensitivity to the increased main survey participation under the higher incentive amount. There was a 3.9 percentage point increase in respondents with first sex between 15 and 17 years of age. There was also a 3.2 percentage point increase in respondents intending to have a birth or another birth.


In addition to the results on improved response rates, representation, and impact on key survey estimates, there is a pragmatic aspect to the incentive experiment. With post-COVID field interviewer recruitment and retention challenges, it is of critical importance in the new multimode NSFG data collection design to increase participation through increased web completion. The higher incentive amount achieves increased participation almost exclusively through web completion.



References


Brick, J. M. and R. Tourangeau (2017). "Responsive Survey Designs for Reducing Nonresponse Bias." 33(3): 735.

Groves, R. M., M. P. Couper, S. Presser, E. Singer, R. Tourangeau, G. P. Acosta and L. Nelson (2006). "Experiments in Producing Nonresponse Bias." Public Opinion Quarterly 70(5): 720-736.

Groves, R. M., S. Presser and S. Dipko (2004). "The role of topic interest in survey participation decisions." Public Opinion Quarterly 68(1): 2-31.



1 Interim results from Quarter 2 of 2023 may be used based on the current schedule for submission of the OMB 30-day package.

2 The American Association for Public Opinion Research. 2023. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 10th edition. AAPOR.

Page 8 of 8


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
Authorayc3
File Modified0000-00-00
File Created2024-07-23

© 2024 OMB.report | Privacy Policy