NPSAS24 Field Test Results

2023–24 NATIONAL POSTSECONDARY STUDENT AID STUDY (NPSAS:24)

Full-Scale Study

Student Data Collection and Student Records

Appendix I

Results of the NPSAS:24 Field Test Experiment

OMB # 1850-0666 v. 36

Submitted by

National Center for Education Statistics

U.S. Department of Education

September 2023

Results of the NPSAS:24 Field Test Experiment

The use of incentives to motivate completion of the student survey can provide significant advantages to the government in terms of increased response rates and higher quality data with minimized nonresponse bias (Groves, Singer, and Corning, 2000).¹ In addition, the use of incentives may also result in decreased data collection costs due to improved efficiency. Providing higher incentives to respondents is a solution that in most cases would yield higher response rates; however, it is not always possible. Another solution is to find an optimal incentive distribution given fixed cost (for example, provide lower incentive at the beginning of the survey, but offer a boost as response rate begins to decline). While some respondents will receive an increased incentive amount relative to the others, on average, the total incentive cost per incentive group is very similar and the cost of the higher incentive later in data collection offsets the overall cost of data collection by reducing the need for intensive nonrespondent follow-up efforts.

During the NPSAS:24 field test, we conducted an experiment to determine the optimal amount and delivery timing of incentives, given fixed overall data collection costs (e.g., the costs of incentives, nonresponse follow-up). The field test sample size of 6,000 student sample members was randomly assigned to one of three groups. The sample design was sufficient to detect differences in the three groups no smaller than 4 percentage points at alpha=0.05 and 80 percent power. Table 2 describes the experimental plan across the three groups.

Group 1 received an initial incentive offer of $25, followed by an offer of an additional $10 to those who remained nonrespondents after four weeks of data collection (i.e., a boosted incentive). Group 2 received an initial offer of $25 as well, but with an additional $20 boost offered to nonrespondents at the eight-week point. At this eight-week follow-up point, boosts in incentive amounts might have offset other data collection costs by reducing the need for intensive nonrespondent follow-up efforts, for example. A third, control group, Group 3, received an incentive offer of $30 throughout the entire period of data collection. The base incentive amount of $30 in Group 3 has been used as the base incentive amount since NPSAS:04. All eligible cases in the NPSAS:24 field test study were offered one of the three monetary incentives for completing the student survey, paid by the sample member’s choice of check or PayPal.

Table 1. Experimental design by condition and phase of data collection

	Group 1 n = 2,000	Group 2 n = 2,000	Group 3 (Control) n = 2,000
Phase 1	$25 promised	$25 promised	$30 promised
Phase 2 (nonresponse follow-up)	+$10 at 4 weeks	+$20 at 8 weeks	$30 promised

Overall Response Rates. Comparison of the response rates across experimental groups revealed no statistically significant differences, suggesting no effect of the incentive boost or when it is introduced. Table 3 presents response rates by condition. The response rate difference between Group 1 and Group 3 failed to reach statistical significance (z=-0.96, p=0.34). Similarly, a comparison between Group 2 and Group 3 did not reach statistical significance (z=-1.23, p=0.22). Finally, a comparison between the two experimental conditions was also nonsignificant (z=0.28, p=0.78).

Table 2. Overall response rate by experimental condition (in percent)

	Group 1 n = 2,000	Group 2 n = 2,000	Group 3 (Control) n = 2,000
Response Rate	58.5	58.9	57.0

Representativeness. In addition to monitoring response rates, we conducted nonresponse bias analyses to assess the representativeness of the responding sample for each data collection group across key demographic characteristics, such as control and level of institution, age group, gender, race, ethnicity, and student type. Table 4 displays summary measures for the demographic distributions of the full sample and of survey respondents, by group and overall sample. Using chi-squared tests of the distributions of the characteristics, we failed to reject the null hypotheses of no difference across group - the three data collection protocols did not yield respondents with significantly different composition, suggesting no differential nonresponse bias.

Table 3. Distribution of the sample overall and of survey respondents, overall and by experimental group

		Survey respondents
Demographic characteristics	Full sample	Overall	Group 1	Group 2	Group3
Control and level of institution
Public less-than-2-year	2.1	1.6	1.7	1.2	1.9
Public 2-year	7.5	6.9	7.0	6.1	7.7
Public 4-year, non-doctorate-granting, primarily sub-baccalaureate	11.5	10.3	10.7	10.6	9.7
Public 4-year, non-doctorate-granting, primarily baccalaureate	14.4	14.4	12.9	15.0	15.4
Public 4-year, doctorate-granting	0.0	0.0	0.0	0.0	0.0
Private nonprofit less-than-4-year	2.5	2.8	3.1	2.7	2.6
Private nonprofit 4-year, non-doctorate-granting	26.2	28.3	30.1	27.9	27.0
Private nonprofit 4-year, doctorate-granting	26.8	26.1	24.8	27.7	26.0
Private for-profit less-than-2-year	2.8	2.4	2.1	2.6	2.5
Private for-profit 2-year	1.0	1.0	1.2	0.8	1.2
Private for-profit 4-year	5.2	6.0	6.4	5.5	6.2
Age group
Younger than 24	61.7	60.8	60.3	61.8	60.1
24-29	16.2	16.7	17.8	15.3	16.9
30 or older	22.1	22.6	22.0	22.8	23.0
Gender
Unknown	12.1	12.2	12.8	11.4	12.4
Male	33.1	29.6	29.7	29.4	29.8
Female	54.7	58.1	57.5	59.2	57.7
Other	0.1	0.1	0.1	0.1	0.1
Race - White
Unknown	25.6	26.2	25.9	25.7	27.0
No	20.4	19.7	20.1	19.5	19.5
Yes	54.0	54.1	54.0	54.8	53.6
Race - Black
Unknown	27.1	27.8	27.6	27.2	28.7
No	62.8	62.1	61.9	63.2	61.1
Yes	10.1	10.1	10.5	9.5	10.1
Race - Asian
Unknown	27.5	28.1	28.3	27.2	29.0
No	69.0	68.5	68.3	69.2	67.9
Yes	3.5	3.4	3.4	3.7	3.1
Race – American Indian or Alaska Native
Unknown	27.7	28.4	28.3	27.7	29.1
No	70.8	70.0	70.4	70.7	68.9
Yes	1.5	1.6	1.3	1.5	2.1
Race – Native Hawaiian or Pacific Islander
Unknown	28.8	29.3	29.4	28.6	29.9
No	70.3	69.9	69.8	70.8	69.0
Yes	0.9	0.8	0.9	0.6	1.1
Ethnicity
Unknown	22.7	23.1	23.1	22.5	23.7
Not Hispanic or Latino	65.5	65.7	66.2	65.5	65.3
Hispanic or Latino	11.8	11.3	10.7	12.0	11.0
Student type
Graduate	5.2	5.7	5.9	5.7	5.5
Undergraduate	94.8	94.3	94.1	94.3	94.5

NOTE: Detail may not sum to totals due to rounding.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2023–24 National Postsecondary Student Aid Study (NPSAS:24) Field Test.

Cost. We also examined the cost per complete for each condition to determine whether one of the experimental groups provided significant savings. The cost per complete considered the total incentive cost for each completed interview, including the base incentive plus the applicable boost incentive received. We also considered the cost of data collection efforts associated with sending mailings, text messages, and making outbound phone call reminders. The total costs for each group are shown below.

Table 4. Estimated average cost per completed survey, by experimental group

	Group 1 n = 2,000	Group 2 n = 2,000	Group 3 (Control) n = 2,000
Cost per complete	$35.24	$37.35	$36.94

Conclusion

Overall, we failed to reject the null hypotheses of no difference across experimental groups in response rates and representativeness. We also failed to find evidence of significant cost savings that could be incurred with one of the experimental groups. Because of that, we recommend proceeding with the incentive design for the control group ($30 promised) and, if necessary, consider an incentive boost for groups that underperform during data collection.

1 Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: description and an illustration. The Public Opinion Quarterly, 64(3), 299-308.

I-19