NPSAS24 Field Test Results

Appendix I Results of the NPSAS24 Field Test Experiment.docx

2023-24 National Postsecondary Student Aid Study (NPSAS:24) Full-Scale Study - Student Data Collection and Student Records

NPSAS24 Field Test Results

OMB: 1850-0666

Document [docx]
Download: docx | pdf

2023–24 NATIONAL POSTSECONDARY STUDENT AID STUDY (NPSAS:24)

Full-Scale Study

Student Data Collection and Student Records


Appendix I

Results of the NPSAS:24 Field Test Experiment

OMB # 1850-0666 v. 36


Submitted by

National Center for Education Statistics

U.S. Department of Education



September 2023







Results of the NPSAS:24 Field Test Experiment

The use of incentives to motivate completion of the student survey can provide significant advantages to the government in terms of increased response rates and higher quality data with minimized nonresponse bias (Groves, Singer, and Corning, 2000).1 In addition, the use of incentives may also result in decreased data collection costs due to improved efficiency. Providing higher incentives to respondents is a solution that in most cases would yield higher response rates; however, it is not always possible. Another solution is to find an optimal incentive distribution given fixed cost (for example, provide lower incentive at the beginning of the survey, but offer a boost as response rate begins to decline). While some respondents will receive an increased incentive amount relative to the others, on average, the total incentive cost per incentive group is very similar and the cost of the higher incentive later in data collection offsets the overall cost of data collection by reducing the need for intensive nonrespondent follow-up efforts.

During the NPSAS:24 field test, we conducted an experiment to determine the optimal amount and delivery timing of incentives, given fixed overall data collection costs (e.g., the costs of incentives, nonresponse follow-up). The field test sample size of 6,000 student sample members was randomly assigned to one of three groups. The sample design was sufficient to detect differences in the three groups no smaller than 4 percentage points at alpha=0.05 and 80 percent power. Table 2 describes the experimental plan across the three groups.

Group 1 received an initial incentive offer of $25, followed by an offer of an additional $10 to those who remained nonrespondents after four weeks of data collection (i.e., a boosted incentive). Group 2 received an initial offer of $25 as well, but with an additional $20 boost offered to nonrespondents at the eight-week point. At this eight-week follow-up point, boosts in incentive amounts might have offset other data collection costs by reducing the need for intensive nonrespondent follow-up efforts, for example. A third, control group, Group 3, received an incentive offer of $30 throughout the entire period of data collection. The base incentive amount of $30 in Group 3 has been used as the base incentive amount since NPSAS:04. All eligible cases in the NPSAS:24 field test study were offered one of the three monetary incentives for completing the student survey, paid by the sample member’s choice of check or PayPal.

Table 1. Experimental design by condition and phase of data collection


Group 1
n = 2,000

Group 2
n = 2,000

Group 3 (Control)
n = 2,000

Phase 1

$25 promised

$25 promised

$30 promised

Phase 2 (nonresponse follow-up)

+$10 at 4 weeks

+$20 at 8 weeks

$30 promised



Overall Response Rates. Comparison of the response rates across experimental groups revealed no statistically significant differences, suggesting no effect of the incentive boost or when it is introduced. Table 3 presents response rates by condition. The response rate difference between Group 1 and Group 3 failed to reach statistical significance (z=-0.96, p=0.34). Similarly, a comparison between Group 2 and Group 3 did not reach statistical significance (z=-1.23, p=0.22). Finally, a comparison between the two experimental conditions was also nonsignificant (z=0.28, p=0.78).



Table 2. Overall response rate by experimental condition (in percent)


Group 1
n = 2,000

Group 2
n = 2,000

Group 3 (Control)
n = 2,000

Response Rate

58.5

58.9

57.0



Representativeness. In addition to monitoring response rates, we conducted nonresponse bias analyses to assess the representativeness of the responding sample for each data collection group across key demographic characteristics, such as control and level of institution, age group, gender, race, ethnicity, and student type. Table 4 displays summary measures for the demographic distributions of the full sample and of survey respondents, by group and overall sample. Using chi-squared tests of the distributions of the characteristics, we failed to reject the null hypotheses of no difference across group - the three data collection protocols did not yield respondents with significantly different composition, suggesting no differential nonresponse bias.



Table 3. Distribution of the sample overall and of survey respondents, overall and by experimental group



Survey respondents

Demographic characteristics

Full sample

Overall

Group 1

Group 2

Group3

Control and level of institution






Public less-than-2-year

2.1

1.6

1.7

1.2

1.9

Public 2-year

7.5

6.9

7.0

6.1

7.7

Public 4-year, non-doctorate-granting, primarily sub-baccalaureate

11.5

10.3

10.7

10.6

9.7

Public 4-year, non-doctorate-granting, primarily baccalaureate

14.4

14.4

12.9

15.0

15.4

Public 4-year, doctorate-granting

0.0

0.0

0.0

0.0

0.0

Private nonprofit less-than-4-year

2.5

2.8

3.1

2.7

2.6

Private nonprofit 4-year, non-doctorate-granting

26.2

28.3

30.1

27.9

27.0

Private nonprofit 4-year, doctorate-granting

26.8

26.1

24.8

27.7

26.0

Private for-profit less-than-2-year

2.8

2.4

2.1

2.6

2.5

Private for-profit 2-year

1.0

1.0

1.2

0.8

1.2

Private for-profit 4-year

5.2

6.0

6.4

5.5

6.2

Age group






Younger than 24

61.7

60.8

60.3

61.8

60.1

24-29

16.2

16.7

17.8

15.3

16.9

30 or older

22.1

22.6

22.0

22.8

23.0

Gender






Unknown

12.1

12.2

12.8

11.4

12.4

Male

33.1

29.6

29.7

29.4

29.8

Female

54.7

58.1

57.5

59.2

57.7

Other

0.1

0.1

0.1

0.1

0.1

Race - White






Unknown

25.6

26.2

25.9

25.7

27.0

No

20.4

19.7

20.1

19.5

19.5

Yes

54.0

54.1

54.0

54.8

53.6

Race - Black






Unknown

27.1

27.8

27.6

27.2

28.7

No

62.8

62.1

61.9

63.2

61.1

Yes

10.1

10.1

10.5

9.5

10.1

Race - Asian






Unknown

27.5

28.1

28.3

27.2

29.0

No

69.0

68.5

68.3

69.2

67.9

Yes

3.5

3.4

3.4

3.7

3.1

Race – American Indian or Alaska Native






Unknown

27.7

28.4

28.3

27.7

29.1

No

70.8

70.0

70.4

70.7

68.9

Yes

1.5

1.6

1.3

1.5

2.1

Race – Native Hawaiian or Pacific Islander






Unknown

28.8

29.3

29.4

28.6

29.9

No

70.3

69.9

69.8

70.8

69.0

Yes

0.9

0.8

0.9

0.6

1.1

Ethnicity






Unknown

22.7

23.1

23.1

22.5

23.7

Not Hispanic or Latino

65.5

65.7

66.2

65.5

65.3

Hispanic or Latino

11.8

11.3

10.7

12.0

11.0

Student type






Graduate

5.2

5.7

5.9

5.7

5.5

Undergraduate

94.8

94.3

94.1

94.3

94.5

NOTE: Detail may not sum to totals due to rounding.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2023–24 National Postsecondary Student Aid Study (NPSAS:24) Field Test.



Cost. We also examined the cost per complete for each condition to determine whether one of the experimental groups provided significant savings. The cost per complete considered the total incentive cost for each completed interview, including the base incentive plus the applicable boost incentive received. We also considered the cost of data collection efforts associated with sending mailings, text messages, and making outbound phone call reminders. The total costs for each group are shown below.



Table 4. Estimated average cost per completed survey, by experimental group


Group 1
n = 2,000

Group 2
n = 2,000

Group 3 (Control)
n = 2,000

Cost per complete

$35.24

$37.35

$36.94



Conclusion

Overall, we failed to reject the null hypotheses of no difference across experimental groups in response rates and representativeness. We also failed to find evidence of significant cost savings that could be incurred with one of the experimental groups. Because of that, we recommend proceeding with the incentive design for the control group ($30 promised) and, if necessary, consider an incentive boost for groups that underperform during data collection.

1 Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: description and an illustration. The Public Opinion Quarterly64(3), 299-308.



I-19

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorTate, Nicole
File Modified0000-00-00
File Created2024-07-26

© 2024 OMB.report | Privacy Policy