Justification

National Center for Education Statistics (NCES)

Volume I

Supporting Statement

2019-20 National Postsecondary Student Aid Study (NPSAS:20) Online Pretesting Round 2

Opinion Survey Using Online Crowdsourcing Platform

OMB# 1850-0803 v. 247

February 2019

Attachments

Attachment I – Student Opinion Survey Using Crowdsourcing Platform

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews to improve methodologies, survey questions, and/or delivery methods.

This request is to test components of the 2019-20 National Postsecondary Student Aid Study (NPSAS:20) full-scale survey instrument pertaining to emergency financial aid and food and housing security with participants currently or recently enrolled in postsecondary education using a crowdsourcing platform online opinion survey. Testing will begin in March 2019, in preparation for the NPSAS:20 full-scale data collection (OMB# 1850-0666), which will begin in October 2019. RTI International (RTI) will collect NPSAS:20 data from institutions and students on behalf of NCES under contract to the U.S. Department of Education.

The overarching purpose of NPSAS is to collect data on how students pay for their postsecondary education in a specific academic year. The majority of the content of the student survey to be used for the upcoming NPSAS:20 full-scale data collection has been previously tested or included in prior NPSAS surveys, other NCES studies, or other surveys of postsecondary students. The testing described in this submission allows NCES to evaluate selected components that are either new to NPSAS or revised for its use, before their inclusion in the NPSAS:20 full-scale data collection.

This submission describes all aspects of the MTurk testing, including recruitment and screening of participants, participant consent, and survey items. Amazon’s Mechanical Turk (MTurk) will provide a convenience sample for this online opinion survey about emergency financial aid and food and housing security. RTI will coordinate the MTurk work. The results, which will inform potential survey and design modifications intended to refine the full-scale student survey content, will be submitted for clearance in July 2019 as part of the NPSAS:20 full-scale student data collection request (OMB# 1850-0666).

Background

NPSAS:20, conducted by NCES, is a nationally representative study of how students and their families finance education beyond high school. NPSAS:20, the tenth cycle in the series, will be conducted from October 2019 to December 2020 to capture information pertaining to the 2019-20 academic year. NPSAS:20 will also serve as the base year data collection for the Beginning Postsecondary Students Longitudinal Study (BPS) – a study of first time, beginning college students at two years (BPS:20/22) and five years (BPS:20/25) after their first year of college enrollment.

The proposed here opinion survey using Amazon’s MTurk will allow NCES to quickly recruit participants and test survey items intended to explore emergency financial aid and food and housing security of students enrolled in the 2018-19 academic year. This testing will allow the collection of preliminary emergency financial aid and food and housing security data which are constructs new to the NPAS:20 survey. The results will be used to design the NPSAS:20 student survey questions on these topics.

NPSAS:20 first-time online pretesting with MTurk was approved recently to evaluate the appeal of data collection materials and various incentive options (OMB# 1850-0803 v. 243). In addition, the National Center for Science and Engineering Statistics (NCSES) at the National Science Foundation (NSF) selected MTurk as one of the most promising because it has a larger available sample than other crowdsourcing platforms and has the most extensive features for managing a convenience sample. Besides NCSES, the Bureau of Labor Statistics (BLS) and the National Cancer Institute (NCI) at the National Institutes of Health (NIH) also use MTurk successfully to recruit participants to complete online surveys.

Design and Context

The described here opinion survey will be conducted using a web survey administered to participants recruited through Amazon’s MTurk online crowdsourcing platform, who are referred to as “workers.” The MTurk recruitment post to be used is provided in Attachment I. Qualified workers based on MTurk’s qualifications will be able to view the request upon logging into their worker account. We will use MTurk’s qualifications to recruit respondents based on reported geographical location and a prior approval rating of at least 80%. The approval rating measures the ability of a worker to provide successful results for various types of tasks across MTurk. An approval rating of at least 80% allows access to a larger worker pool and for the inclusion of new workers who may have less experience on the MTurk site but are still able to provide quality work. The first 1,200 qualified MTurk workers determined to be eligible will be surveyed via a web instrument (the survey facsimile is provided in Attachment I), which can be completed on any desktop or mobile device that is web-capable. The survey will require a maximum of 10 minutes to complete. To appeal to potential respondents and increase the likelihood of participation, survey respondents will be paid $2.00 for a completed survey. This rate of pay is expected to shorten the data collection period because advertisements for potential tasks available to MTurk workers are prioritized by pay. This rate is slightly higher than the common practice for MTurk workers, which is based on the federal minimum wage of $7.25/hour for their participation.

The results of this opinion survey, in combination with results from the other pretesting activities currently underway (OMB# 1850-0803 v.243), will be used to inform which items will be included in the NPSAS:20 full-scale student survey instrument.

Estimated Respondent Burden

The estimated respondent burden for this testing is provided in table 1.

Table 1. Estimated respondent burden

	Number of respondents	Number of responses	Minutes per respondent	Total burden hours
Opinion survey with students using MTurk survey	1,200	1,200	10	200

Estimate of Costs for Recruiting and Paying Respondents

In order to be able to recruit a representative range of respondents and to thank them for their time and participation, we will offer participants $2.00 for completing the 10 minute interview.

Estimate of Cost Burden

There are no direct costs for respondents.

Cost to Federal Government

The cost to the federal government for conducting the pretesting activities requested in this submission is approximately $3,045. This cost includes recruitment and screening and participant incentives.

Assurance of Confidentiality

Participants will be informed that their participation is voluntary and that:

This survey is being administered by RTI International on behalf of the National Center for Education Statistics (NCES) under OMB No. 1850-0803 and resides on a server outside of the NCES domain. NCES cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this survey.

Schedule for NPSAS:20 OMB Requests and Related Activities

Recruiting for and the testing of the online opinion survey will begin upon OMB approval in March 2019.