Change Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: April 16, 2020

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, OMB Liaison, Avar Consulting in contract with NCES

FROM: David Richards, NCES

SUBJECT: 2020-22 Beginning Postsecondary Students Longitudinal Study (BPS:20/22) Qualitative Testing Recruitment Update (OMB# 1850-0803 v.267 – revised v.260)

The 2020-22 Beginning Postsecondary Students Longitudinal Study (BPS:20/22) is one of several studies conducted by NCES to respond to the need for a national, comprehensive database concerning significant issues in access, choice, enrollment, persistence, progress, and attainment in undergraduate postsecondary education. Students in the BPS:20/22 sample are initially identified in the 2019-20 National Postsecondary Student Aid Study (NPSAS:20). NPSAS:20 is a cross-sectional study that examines the characteristics of students in postsecondary education, with special focus on how they finance their education. BPS:20/22 is the first follow-up survey with a subsample of NPSAS:20 sample members who were identified as first-time beginning college students (FTBs) during 2019-20 academic year. A second BPS follow-up is planned for 2025.

As a longitudinal study, BPS:20/22 is designed to follow a cohort of students who enroll in postsecondary education for the first time during the NPSAS academic year of interest, irrespective of the date of high school completion. The study collects data on student persistence in, and completion of, postsecondary education programs; their transition to employment; demographic characteristics; and changes over time in their goals, marital status, income, and debt, among other indicators. Data from BPS are used to help researchers and policymakers better understand how financial aid influences persistence and completion, what percentages of students complete various degree programs, what early employment and wage outcomes are for certificate and degree attainers, and why students leave postsecondary education.

Cognitive and usability testing is conducted to ensure quality, performance, and reliability of student survey questions. The results will be presented to a Technical Review Panel (TRP) for discussion of potential survey modifications prior to the field test and will be submitted to OMB for review in November 2020 as part of the BPS:20/22 field test data collection request.

This request seeks to do two things: (a) supplement cognitive and usability subject recruiting procedures to include use of Amazon’s MTurk, and (b) expand dates of eligibility to exclude fewer potential participants.

MTurk: Thus far recruiting eligible subjects has been challenging, with only six of a planned 30 interviews completed. Adding MTurk, an online crowdsourcing platform, will allow NCES to administer an eligibility screener quickly to a new group of potential subjects. MTurk was recently used successfully for online opinion survey research for the NPSAS:20 survey. NPSAS:20’s use of MTurk was based on recommendations from the National Center for Science and Engineering Statistics (NCSES) at the National Science Foundation (NSF), the Bureau of Labor Statistics (BLS), and the National Cancer Institute (NCI) at the National Institutes of Health (NIH).

We will use the premium qualifications features of MTurk to recruit respondents based on reported education levels, which will be matched to those expected of BPS:20/22 sample members. Approximately 1,200 additional respondents will be screened via the eligibility screener. Respondents determined eligible will then be followed up with to schedule cognitive interviews.

Dates of Eligibility: In addition to revising recruitment procedures, we are also revising the first attendance date range to expand eligibility of potential participants. In order to reduce the eligibility restrictions, we are widening the dates of eligibility to include those who first attended postsecondary education anytime between January 1, 2017 and June 30, 2020. The survey questions will remain focused on attendance and other activities between the 2017-2018 academic year (July 1, 2017 – June 30, 2018) and June 30, 2020. However, item wording for five questions was modified to remove references to the academic year when the subject was first enrolled to accommodate the widened eligibility range.

Changes in this request include revisions to the following Volume I sections: Submittal-Related Information, Design and Context, Estimated Respondent Burden, Estimate of Costs for Recruiting and Paying Respondents, and Schedule. Additionally, changes include revision to Attachment I Recruitment Procedures and Materials, Attachment II Eligibility Screening Questions, Attachment III Consent to Participate in Cognitive Interview, and Attachment V Survey Facsimile. Interview protocol (Attachment IV) is unchanged. Although respondent burden has changed, the estimated cost to the federal government is unchanged.

The approved version of Volume 1, Attachment I, Attachment II, Attachment III, and Attachment V have been revised to reflect the changes to recruiting procedures. Changes are listed below. Text added since the last approved version is marked burgundy font color. Text deleted since the last approved version is marked in crossed-out burgundy font color. Text moved is marked in double-strikethrough. All unchanged text is shown in black font.

The following updates were made to Vol I:

Submittal-Related Information

The following was added on p. 3:

The request to perform cognitive and usability testing was approved in December 2019 (OMB # 1850-0803 v.260). Based on results of recruiting efforts, conducted from January to April 2020, this request includes updates to Volume 1 to add additional recruiting methods and to broaden the timeframe of eligibility. Revisions have been made to the following sections of Volume 1: Design and Context, Estimated Respondent Burden, Estimate of Costs for Recruiting and Paying Respondents, Costs to Federal Government, and Schedule. Changes to attachments I, II, III, and V reflect these additional recruiting methods and updates to the academic year when the subject was first enrolled to accommodate the widened eligibility range.

Modifications to Design and Context

The following was added on pp. 5-6:

The cognitive and usability testing sample will include individuals who first began their postsecondary education since between July January 1, 2017and June 30, 2018, and who have similar characteristics as those who will participate in the BPS:20/22 field test data collection (as identified in the eligibility screener; see Attachment II for specific eligibility screener questions). The sample will include both degree and certificate completers, those who left prior to completing a degree, and those still enrolled. Individuals will be recruited from institutions with varied characteristics, including level of degree offered (less-than 2-year, 2-year, and 4-year) and control (public, private not-for-profit, private for-profit). See Table 1 for the expected number of testing participants by respondent type.

Recruits will be identified using two methods, by EurekaFacts, and a supplemental recruitment process using Amazon’s MTurk. Both methods of recruiting potential cognitive interview respondents will be conducted using an online recruitment screener containing eligibility criteria questions specific to this study to ensure that participants qualify for the study (see Attachment II for specific eligibility screener questions).

For the first method, recruits will be identified using EurekaFacts database of potential research respondents in the Washington, DC metro area. The database includes information on key demographic criteria, including gender, age, and race/ethnicity, which will be used to diversify the sample within the constraints identified in the previous paragraph. Referrals, advertisements in student newspapers and online forums, and social media postings may also be used to recruit respondents. All recruitment of potential cognitive interview respondents well be conducted using an online recruitment screener containing eligibility criteria questions specific to this study to ensure that testing participants qualify for the study.

Additionally, supplemental recruitment will be conducted by Amazon’s MTurk, which will allow NCES to quickly recruit participants by administering the eligibility screener using the online crowdsourcing platform. Based on recommendations from the National Center for Science and Engineering Statistics (NCSES) at the National Science Foundation (NSF), the Bureau of Labor Statistics (BLS), and the National Cancer Institute (NCI) at the National Institutes of Health (NIH), NCES used MTurk successfully to complete an online opinion survey research for the NPSAS:20 survey.

This supplemental recruitment will be conducted by administering the eligibility screener to participants recruited through Amazon’s MTurk, also referred to as “workers.” “Workers” determined eligible will then be included in EurekaFact’s list of prospective participants. The General Social Media Post (see Attachment I) will be posted to MTurk. Qualified “workers” based on MTurk’s premium qualifications will be able to view the request upon logging into their worker account. We will use the premium qualifications to recruit respondents based on reported education level and a prior approval rating of 80%. MTurk worker approval ratings are based on the percent of work accepted by other requesters. Approximately 1,200 workers will be screened via the eligibility screener (Attachment II). The screener will require up to 5 minutes to complete. Common practice for MTurk workers is to set a rate based on the federal minimum wage of $7.25/hour for their participation. Thus, eligibility screener respondents will be paid $0.60 for a completed screener. This “worker” payment for the MTurk supplemental recruitment screener is considered a typical rate for similar tasks.

Table 1. Screening and participant numbers by respondent type

Respondent type	Screened	Testing participants
Currently enrolled	1,072 265	20
Not currently enrolled	528 135	10
Total	1,600 400	30

Modifications to Estimated Respondent Burden

The following was added on p. 6:

To yield 30 completed interviews, we anticipate screening up to 400 1,600 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent characteristics. The screening process, on average, is estimated to take about 5 minutes per person (see Attachment II). Each cognitive and usability testing session will last a maximum of 60 minutes.

Table 2. Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	1,600 400	1,600 400	5	133 33
Cognitive and usability interview	30*	30	60	30
Study total	1,600 400	1,630 400		163 63

* Subset of the screened group.

Modifications to Estimate of Costs for Recruiting and Paying Respondents

The following was added to p. 6:

In order to be able to recruit a representative range of respondents, and to thank them for their time and participation, we will offer prospective participants a $50 gift card from a major credit card company for completing the 60-minute cognitive interview. Participants of the supplemental MTurk recruitment will be offered $0.60 as payment for spending approximately 5 minutes to complete the screener, this compensation is based on the federal minimum wage of $7.25/hour.

Costs to Federal Government

The following was added to p. 6:

The cost to the federal government for conducting cognitive interviews will be $86,067 under the EurekaFacts subcontract to RTI. This cost includes recruitment, supplemental recruitment via MTurk, conducting interviews, analyses, report writing, and participant incentives. The supplemental MTurk recruitment is estimated to cost approximately $1,700 based on an estimated 1,200 potential subjects spending approximately 5 minutes to complete the screener, and other additional fees.

Modifications to Schedule

The following was added to p. 7:

EurekaFacts will begin recruiting for the cognitive and usability testing upon receiving OMB clearance, and the testing is scheduled to begin by February 2020. Informed by the testing, a final draft of the survey will be used in a field test with approximately 3,500 sample members, beginning in March 2021.

Recruit participants	January – February May 2020
Conduct cognitive testing	February – April May 2020
Finalize revisions to item wording	April May – October 2020

The following updates were made to Attachment I:

Identification of Respondents

The following was added to p. 3:

EurekaFacts will locate potential participants through multiple outreach/contact methods and resources, such as cold calling from directory listings, advertisements within organizations and education institutions, referrals and snowball sampling, and social media posts. Based on experience, EurekaFacts recommends recruitment efforts to be conducted by directly targeting and contacting college students. Our approach to recruitment will utilize four primary vehicles:

• Usage/purchase of student e-mails available from a highly reputable commercial vendor that provides similar direct marketing outreach services for major national corporation and consumer brands.

• Placement of advertisements in highly selective/targeted venues to include Facebook and Google ads where they are restricted to users in university environments or with .edu websites.

• Use of classified advertisements or more general announcements to appear in online and print versions of campus newspapers as well as university-based student affairs/career centers.

Utilize social media and interaction with campus staff and student groups to supplement outreach and amplify visibility and credibility of the recruitment effort.

In response to these outreach/contact methods and resources, potential participants will inform EurekaFacts of their interest to participate in the study by calling or e-mailing the EurekaFacts contact information found in the outreach/contact materials or following a link to the self-administered screener.

In addition to EurekaFacts identification of respondents, the eligibility screener will also be advertised on Amazon’s MTurk crowdsourcing platform, in an effort to quickly recruit potential participants. Qualified “workers” based on MTurk’s premium qualifications, such as education level, will be able to view the request upon logging into their worker account.

EurekaFacts will review the responses to the self-administered screeners of potential participants to identify eligible participants resulting from EurekaFacts’ outreach/contact methods and MTurk “worker” task completions. Potential participants will then be offered the option to participate in either an in-person interview or a remote interview.

Platform advertisement

The following was added to p. 3:

Survey description: Paid Online Study Opportunity – Seeking students who attended a college, university, or trade school within the past three years. Tap here to get started <INSERT LINK TO SCREENER>

Time allotted: 5 minutes

Expires 5/31/2020

Qualifications required: Resides in U.S. and enrolled in postsecondary education

Figure 1. Example advertisement from a previous study

Sample Recruitment Advertisements

The following was added to p. 4:

Advertisements conducted by EurekaFacts will target individuals qualified for BPS:20/22, describing briefly the purpose of the testing and the opportunity for participants to contribute to the development of a survey. The advertisements will identify the need for individuals who attended a college, university, or trade school within the past three years, the time commitment of the interview, and the incentive amount offered for participation.

Sample advertisements using a variety of methods are presented below. The main method will include direct e-mails/ letters sent to individuals and organizations with members who may qualify. Advertisements to organizations will specify the recruitment of first-time beginning students in the 2017-2018 academic year since January 1, 2017 and the initial advertisements to individuals will reference attendance in “the past three years” so that this eligibility requirement is not directly provided to potential participants. The e-mail or letter will also include a flyer and an FAQ page. In addition, social media post language has been included for online advertisements. As part of the recruitment process, recruitment scripts, confirmation phone scripts and e-mails, interview invitation e-mails, and thank you e-mails may be used and are provided in this document.

Recruitment E-mail/Letter to Organizations

The following was added to p. 6:

We are looking for current or recent students whose first attendance at who first attended a college, university, or trade school occurred between July 1, 2017 and June 30, 2018 at some point since January 1, 2017 to participate in voluntary interviews.

Specific Social Media Post

The following was added to p. 9

Seeking students who attended college, university, or trade school for the first time between June 1, 2017 and June 30, 2018 at some point since January 1, 2017.

Specific Flyer

The following was added to p. 12:

Did you first attend a college, university, or trade school at some point since between JulyJanuary 1, 2017 and June 30, 2018?

The following was added to p. 13:

We are looking for current or recent students whose first attendance attended at a college, university, or trade school at some point since occurred between JulyJanuary 1, 2017 and June 30, 2018.

Frequently Asked Questions (FAQs)

The following was added to p. 14:

Some topics may include your enrollment experiences since the 2017-18 academic year January 1, 2017, social and academic activities during college, and any employment information since beginning college.

In-person Participant Confirmation Phone Script

The following was added to p. 18:

During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since the 2017-18 academic year January 1, 2017, social and academic activities during college, and any employment information since beginning college.

Remote Participant Confirmation Phone Script

The following was added to p. 19:

During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since the 2017-18 academic year January 1, 2017, social and academic activities during college, and any employment information since beginning college.

The following updates were made to Attachment II:

The following was added to p. 21:

Respondents who click the screener link provided in the MTurk platform advertisement, targeted social media ads, and e-mails will go directly to the online recruitment screener.

Eligibility Screener (Self-Administered and Over the Telephone)

The following was added to p. 23:

During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since the 2017-18 academic year January 1, 2017. Some topics may include your enrollment experiences since the 2017-18 academic year January 1, 2017, social and academic activities during college, and any employment information since beginning college.

The following was added to p. 24:

1. Were you attending high school or completing any high school requirements at any time between JulyJanuary 1, 2017 and today?

   • Yes  THANK AND TERMINATE (Ineligible)

   • No  CONTINUE

**If Q3 = (JulyJanuary 2017 – June 20182020)  CONTINUE, else THANK AND TERMINATE (Ineligible)

5. Which of the following were you working on at [NPSAS INSTITUTION] when you first attended in the [{if Q3 between Jan 2017 and June 2017} 2016-2017 academic year (July 1, 2016 – June 30, 2017) {else if Q3 between July 2017 and June 2018} 2017-2018 academic year (July 1, 2017 - June 30, 2018) {else if Q3 between July 2018 and June 2019} 2018-2019 academic year (July 1, 2018 – June 30, 2019) {else} 2019-2020 academic year (July 1, 2019 – June 30, 2020)]?

7. Did you complete all the requirements for the [NPSAS DEGREE] you began in the 2017-2018[Q3 MONTH] [Q3 YEAR] academic year at [NPSAS INSTITUTION]?

The following updates were made to Attachment III:

The following was added to p. 28:

Some topics may include your enrollment experiences since the 2017-18 academic year January 1, 2017, social and academic activities during college, and any employment information since beginning college.

The following updates were made to Attachment V:

The following was removed from p. 6:

B22ACMPDGN

[If [started survey before July 1, 2020]]

Have you completed all the requirements for the [NPSAS degree] you began at [NPSAS institution] in the 2017-2018 academic year?

(Answer "No" if you transferred schools and completed your [NPSAS degree] at a different school. [{If usermode = WEB} We will {else} I'll] ask you about your enrollment at any other colleges, universities, or trade schools later.)

[else]

Did you complete all the requirements for the [NPSAS degree] you began at [NPSAS institution] in the 2017-2018 academic year before July 1, 2020? 

(Answer "No" if you transferred schools and completed your [NPSAS degree] at a different school. [{If usermode = WEB} We will {else} I'll] ask you about your enrollment at any other colleges, universities, or trade schools later.)

The following was removed from p. 7:

B22ANENRLFTA

[If [did not complete base-year degree]]

Next, [{if usermode = WEB} we'd {else} I'd] like to ask you to provide the months in which you attended [NPSAS institution] for your [NPSAS degree] two ways. In this first version, [{if usermode = WEB} we {else} I] will begin by collecting information about any full-time attendance. 

Since July 1, [{if [base-year nonrespondent]} 2017, {else} 2018,] [{if B22ACURENR = 1 or [started survey before July 1, 2020]} have you, or do you plan to {else} did you] attend [NPSAS institution] as a full-time student for the your [NPSAS degree] you began in the 2017-2018 academic year? 

[else]

Next, [{if usermode = WEB} we'd {else} I'd] like to ask you to provide the months in which you attended [NPSAS institution] for your [NPSAS degree] two ways. In this first version, [{if usermode = WEB} we {else} I] will begin by collecting information about any full-time attendance. 

Between July 1, [{if [base-year nonrespondent]} 2017 {else} 2018] and June 30, [{if [completed base-year degree in year 2]} 2019 {else} 2020], did you attend [NPSAS institution] as a full-time student for the your [NPSAS degree] you began in the 2017-2018 academic year? 

The following was removed from p. 8:

B22ANENRLPTA

[If [did not complete base-year degree]]

Thank you for providing information about your full-time attendance at [NPSAS institution]. Now [{if usermode = WEB} we'd {else} I'd] like to collect information about any part-time attendance. 

Since July 1, [{if [base-year nonrespondent]} 2017, {else} 2018,] [{if B22ACURENR = 1 or [started survey before July 1, 2020]} have you, or do you plan to {else} did you] attend [NPSAS institution] as a part-time student for the your [NPSAS degree] you began in the 2017-2018 academic year? 

[else] 

Thank you for providing information about your full-time attendance at [NPSAS institution]. Now [{if usermode = WEB} we'd {else} I'd] like to collect information about any part-time attendance. 

Between July 1, [{if [base-year nonrespondent]} 2017 {else} 2018] and June 30, [{if [completed base-year degree in year 2]} 2019 {else} 2020], did you attend [NPSAS institution] as a part-time student for the your [NPSAS degree] you began in the 2017-2018 academic year? 

The following was removed from p. 10:

B22ANPOTHENR

Other than the [NPSAS degree] [{if B22ACMPDGN = 1} you completed {else} you began in the 2017-2018 academic year], have you attended [NPSAS institution] for any degree or certificate programs between July 1, 2017 and [{if [started survey before July 1, 2020]} today {else} June 30, 2020]? 

The following was removed from p. 21:

B22BNUMAPP

Thinking back to when you were first applying to [NPSAS institution] for attendance in the 2017-2018 academic year, how many other colleges, universities, and trade schools did you apply to?