Justification

National Center for Education Statistics

Volume I

Supporting Statement

2015-16 National Postsecondary Student Aid Study (NPSAS:16) Student Survey and Student Records Qualitative Evaluation

OMB# 1850-0803 v.137

Attachments:

Attachment I – Student Cognitive Interview Respondent Contacting Materials

Attachment II – Student Cognitive Interview Consent to Participate in Research

Attachment III – Focus Group Respondent Contacting Materials

Attachment IV – Focus Group Consent to Participate in Research

Attachment V – Student Cognitive Interview Recruitment and Eligibility Screening Materials

Attachment VI – Focus Groups Recruitment and Eligibility Screening Materials

Attachment VII – Student Cognitive Interview Protocol

Attachment VIII – Student Cognitive Interview Instrument Facsimile

Attachment IX – Focus Groups Moderator’s Guide

Attachment X – Focus Groups Student Records Instrument Facsimile

May 4, 2015

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as pilot tests, focus groups, and cognitive interviews, to improve methodologies, survey questions, and/or delivery methods.

This request is to conduct qualitative testing in June 2015 in preparation for the 2015-16 National Postsecondary Student Aid Study (NPSAS:16) full-scale data collection (OMB# 1850-0666), which will begin in February 2016. RTI International will collect NPSAS:16 data on behalf of NCES under contract to the U.S. Department of Education (Contract # ED-IES-13-C-0070). Shugoll Research is RTI’s subcontractor for aspects of NPSAS:16 qualitative testing. The student survey will be tested using cognitive/usability interviews with students, while aspects of the student records collection will be explored using focus groups with postsecondary institution staff.

This document describes recruiting, screening, and cognitive/usability interview procedures for questionnaire items that are being considered for inclusion in the NPSAS:16 full-scale data collection. The interviews are designed to ensure quality, performance, and reliability of the current and new NPSAS:16 survey items and of the overall usability of the survey. Interview results will be used to refine the full-scale survey items. The finalized survey will be submitted to OMB for review in November 2015 as part of the NPSAS:16 full-scale student survey submission.

This document also describes recruiting, screening, and moderating procedures for the focus group sessions. The sessions will solicit feedback from institution staff about their experiences participating in the NPSAS:16 field test student records collection. Focus group results will be used to refine the usability of the instrument for the NPSAS:16 full-scale data collection.

Background

NPSAS:16, which is conducted by NCES, is a nationally representative study of how students and their families finance education beyond high school. NPSAS:16, the ninth cycle in the series, will be conducted from February to September 2016 to capture information pertaining to the 2015–16 academic year. NPSAS:16 will also serve as the base year data collection for the Baccalaureate and Beyond Longitudinal Study (B&B) – a study of college graduates one year (B&B:16/17), four years (B&B:16/20), and possibly ten years (B&B:16/26) following completion of their bachelor’s degree.

Cognitive/Usability Testing of the Student Survey

Results of the cognitive/usability testing will be used to:

• Evaluate the field test survey changes implemented based upon the 2014 TRP recommendations and the graduate students cognitive interviews conducted from August through September 2014;

• Evaluate the extent to which terms in questions are comprehended;

• Update and add terminology;

• Examine the thought processes used to arrive at answers to survey questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and respondent stress;

• Observe how real users interact with the student survey optimized during the field test to adjust to different screen sizes, including smaller mobile devices; and

• Elicit feedback on the survey design and ease of survey navigation on all devices including desktop, laptop, and mobile devices (tablet or smartphone).

Focus Groups with Institution Staff

For student records, the results of focus groups will help to improve the usability of the instrument and the quality of the data collected, with the goal of minimizing burden on participating institutions. A prior round of focus groups, conducted in 2014, focused primarily on instrument terminology, such as establishing clear definitions of the data elements requested. The purpose of this second round of focus groups is to obtain participants’ feedback on the instrument that the participants, themselves, used during the NPSAS:16 field test data collection. Participants, recruited from the set of institutions that completed the NPSAS:16 field test student record collection, will be asked about the usability and function of the instrument; in addition, some of the items that were revised based on feedback from the first round of focus groups will be revisited to ensure that the updated terms are well understood by institution staff.

Design and Context

The purpose of this research is to recruit and perform qualitative testing with the types of individuals that will participate in the NPSAS:16 full-scale data collection. Shugoll Research (hereafter referred to as Shugoll) will take responsibility for recruiting all respondents for both the cognitive/usability testing of the student survey and the institution staff focus groups, providing the research facilities, conducting the interviews, compiling video and audio recordings of the interviews, and reporting the results. Shugoll’s staff has extensive experience in qualitative testing methodologies and is well-equipped both in resources and expertise to perform the task at hand.

Cognitive/Usability Testing of the Student Survey

For the student survey, cognitive/usability testing will be conducted with the types of students that will comprise the NPSAS:16 full-scale sample: graduate student respondents, B&B-eligible respondents (who completed requirements for a bachelor’s degree in the 2014-2015 school year), and other undergraduate student respondents.

Cognitive and usability testing will be conducted simultaneously using the programmed NPSAS:16 field test survey and the addition of several new items (see the survey facsimile, Attachment VIII, for all items to be tested). For the cognitive testing component of the interviews, respondents will be asked to “think aloud” as they come up with their responses to each question, and will be prompted to explain the mental steps they took to arrive at the answer. This will help in identifying why students answer in certain ways and help in the wording of future questions to improve comprehension. Interviewers will also be using “general probing” throughout the duration of the interview when respondents give an indication of difficulty with the question. General probes are generic in nature and aim to identify the source of confusion and level of respondent understanding (see Attachment VII for a list of general probes). In addition, “specific probes” will be used during the cognitive interview process. Specific probes are prepared before testing and will be administered to each respondent on predetermined questions (see the survey facsimile, Attachment VIII, for a list of targeted survey items and their specific probes). These specific probes will be administered at the same time the respondent answers those questions, and will garner feedback to targeted items of interest about student enrollment, financial aid, financial literacy, employment, and usability of the survey, among others.

For the usability testing component of the interviews, interviewers will observe and probe on respondents’ ease of navigation through the survey, and will debrief respondents following the survey on their overall experience with answering questions, specifically on their desktop, laptop, or mobile device.

Interviews will be conducted remotely where respondents use their own computers or mobile devices to go through the interview. Remote testing is convenient/flexible for respondents because they can schedule the session to fit their needs and can participate from their home, school, or other location. It allows respondents to use the survey in a real world environment rather than in a lab setting. Shugoll’s web-based remote interviewing/usability solution includes webcam technology, streaming video, and an audio connection to provide real-time face-to-face interaction between the respondent and interviewer via a shared desktop on their computer and will evaluate the efficacy of self-administered interviews on several devices: laptop, desktop, or mobile device (tablet or smartphone). Respondents who do not have access to a webcam will be provided with one for use during the testing session. After the testing is completed, respondents will return the webcam using a postage-paid package provided by Shugoll.

Each interview is conducted through an audio connection as both the respondent and interviewer view the survey. Those recruited for usability testing on a mobile device will have been sent a document camera and Shugoll’s project technical advisor will instruct the respondent on how to position the camera over their mobile device. In so doing, their mobile device screen can be viewed on their computer as they are navigating the questionnaire. This allows both the interviewer and respondent to view the mobile screen in real-time. This same methodology applies for those completing the questionnaire on a desktop or laptop, but no document camera is necessary for this type of testing. Observers can log on, watch the respondent’s face, listen to the interview, follow the respondent's screen as they complete the survey on their computer or mobile device, and listen to the debriefing. Observers can also communicate with each other and the interviewer via a separate chat room.

A total of 30 students will be invited to participate. This will include an equal distribution of graduate students, B&B-eligible students, and other undergraduates. See Table 1 for the expected number of recruits and respondents by type. Additionally, Shugoll will recruit a robust sample of students attending 4-year institutions, 2-year institutions, and less-than-2 year institutions.

Table 1: Recruitment and Respondent Numbers by Degree Type

Degree Type	Recruited	Respondents
Graduate	20	10
B&B-eligible	20	10
Other undergraduates	20	10
Total	60	30

To meet these intended numbers, Shugoll will identify and recruit 60 students. Recruits will be identified using Shugoll’s own database of potential research respondents in the Washington, DC metro area. The database includes information on key demographic criteria including college/university, gender, age, and race/ethnicity. To supplement the database, Shugoll may also reach out to professors and administrators at institutions in the Washington, DC area with whom they have developed relationships. Referrals, advertisements in student newspapers and online forums, and social media postings may also be used to recruit student respondents.

Trained staff from Shugoll will administer a recruitment screener to interested students, which will include questions that ensure respondents qualify for the research. Each interview will last a maximum of 90 minutes, and all respondents will receive $40 as a token of appreciation for their participation. Audio and video recordings of each interview will be available to NCES for review.

Attachment I, submitted with this memorandum, presents the materials that will be used for recruitment of interview respondents. Attachment V presents the screening questions that will be used to determine eligibility for cognitive/usability testing. Attachment II contains the Assurance of Confidentiality. Attachment VII contains the cognitive/usability interviewing protocol, and Attachment VIII presents a facsimile of the student survey, including a table listing items with specific probes.

Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. Shugoll will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to NCES and to RTI’s NPSAS:16 project staff.

Focus Groups with Institution Staff

Focus group participants will be drawn from institutions that completed the NPSAS:16 field test student records collection. Selected participants will be the individuals responsible for collecting and providing the requested data. Sampled individuals will be contacted by telephone and screened to ensure they are responsible for completing or overseeing completion of the student records collection. Once the individuals have verified their responsibility for completing student records in NPSAS:16, they will be invited to participate in one of the three focus groups. Participants will need to have access to a computer with high speed internet access at the location where they will be participating in the focus group. Because all sessions will be conducted remotely through an online video, participants’ computers must have a webcam. Participants who do not have access to a webcam will be provided with one for use during the focus group session. At the conclusion of the focus group session, participants will return the webcam using a postage-paid package provided by Shugoll.

Participants will be recruited for one of three focus group sessions, based on the instrument mode they selected to complete the NPSAS:16 field test student record collection – Web, Excel template, or CSV modes. Up to 10 participants will be recruited from the institutions:

Group 1: Web mode – that completed NPSAS:16 field test using the web-based instrument.

Group 2: Excel template mode – that provided data using the Excel template.

Group 3: CSV mode – that provided data as CSV files.

Some staff may not be able to participate simply because they are unavailable during the focus group session; therefore, we plan to select up to 50 potential institutions for each focus group with the aim of recruiting 10 participants per session. To the extent possible, Web and Excel mode participants will be selected such that a range of institution sizes and types are represented. Because of the smaller size of the field test, it is possible that a small number of institutions will opt for the CSV mode; therefore, all CSV completers from the field test collection may be invited to participate in the focus group. Table 2 lists the expected distribution of focus group participants by mode.

Table 2: Focus Group Composition by Mode

NPSAS:16 Field test Mode	Maximum Number of Contacted Institutions	Focus Group Participants
Web mode (Group 1)	50	10
Excel mode (Group 2)	50	10
CSV mode (Group 3)	50	10
Total	150	30

Recruiters will collect the e-mail addresses of those who agree to participate in the research and a confirmation e-mail will be sent to participants with information about the study and the date and time of their focus group. A link to the online video interface for the focus group and log-in information will be provided in another e-mail the day before the focus group session. Right before the focus group begins, all participants will be re-screened and a technical check will be performed to ensure that they can log into the focus group interface and that all equipment, including the webcam, is functioning properly.

Attachment VI, submitted with this memorandum, presents the screening questions that will be used to determine eligibility for participating in focus groups. Attachment III presents the materials that will be used for contacting institution staff. Attachment IV contains the Assurance of Confidentiality, which all participants will be asked to sign before the session begins.

Trained staff from Shugoll will remotely moderate the focus group sessions. The moderator will guide the group discussion, following a list of pre-determined discussion topics. The moderator’s guide for conducting the focus group sessions, including the list of discussion topics, is presented in Attachment IX. Some topics will be discussed by all groups, while other topics will be specific to an individual group. Participants will be asked about their understanding of data elements requested and their experiences using the students records instrument to provide data for the NPSAS:16 field test. The field test student records instrument is provided for reference in Attachment X. The programmed field test instrument may be presented as a visual aid to facilitate discussion, but the instrument will not be administered during the session. Participants will be asked for feedback concerning their prior experiences using the instrument and their suggestions for future improvements. Each focus group session will last for a maximum of 90 minutes, and participants will be offered a $25 Amazon gift card to thank them for their time.

Audio and video recordings of each interview will be available to NCES for review. Immediately following the conclusion of each session, methodologists will review the session recordings and notes, highlighting potential themes that may have arisen, and archive the digital audio recording for qualitative analysis. Shugoll will organize the observations and summarize the common themes, insights, and ideas emerging from each of the sessions into a report that will be submitted to NCES and to RTI’s NPSAS:16 project staff.

Assurance of Confidentiality

Both cognitive/usability interview and focus group will be informed that their participation is voluntary and that:

“Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES) of the U.S. Department of Education. NCES is authorized to conduct this study under the Education Sciences Reform Act (20 U.S.C., § 9543). Your participation is voluntary. Your responses are protected from disclosure by federal statute (20 U.S.C., § 9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.”

All respondents will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials together. The respondent ID will not be linked to the respondent’s name in any way. Respondents will be sent a consent to participate in research form via email, which they will need to sign, scan, and send back to Shugoll’s office in order to confirm their participation. The signed consent forms will be kept separately from the interview files for the duration of the study and will be destroyed after the final report is released.

Schedule for NPSAS:16 Qualitative Testing Activities

Shugoll will begin to recruit upon receiving OMB clearance. Cognitive testing and focus group sessions will take place from June through July of 2015.

Estimates of Respondent Burden

For the cognitive/usability testing, up to 60 respondents will be recruited, and up to 30 individual cognitive interviews are planned. Interview sessions are expected each to be a maximum of 90 minutes.

For each of three focus group sessions, up to 50 institutions will be contacted to recruit 6 to 10 participants per session (not to exceed an overall total of 30 participants). Each session is expected to last approximately 90 minutes.

Total burden is provided in table 3.

Table 3. Estimated Cognitive/Usability Interview Respondent Burden

	Number of Respondents*	Number of Responses	Minutes per respondent	Total Burden Hours
Cognitive/Usability Interview
Recruitment	60	60	4	4
Interview Participation	30	30	90	45
Subtotal	60	90	-	49
Focus Groups
Recruitment	150	150	4	10
Focus Group Participation	30	30	90	45
Subtotal	150	180	-	55
Total	210	270	-	104

* Subset of the screened group.

Estimate of Costs for Recruiting and Paying Respondents

Each cognitive/usability interview respondent will receive $40 to participate to encourage them to participate. No token of appreciation will be offered for the initial screening process as the response burden is minimal.

To maximize the likelihood of participation, focus groups will be held during the work day. However, taking time in an individual’s work day may necessitate that the time away from work be made up either by an extended day or on a weekend. Given that potential inconvenience and to encourage particpation, we will offer an emailed $25 gift certificate from Amazon.com. In addition, some staff may not be able to personally accept this token and have to turn it over to the institution. In such cases, this incentive amount will serve to express gratitude to the institution for any inconvenience they experience in allowing their employee to participate in the 90 mintes focus group.

Estimate of Cost Burden

There are no direct costs for to respondents.

Cost to Federal Government

The cost to the federal government for conducting cognitive interviews will be $72,700 and for focus groups $36,000, for a total of $108,700 under the Shugoll subcontract to RTI. This cost includes recruitment, conducting interviews, focus group moderation, analyses, report writing, and participant incentives.