Justification

National Center for Education Statistics

Volume I

Supporting Statement

2008-18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) Cognitive and Usability Testing

OMB# 1850-0803 v.204

August 2017

Attachments:

Attachment I – Recruitment Procedures and Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Research

Attachment IV – Interview Protocol

Attachment V – Interview Facsimile

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey questions, and/or delivery methods.

This request is to conduct cognitive and usability testing starting in August 2017 in preparation for the 2008/18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) full-scale data collection (OMB# 1850-0729), which will begin in July 2018. RTI International will collect B&B:08/18 data on behalf of NCES under contract to the U.S. Department of Education. Shugoll Research (hereafter referred to as Shugoll) is RTI’s subcontractor for aspects of B&B:08/18 cognitive and usability testing.

The overarching purpose of B&B is to collect data on labor market outcomes, employment experiences, family formation, and debt and finances. Many survey items planned for the upcoming B&B:08/18 full-scale data collection have been previously tested or were included in prior B&B surveys or other NCES studies. The cognitive and usability testing described in this submission allows NCES to test, before their inclusion in the B&B:08 full-scale data collection, selected survey items that are either new to this B&B cohort or have been revised from existing items. Specifically, the items to be tested are from an enhanced employment section that aims to collect employment and K-12 teaching experiences across a six-year period as efficiently as possible, and a background section that includes items related to family formation, student loan debt, and financial well-being. In addition to survey items, a portion of the cognitive interview will be dedicated to asking participants about the accuracy and completeness of their résumés to ascertain the viability of using résumés as an additional source of information about employment outcomes.

This submission describes the cognitive and usability testing recruitment, screening, and procedures designed to ensure quality, performance, and reliability of the tested items and of the overall survey usability. The results will be presented to a TRP panel as discussion for potential survey modifications which will be used to refine the full-scale survey, and will be submitted to OMB for review in March 2018 as part of the B&B:08/18 full-scale data collection request.

Background

The B&B-eligible cohort is initially identified in the National Postsecondary Student Aid Study (NPSAS). The B&B:08 cohort was selected from NPSAS:2008, which became the base year for follow-up interviews in 2009 and 2012. B&B examines students’ education and work experiences after they complete a bachelor’s degree, with a special emphasis on the experiences of K-12 school teachers. The B&B:08 cohort also includes an over-sample of graduates in the fields of science, technology, engineering, and mathematics (STEM). The B&B:08/18 survey builds upon the previous interviews and will collect information about the six-years since the last survey in 2012.

The requested cognitive and usability testing will be used to refine the survey questions, maximize the quality of data collected, and provide information on issues with important implications for the survey design, such as the following:

• Identify whether respondents can provide accurate data across a six-year span of time;

• Evaluate the extent to which terms in questions are comprehended;

• Update and add terminology;

• Examine the thought processes used to arrive at answers to survey questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and respondent stress;

• Observe how real users interact with the survey which has been optimized to adjust to different screen sizes, including smaller mobile devices;

• Elicit feedback on the survey design and ease of survey navigation on all devices, including desktop, laptop, and mobile devices (tablet or smartphone); and

• Evaluate the accuracy of information provided on résumés.

Design and Context

The purpose of this study is to conduct cognitive and usability testing with individuals who are similar to the B&B:08 cohort. Shugoll staff have extensive experience in cognitive and usability testing methodologies, and will recruit cognitive and usability testing participants, conduct the interviews, compile interview video and audio recordings, and report the results.

Cognitive and usability testing will be conducted simultaneously using a subset of items proposed for inclusion in the B&B:08/18 full-scale survey (see the survey facsimile, Attachment V, for all items to be tested). For the cognitive testing component of the interviews, respondents will read the questions quietly to themselves and will be asked to “think aloud” as they come up with their responses to each question. They will be prompted by the interviewer to explain the mental steps they took to arrive at the answer. Interviewers will also use “general probing” throughout the duration of the interview when respondents give an indication of difficulty with the question to identify the source of confusion (see Attachment IV for a list of general probes). In addition, “specific probes” will be administered to respondents on targeted questions after the respondent has read the question and provided a response. After a response has been provided and the respondent clicks “Next,” a prompt will appear on screen alerting the interviewer and respondent that we are interested in learning more about this question through specific probes (see the survey facsimile, Attachment V, for a list of targeted survey items and their specific probes). These three types of probes will help to identify how respondents understand the questions and formulate their answers to help evaluate and revise question wording as needed.

All interviews will be conducted remotely requiring respondents to have access to a desktop or laptop computer with a high-speed internet connection to establish an audio and video connection with the interviewer. A total of 30 respondents will be invited to participate. Of these 30 interviews, 20 respondents will complete the survey via the desktop or laptop computer. The remaining 10 respondents will complete the survey on a mobile device, with 5 respondents using smartphones and 5 using tablets. The 10 respondents completing a mobile survey will use their desktop or laptop computer for interviewing purposes including audio and video capabilities to share information with the interviewer, while completing the survey via the mobile device.

Remote testing is convenient and flexible for respondents because they can schedule the session to fit their needs and can participate from their home, school, or other location. It allows respondents to use the survey in a real-world environment rather than in a lab setting. Shugoll’s web-based remote interviewing/usability solution includes webcam technology, streaming video, and an audio connection to provide real-time face-to-face interaction between the respondent and interviewer via a shared desktop on their computer and will evaluate the efficacy of self-administered interviews on several devices: laptop, desktop, and mobile device (tablet or smartphone). Respondents who do not have access to a webcam will have one shipped to their address for use during the testing session. After the testing is completed, respondents will return the webcam using a postage-paid packaging provided by Shugoll.

Each interview will be conducted through an audio connection while both the respondent and interviewer view the survey. Those recruited for usability testing on a mobile device will have been sent a document camera and Shugoll’s project technical advisor will instruct the respondent on how to position the camera over their mobile device. In so doing, their mobile device screen can be viewed on their computer as they are navigating the interview. This allows both the interviewer and respondent to view the mobile screen in real-time. This same methodology applies for those completing the interview on a desktop or laptop, but no document camera is necessary for this type of testing. Observers can log on, watch the respondent’s face, listen to the interview, follow the respondent's screen as they complete the survey on their computer or mobile device, and listen to the debriefing. Observers can also communicate with each other and the interviewer via a separate chat room.

For the usability testing component of the interviews, interviewers will observe and probe on respondents’ ease of navigation through the survey, and will debrief respondents following the survey on their overall experience with answering questions on their desktop, laptop, or mobile device (as applicable).

This sample will include individuals who received a baccalaureate degree between July 1, 2007 and June 30, 2008 and who have similar characteristics as those who will participate in the B&B:08/18 full-scale data collection, specifically those who have taught at the K-12 level or have attended a postsecondary institution for postbaccalaureate education (as identified in the eligibility screener, see Attachment II for specific eligibility screener questions). Of the 30 respondents, 15 will be asked to provide résumés and participate in a targeted discussion about the accuracy of their résumés. See Table 1 for the expected number of testing participants by respondent type.

Recruits will be identified using Shugoll’s database of potential research respondents in the Washington, DC metro area. The database includes information on key demographic criteria, including college/university, gender, age, and race/ethnicity. Referrals, advertisements in student newspapers and online forums, and social media postings may also be used to recruit respondents. All recruitment of potential cognitive interview respondents will be conducted using an online recruitment screener containing eligibility criteria questions specific to this study to ensure that testing participants qualify for the study.

Audio and video recordings of each interview will be available to NCES and B&B:08/18 staff at RTI for review. Immediately following the conclusion of each interview, Shugoll will organize their observations and summarize the common themes and insights from the interviews to date.

Table 1: Screening and Participant Numbers by Respondent Type

Respondent Type	Screened	Testing Participants
Teachers	225	15
Postbaccalaureate Enrollment	75	5
Other Bachelor’s Degrees	150	10
Total	450	30

Attachment I in this submition presents the materials that will be used for recruitment of testing participants; Attachment II the screening questions that will be used to determine eligibility for cognitive and usability testing; Attachment III the Assurance of Confidentiality; Attachment IV the cognitive and usability testing protocol; and Attachment V a facsimile of the survey, including a table listing items with specific probes.

Estimated respondent burden

To yield 30 completed interviews, we anticipate screening up to 450 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 4 minutes per person (see Attachment II). Each testing session will last a maximum of 90 minutes.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	450	450	4	30
Cognitive and usability interview	30*	30	90	45
Study Total	450	480		75

* Subset of the screened group.

Estimate of Costs for Recruiting and Paying Respondents

In order to be able to recruit a representative range of respondents, and to thank them for their time and participation, we will offer prospective participants $40 for completing the 90-minute cognitive interview.

Estimate of Cost Burden

There are no direct costs for respondents.

Cost to Federal Government

The cost to the federal government for conducting cognitive interviews will be $99,054 under the Shugoll subcontract to RTI. This cost includes recruitment, conducting interviews, analyses, report writing, and participant incentives.

Assurance of Confidentiality

Cognitive and usability testing respondents will be informed that their participation is voluntary and that:

Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All respondents will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials for each respondent together. The respondent ID will not be linked to the respondent’s name. Respondents will be sent a consent form via email, which they will need to sign, scan, and send back to Shugoll’s office in order to confirm their participation. The signed consent forms will be kept separately from the interview files for the duration of the study and will be destroyed after the final report is released.

Schedule for B&B:08/18 OMB requests and related activities

Shugoll will begin recruiting for the cognitive and usability testing upon receiving OMB clearance, and the testing is scheduled to begin by September 2017. Informed by the testing, a final draft of the survey will be used in a full-scale with approximately 17,040 sample members, beginning in July 2018.

Recruit participants	August 2017
Conduct cognitive testing	September – November 2017
Finalize revisions to item wording	December 2017 – February 2018