Justification

National Center for Education Statistics

Volume I

Supporting Statement

2008-18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) Cognitive and Usability Testing

OMB# 1850-0803 v.165

August 2016

Attachments:

Attachment I – Recruitment Procedures and Materials

Attachment II – Consent to Participate in Research

Attachment III – Eligibility Screening Questions

Attachment IV – Interview Protocol

Attachment V – Interview Facsimile

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey questions, and/or delivery methods.

This request is to conduct cognitive and usability testing starting in September 2016 in preparation for the 2008/18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) field test data collection (OMB# 1850-0729), which will begin in July 2017. RTI International will collect B&B:08/18 data on behalf of NCES under contract to the U.S. Department of Education. Shugoll Research (hereafter referred to as Shugoll) is RTI’s subcontractor for aspects of B&B:08/18 cognitive and usability testing.

The overarching purpose of B&B is to collect data on labor market outcomes, employment experiences, family formation, and debt and finances. Many survey items planned for the upcoming B&B:08/18 field test data collection have been previously tested or were included in prior B&B surveys or other NCES studies. The cognitive and usability testing described in this submission allows NCES to test, before their inclusion in the B&B:08 field test data collection, selected survey items that are either new to this B&B cohort or have been revised from existing items. Specifically, the items to be tested are from survey sections focused on respondent identity verification, potential use of external sources to provide some of the survey responses, an enhanced employment section that aims to collect employment information across a six-year period as efficiently as possible, and a background section that includes items related to family formation, sexual identification, gender orientation, and the impact of undergraduate education.

This submission describes the cognitive and usability testing recruiting, screening, and procedures designed to ensure quality, performance, and reliability of the tested items and of the overall survey usability. The results will be used to refine the field test survey, which will be submitted to OMB for review in December 2016 as part of the B&B:08/18 Student Interview Field Test submission.

Background

The B&B:08/18 survey, conducted by NCES, has a sample that is drawn from the 2007-08 National Postsecondary Student Aid Study (NPSAS:08). B&B:08/18 is the third and final follow-up of 2007-08 baccalaureate degree recipients and will be the fourth time that the B&B:08 cohort will be contacted. The first contact was made in 2008 as part of NPSAS:08. Those NPSAS:08 sample members who completed a baccalaureate degree were contacted again in 2009 (B&B:08/09) and then again in 2012 (B&B:08/12). The B&B:08/18 student interview builds upon the previous interviews and will collect information about the six year time period from the last collection in 2012 through the present.

B&B:08/18 tracks sample members’ experiences with graduate education, employment, debt and finances, family formation, civic engagement, and volunteerism. There is a strong focus on entry into and persistence within the K-12 teacher workforce. Of particular relevance to B&B:08/18 are graduates in the fields of science, technology, engineering, and mathematics (STEM).

The requested cognitive and usability testing will be used to refine the survey questions, maximize the quality of data collected, and provide information on issues with important implications for the survey design, such as the following:

• Evaluate the extent to which terms in questions are comprehended;

• Update and add terminology;

• Examine the thought processes used to arrive at answers to survey questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and respondent stress;

• Identify whether respondents are able to provide accurate data across a six year span of time;

• Observe how real users interact with the survey which has been optimized to adjust to different screen sizes, including smaller mobile devices; and

• Elicit feedback on the survey design and ease of survey navigation on all devices, including desktop, laptop, and mobile devices (tablet or smartphone).

Design and Context

The purpose of this study is to conduct cognitive and usability testing with the types of individuals who will participate in the B&B:08/18 field test data collection. Shugoll staff have extensive experience in cognitive and usability testing methodologies, and will recruit cognitive and usability testing participants, conduct the interviews, compile interview video and audio recordings, and report the results.

Cognitive and usability testing will be conducted simultaneously using programmed items proposed for inclusion in the B&B:08/18 Field Test survey (see the survey facsimile, Attachment V, for all items to be tested). For the cognitive testing component of the interviews, respondents will be asked to “think aloud” as they come up with their responses to each question, and will be prompted to explain the mental steps they took to arrive at the answer. Interviewers will also use “general probing” throughout the duration of the interview when respondents give an indication of difficulty with the question to identify the source of confusion (see Attachment IV for a list of general probes). In addition, “specific probes” will be administered to respondents on targeted questions at the time each respondent is answering them (see the survey facsimile, Attachment V, for a list of targeted survey items and their specific probes). These three types of probes will help to identify how respondents understand the questions and formulate their answers to help evaluate and revise question wording as needed.

For the usability testing component of the interviews, interviewers will observe and probe on respondents’ ease of navigation through the survey, and will debrief respondents following the survey on their overall experience with answering questions on their desktop, laptop, or mobile device (qs applicable).

Interviews will be conducted remotely allowing respondents to use their own computers or mobile devices to navigate through the interview. Remote testing is convenient and flexible for respondents because they can schedule the session to fit their needs and can participate from their home, school, or other location. It allows respondents to use the survey in a real world environment rather than in a lab setting. Shugoll’s web-based remote interviewing/usability solution includes webcam technology, streaming video, and an audio connection to provide real-time face-to-face interaction between the respondent and interviewer via a shared desktop on their computer and will evaluate the efficacy of self-administered interviews on several devices: laptop, desktop, and mobile device (tablet or smartphone). Respondents who do not have access to a webcam will be provided with one for use during the testing session. After the testing is completed, respondents will return the webcam using a postage-paid packaging provided by Shugoll.

Each interview will be conducted through an audio connection while both the respondent and interviewer view the survey. Those recruited for usability testing on a mobile device will have been sent a document camera and Shugoll’s project technical advisor will instruct the respondent on how to position the camera over their mobile device. In so doing, their mobile device screen can be viewed on their computer as they are navigating the interview. This allows both the interviewer and respondent to view the mobile screen in real-time. This same methodology applies for those completing the interview on a desktop or laptop, but no document camera is necessary for this type of testing. Observers can log on, watch the respondent’s face, listen to the interview, follow the respondent's screen as they complete the survey on their computer or mobile device, and listen to the debriefing. Observers can also communicate with each other and the interviewer via a separate chat room.

A total of 30 respondents will be invited to participate. This sample will include individuals who received a baccalaureate degree between July 1, 2006 and June 30, 2007 and who have similar characteristics as those who will participate in the B&B:08/18 Field Test data collection, specifically those who have taught full-time or part-time at the K-12 level for at least a year or who received a bachelor’s degree in a STEM-related field (as identified in the eligibility screener, see attachment III for specific eligibility screener questions). See Table 1 for the expected number of testing participants by respondent type.

Recruits will be identified using Shugoll’s database of potential research respondents in the Washington, DC metro area. The database includes information on key demographic criteria, including college/university, gender, age, and race/ethnicity. Referrals, advertisements in student newspapers and online forums, and social media postings may also be used to recruit respondents. Trained staff from Shugoll will administer a recruitment screener to interested respondents to ensure that testing particpants qualify for the study.

Audio and video recordings of each interview will be available to B&B:08/18 staff for review. Immediately following the conclusion of each interview, Shugoll will organize their observations and summarize the common themes and insights from the interviews to date.

Table 1: Screening and Participant Numbers by Respondent Type

Respondent Type	Screened	Testing Participants
Teachers	75	5
STEM majors	75	5
Other Bachelor’s Degrees	300	20
Total	450	30

Attachment I in this submition presents the materials that will be used for recruitment of testing participants; Attachment II contains the Assurance of Confidentiality; Attachment III presents the screening questions that will be used to determine eligibility for cognitive and usability testing; Attachment IV contains the cognitive and usability testing protocol; and Attachment V presents a facsimile of the survey, including a table listing items with specific probes.

Estimated respondent burden

To yield 30 completed interviews, we anticipate screening up to 450 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 4 minutes per person (see Attachment III). Each testing session will last a maximum of 90 minutes.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	450	450	4	30
Cognitive and usability interview	30*	30	90	45
Study Total	450	480		75

* Subset of the screened group.

Estimate of Costs for Recruiting and Paying Respondents

In order to be able to recruit a representative range of respondents, and to thank them for their time and participation, we will offer prospective participants $40 for completing the 90-minute cognitive interview.

Estimate of Cost Burden

There are no direct costs for respondents.

Cost to Federal Government

The cost to the federal government for conducting cognitive interviews will be $94,336 under the Shugoll subcontract to RTI. This cost includes recruitment, conducting interviews, analyses, report writing, and participant incentives.

Assurance of Confidentiality

Cognitive and usability testing respondents will be informed that their participation is voluntary and that:

“Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study under the Education Sciences Reform Act (20 U.S.C., § 9543). Your responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).”

All respondents will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials together. The respondent ID will not be linked to the respondent’s name. Respondents will be sent a consent to participate in research form via email, which they will need to sign, scan, and send back to Shugoll’s office in order to confirm their participation. The signed consent forms will be kept separately from the interview files for the duration of the study and will be destroyed after the final report is released.

Schedule for B&B:08/18 OMB requests and related activities

Shugoll will begin recruiting for the cognitive and usability testing upon receiving OMB clearance, and the testing is scheduled to begin by September 2016. Informed by the testing, a final draft of the survey will be used in a field test with approximately 1,580 sample members, beginning in July 2017.

Recruit participants	September 2016
Conduct cognitive testing	September - November 2016
Finalize revisions to item wording	December 2016