Volume I - B&B 2016-20 Tryouts & Focus Groups

National Center for Education Statistics (NCES)

Volume I

Supporting Statement

2016/20 Baccalaureate and Beyond Longitudinal Study (B&B:16/20) Tryouts and Focus Groups

OMB# 1850-0803 v.242

November 2018

Attachments:

Attachment I – Recruitment Procedures and Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Research

Attachment IV – Focus Group Protocol

Attachment V – Survey Facsimile

Attachment VI – Contacting Material Designs

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey questions, and/or delivery methods.

This request is to test sections of the 2016/20 Baccalaureate and Beyond Longitudinal Study (B&B:16/20) full-scale survey on a group of 300 participants using survey tryouts and follow-up focus group sessions. The focus group sessions will include a survey debrief and evaluate selected contacting material designs. This testing will begin in November 2018, in preparation for the B&B:16/20 full-scale data collection (OMB# 1850-0926), which will begin in July 2020. RTI International will collect B&B:16/20 data on behalf of NCES under contract to the U.S. Department of Education. EurekaFacts is RTI’s subcontractor for aspects of B&B:16/20 survey tryouts and focus groups.

The overarching purpose of B&B is to collect data on labor market outcomes, employment experiences, family formation, and debt and finances. Many survey items planned for the upcoming B&B:16/20 full-scale data collection have been previously tested or were included in prior B&B surveys or other NCES studies. The survey tryouts described in this submission allow NCES to test, before their inclusion in the B&B:16/20 full-scale data collection, selected survey items that are either new to this B&B cohort or have been revised from existing items. Specifically, the items to be tested are from: (1) a financial aid section that collects student debt and repayment information, (2) an enhanced employment section that aims to collect employment details and pre-K-12 teaching experiences across a four-year period as efficiently as possible, and (3) a set of items on civic engagement activities. In addition to comprehension and usability of survey items, a portion of the focus groups will be dedicated to asking participants about design options for envelopes, brochures, and infographics to inform design decisions for the B&B:16/20 full-scale data collection contacting materials.

This submission describes the survey tryouts, focus groups, recruitment, screening, and procedures designed to ensure quality, performance, and reliability of testing results. The results will be used to inform potential survey and contacting material modifications that are intended to refine the survey and improve response rates for full-scale data collection, and will be submitted for clearance review in December 2019 as part of the B&B:16/20 full-scale data collection request (OMB# 1850-0926).

Background

The B&B-eligible cohort is initially identified in the National Postsecondary Student Aid Study (NPSAS). The latest in the B&B series, the B&B:16 cohort, was selected from NPSAS:16, which became the base year for the first B&B:16 follow-up interview in 2017. B&B examines students’ education and work experiences after they complete a bachelor’s degree, with a special emphasis on the experiences of pre-K-12 school teachers. The B&B:16/20 survey builds upon the previous interviews and will collect information about either the three years since the last survey in 2017 or, if a sample member did not respond to the first follow-up, the four years since the base year in 2016.

The requested survey tryouts and focus groups will be used to refine the survey questions, maximize the quality of data collected, provide information on issues with important implications for the survey design, and supply respondent feedback on the effectiveness of contacting materials, such as the following:

• Identify whether respondents can provide accurate data across a three- or four-year span of time;

• Evaluate the extent to which terms in questions are comprehended, including updated and added terminology;

• Examine the thought processes used to arrive at answers to survey questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and respondent stress;

• Observe how users interact with the survey, which has been optimized to adjust to different screen sizes, including smaller mobile devices;

• Elicit feedback on the survey design and ease of survey navigation on all devices, including desktop, laptop, and mobile devices (tablet or smartphone);

• Evaluate the appeal of select envelope, brochure, and infographic designs; and

• Assess how likely participants would be to respond during data collection based on the appeal of the contacting materials.

Design and Context

The purpose of this study is to conduct survey tryouts and focus groups with individuals who are similar to the B&B:16 cohort. EurekaFacts staff have extensive experience in cognitive and usability testing methodologies and focus group moderation, and will recruit participants, conduct the focus group sessions, compile audio and video recordings of each session, and report the results.

Survey tryouts will be conducted using a subset of items proposed for inclusion in the B&B:16/20 full-scale survey and debriefing questions embedded after critical survey items to elicit item-level information about cognition, comprehension, and usability (see the survey facsimile, Attachment V, for all debriefing questions and survey items to be tested). All surveys will be completed by respondents as a self-administered web survey. During the B&B:16/17 full-scale study, 89% of respondents participated via Web, while 11% participated via telephone. We expect an even higher percentage of web respondents for the B&B:16/20 full-scale study, so asking tryout respondents to complete the survey via Web will closely replicate a real-world experience for respondents. To help guide respondents through the self-administered survey, on-screen instructions and definitions will be provided on some questions, and help text on all survey forms will be available (see the survey facsimile, Attachment V, for the help text on each survey item). A subsample of survey tryout participants will complete the self-administered survey at EurekaFacts’ offices and will participate in a focus group debriefing session, during which they will be asked to discuss general topics, the usability of the survey, and to react to contacting material designs for full-scale data collection (see Attachment VI for the contacting material designs to be tested; see the focus group protocol, Attachment IV, for a full list of debriefing probes).

In order to achieve 300 completed surveys, we anticipate that up to 4,500 individuals will need to be invited to participate in the screening. Of the 300 self-administered web survey respondents, 250 respondents will complete the survey remotely on a device, such as a desktop or laptop computer, tablet, or smart phone, with high-speed internet access. Remote testing is convenient and flexible for respondents because they can complete the survey when it works within their schedules and can participate from their homes, schools, or other locations. It allows the 250 remote respondents to use the survey in a real-world environment rather than in a lab setting. There are no limits on the number of surveys that can be completed on each type of device. With 300 survey respondents we anticipate that a sufficient number of surveys will be completed with each type of device to identify potential usability concerns by device type and screen size. The self-administered web survey will take approximately 20 minutes to complete.

The remaining 50 self-administered web survey respondents will complete the survey in-person at EurekaFacts offices on a tablet provided by EurekaFacts. Then, they will participate in a focus group in which they will debrief on the survey experience and discuss contacting material designs for full-scale data collection. There will be five focus group debriefing sessions with approximately 10 participants in each. EurekaFacts will schedule each session at a time that is convenient for bachelor’s degree recipients who work and potentially attend postbaccalaureate courses. Each session will last 90 minutes, including approximately 30 minutes for completing the survey and 60 minutes for focus group participation. See Table 1 for the expected number of testing participants by respondent type.

Remote participants will be directed to a landing page, hosted by EurekaFacts, where they will log in with a provided passcode. Survey respondents who also participate in a focus group will complete the survey and focus group in-person and will be given a link at the beginning of the session by the session moderator that will launch them directly into the survey. All in-person sessions will be audio and video recorded with capabilities for live remote observation by B&B staff. Observers can log on in order to watch the respondent’s face and body language and listen to the debriefing.

The target sample will include individuals who received a baccalaureate degree between May 1, 2013 and December 31, 2014 and who have similar characteristics as those who will be part of the B&B:16/20 full-scale data collection, specifically those who have taught at the pre-K-12 level since completing their bachelor’s degree and those who borrowed student loans to pay for their baccalaureate or postbaccalaureate education at a postsecondary institution (as identified in the eligibility screener; see Attachment II for specific eligibility screener questions). While there is not a required number of respondents who have taught since completing their bachelor’s degree or borrowed student loans, recruitment efforts will target respondents who satisfy one or both conditions to ensure adequate representation of these groups in the sample.

EurekaFacts will conduct all recruitment of potentially eligible participants by cold-calling from purchased lists, sending emails and letters to and posting flyers with organizations where bachelor’s degree recipients might work or socialize, posting advertisements on social media, and snowball sampling methods such as referrals and word-of-mouth from other participants. All recruitment of potential survey tryout respondents will be conducted using an online or telephone recruitment screener containing eligibility criteria questions specific to this study to ensure that testing participants qualify for the study (see Attachment II for the eligibility screening questions). Participants completing the survey remotely can be recruited from across the United States while in-person participants will be recruited form the greater Washington D.C. area where EurekaFacts’ offices are located.

Audio and video recordings of each session will be available to NCES and B&B:16/20 staff at RTI for review. Following the conclusion of each session, EurekaFacts will organize their observations and summarize the common themes and insights from the focus groups to date.

Table 1: Screening and participant numbers by respondent type

Respondent type	Screened	Testing participants
Remote survey tryout only	3,750	250
In-person survey tryout with focus group	750	50
Total	4,500	300

Attachment I in this submission presents the procedures and materials that will be used for recruitment of pretesting participants; Attachment II includes the screening questions that will be used to determine eligibility for the survey tryouts and focus groups; Attachment III is the consent form to participate in the research; Attachment IV provides the focus group protocol; Attachment V presents a facsimile of the survey and a table including all survey items with the respective debriefing items embedded in the survey; and Attachment VI includes the contacting material designs to be discussed in focus groups.

Estimated Respondent Burden

To yield 300 completed surveys, we anticipate screening up to 4,500 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 4 minutes per person (see Attachment II). Each remote, self-administered web survey will take approximately 20 minutes to complete while each in-person session, with the survey tryout and focus group, will last a maximum of 90 minutes.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	4,500	4,500	4	300
Remote survey tryout	250*	250	20	84
In-person survey tryout with focus group	50*	50	90	75
Study total	4,500	4,800		459

* Subset of the screened group.

Estimate of Costs for Recruiting and Paying Respondents

In order to be able to recruit a representative range of respondents from among those who received a baccalaureate degree between May 1, 2013 and December 31, 2014 and who have similar characteristics as those who will be part of the B&B:16/20 full-scale data collection (as described earlier in this document), and to thank them for their time and participation, we will offer $30 for completion of a remote survey tryout session and $90 for participation in an in-person survey tryout and a follow-up focus group.

Estimate of Cost Burden

There are no direct costs for respondents.

Cost to Federal Government

The cost to the federal government for conducting cognitive interviews will be approximately $294,780 under the EurekaFacts subcontract to RTI. This cost includes recruitment and screening, administering focus groups, analyses, report writing, and participant incentives.

Assurance of Confidentiality

Survey tryout and focus group participants will be informed that their participation is voluntary and that:

EurekaFacts, LLC and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All respondents will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials for each respondent together. The respondent ID will not be linked to the respondent’s name. Respondents will be given a consent form via email or in-person, which they will need to sign and return back to the EurekaFacts office in order to confirm their participation. The signed consent forms will be kept separately from all survey data and focus group files for the duration of the study and will be destroyed after the final report is completed.

Schedule for B&B:16/20 OMB Requests and Related Activities

Recruiting for the survey tryouts and focus groups will begin upon receiving OMB clearance, in November 2018. Informed by the testing, a final draft of the survey will be developed for full-scale data collection to be administered to the approximately 27,000 B&B:16 sample members, beginning in July 2020.

Recruit participants	November 2018
Conduct survey tryouts and focus groups	November 2018 – June 2019
Finalize revisions to item wording	June 2019 – December 2019