Part B BB 2016-2020 FS Panel Maintenance

2016/20 Baccalaureate and Beyond (B&B:16/20) Full-Scale Study Panel Maintenance

Supporting Statement Part B

OMB # 1850-0926 v.8

Submitted by

National Center for Education Statistics

U.S. Department of Education

March 2019

Contents

B. Collection of Information Employing Statistical Methods 1

1. Respondent Universe – B&B:16/20 Target Population 1

2. Statistical Methodology – B&B:16/20 Sample Design 1

3. Methods for Maximizing Response Rates 1

a. Tracing of Sample Members 1

b. Training for Data Collection Staff 2

c. Case Management System 3

d. Survey Instrument Design 3

e. Refusal Aversion and Conversion 4

4. Tests of Procedures and Methods 4

5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study 4

Tables

Table 1. B&B:16/17 full-scale study sample by eligibility status 1

Table 2. Expected B&B:16/20 full-scale study eligibility and response rates by prior round response status 1

2. Collection of Information Employing Statistical Methods

This request is to conduct the 2016/20 Baccalaureate and Beyond Longitudinal Study (B&B:16/20) full-scale study panel maintenance and administrative record matching activities in preparation for the B&B:16/20 full-scale study data collection. Administrative matching activities are helpful for providing up-to-date address information for sample members (for more details, see Supporting Statement Part A section A.2.c). A request to conduct the B&B:16/20 full-scale study data collection, which will provide additional information pertaining to the data collection design, will follow in winter 2019-20. B&B:16/20 is the second follow-up of sample members from the 2015-16 National Postsecondary Student Aid Study (NPSAS:16) who were baccalaureate recipients during the 2015–16 academic year. For details on the NPSAS:16 sample and full-scale study design, see NPSAS:16 Full Scale (OMB# 1850-0666 v. 15-19) Supporting Statement Part B. B&B cohorts prior to B&B:16 are approved under OMB# 1850-0729 while the B&B:16 cohort is approved under OMB# 1850-0926.

1. Respondent Universe – B&B:16/20 Target Population

The target population for B&B:16/20 full-scale study includes all eligible NPSAS:16 sample members who completed requirements for the bachelor’s degree from NPSAS-eligible institutions during the 2015–16 academic year, that is, between July 1, 2015 and June 30, 2016, and were awarded their baccalaureate degree by the institution no later than June 30, 2017. There is a known and well-defined probability of selection for each student in the B&B sample. Through the institution awarding the degree, each completer has exactly one linkage to the B&B sampling frame.

The final weighted response rate for the most recent student interview of the B&B:16 cohort, during the B&B:16/17 full-scale data collection, was 69 percent. The final weighted response rate for the second follow-up of the previous, B&B:08 cohort, during the B&B:08/12 full-scale data collection, was 78.3 percent.

2. Statistical Methodology – B&B:16/20 Sample Design

The sample design for the B&B:16/20 full-scale study will include all eligible sample members from B&B:16/17. A sample member from B&B:16/17 was considered ineligible for the study if they had not completed their degree requirements for a bachelor’s degree at the institution they were sampled from in the NPSAS study during the 2015-15 academic year (July 1, 2015 through June 30, 2016), and was awarded the degree after June 30, 2017. Sample members identified as deceased during the B&B:16/17 data collection will also be excluded from the B&B:16/20 sample. All other previous B&B:16/17 sample members will be included in the B&B:16/20 sample regardless of prior response status. Table 1 shows the distribution of the B&B:16/17 sample by eligibility status to determine the B&B:16/20 sample.

Table 1. B&B:16/17 full-scale study sample by eligibility status

B&B:16/17 status	Count
Total	28,796
Eligible	26,509
Ineligible*	2,259
Deceased	28

* Sample members who did not complete their degree requirements in the 2015-16 academic year plus those who completed their degree requirements in the 2015-16 academic year from the NPSAS school, but was not awarded the degree until after June 30, 2017.

In NPSAS:16, a sample member had to provide specific data elements through the student interview and administrative records in order to meet the minimum requirements to be a study member. Sample members who did not meet these requirements were considered non-study members. Sample members who were non-study members in NPSAS:16 and were identified as part of the B&B:16 cohort will not be fielded in B&B:16/20 and will be considered nonrespondents to the B&B:16/20 student interview. This is similar to what was done in B&B:16/17. Table 2 shows the distribution of the B&B:16/20 sample by NPSAS study member status. Since eligibility for the B&B:16 cohort has not been confirmed for all sample members, additional ineligible sample members may be identified in the course of the B&B:16/20 data collection.

Table 2. Expected B&B:16/20 full-scale study eligibility and response rates by prior round response status

NPSAS:16status	Count
Total	26,509
NPSAS:16 study member (fielded)	25,261
NPSAS:16 non-study member (not fielded)	1,248

3. Methods for Maximizing Response Rates

Achieving high response rates in the B&B:16/20 full-scale study data collection will depend on successfully identifying and locating sample members, facilitated by the panel maintenance activity for which clearance is requested, and being able to contact them and gain their cooperation.

In the previous, most recent B&B cohort, B&B:08, we conducted a full-scale address update/initial contact operation, incentivized with $10, in preparation for the B&B:08/12 student interview data collection, and achieved the final panel maintenance response rate of 42 percent. In preparation for the B&B:08/18 student interview data collection, we conducted in 2018 a full-scale panel maintenance activity without an incentive and achieved the final panel maintenance response rate of 16 percent. For the un-incentivized panel maintenance activity planned to take place during the fall and winter of 2019-20 in preparation for the B&B:16/20 full-scale study – the subject of this submission – we are expecting the response rate to be similar to that of the B&B:08/18 panel maintenance activity, around 15 percent.

As in B&B:08/12, closer to the B&B:16/20 full-scale data collection, we will send an address update/initial contact mailing/email prior to the start of data collection, but we have not yet determined the final design for this operation, including start dates or whether it will be incentivized. The final design will be provided in the B&B:16/20 full-scale study request in winter 2019-20.

The following sections outline methods for maximizing response to the B&B:16/20 data collection.

1. Tracing of Sample Members

To yield the maximum number of located cases with the least expense, we designed an integrated tracing approach, with the following elements.

• Advance tracing activities, which will occur prior to the start of data collection, include initial batch database searches, such as to the National Change of Address (NCOA) databases, for cases with sufficient contact information to be matched. To handle cases for which contact information is invalid or unavailable, B&B staff will conduct additional advance tracing through proprietary interactive databases to expand on leads found.

• Hard copy mailings, emails, and text messages will be used to maintain ongoing contact with sample members, prior to and throughout data collection. A panel maintenance mailing and email will be sent in late 2019/early 2020 to request that sample members update their contacting information. The panel maintenance contacting materials are provided in Appendix C. All other data collection contacting material that will be used after early 2020 will be provided in the winter 2019-20 request for clearance

• At the start of data collection, initial contact letters will request that prior round nonrespondents, including non-study members, update their contact information. A follow-up reminder email will be sent approximately 2 weeks after the initial letter to remind them to respond. Also, at the start of data collection for study members (both prior round respondents and nonrespondents), we will send a letter to announce the start of data collection. The announcement will include a toll-free number, the study website address, and a Study ID and password, and will request that sample members complete the web survey. After the data collection announcement mailing, an email message mirroring the letter will also be sent.

• The telephone locating and interviewing stage includes calling all available telephone numbers and following up on leads provided by parents and other contacts.

• The pre-intensive batch tracing stage consists of the LexisNexis SSN and Premium Phone batch searches that will be conducted between the telephone locating and interviewing stage and the intensive tracing stage.

• Once all known telephone numbers are exhausted, a case will move into the intensive tracing stage during which tracers will conduct interactive database searches using all known contact information for a sample member. During the B&B:16/17 full-scale study, about 89 percent of sample members who reached the intensive tracing stage were located, and about 25 percent of those located responded to the interview. With interactive tracing, a tracer assesses each case on an individual basis to determine which resources are most appropriate and the order in which each should be used. Sources that may be used, as appropriate, include credit database searches, such as Experian, various public websites, and other integrated database services.

• Other locating activities will take place as needed, including a LexisNexis email search conducted for nonrespondents toward the end of data collection.

2. Training for Data Collection Staff

Telephone data collection will be conducted at the contractor’s call center. B&B staff at the call center will include Performance Team Leaders (PTLs) and Data Collection Interviewers (DCIs). Training programs for these staff members are critical to maximizing response rates and collecting accurate and reliable data.

Performance Team Leaders, who are responsible for all supervisory tasks, will attend project-specific training for PTLs, in addition to the interviewer training. They will receive an overview of the study, background and objectives, and the data collection instrument through a question-by-question review. PTLs will also receive training in the following areas: providing direct supervision during data collection; handling refusals; monitoring interviews and maintaining records of monitoring results; problem resolution; case review; specific project procedures and protocols; reviewing reports generated from the ongoing Computer Assisted Telephone Interviewing (CATI); and monitoring data collection progress.

Training for DCIs is designed to help staff become familiar with and practice using the CATI case management system and the survey instrument, as well as to learn project procedures and requirements. Particular attention will be paid to quality control initiatives, including refusal avoidance and methods to ensure that quality data are collected. DCIs will receive project-specific training on telephone interviewing and answering questions from web participants regarding the study or related to specific items within the interview. At the conclusion of training, all B&B call center staff must meet certification requirements by successfully completing a certification interview. This evaluation consists of a full-length interview with project staff observing and evaluating interviewers, as well as an oral evaluation of interviewers’ knowledge of the study’s Frequently Asked Questions.

3. Case Management System

Interviews will be conducted using a single web-based survey instrument for both web (including mobile devices) and CATI data collection. The data collection activities will be accomplished through a CATI case management system, which is equipped with the numerous capabilities, including: on-line access to locating information and histories of locating efforts for each case; a questionnaire administration module with full “front-end cleaning” capabilities (i.e., editing as information is obtained from respondents); sample management module for tracking case progress and status; and automated scheduling module which delivers cases to interviewers. The automated scheduling module incorporates the following features:

• Automatic delivery of appointment and call-back cases at specified times. This reduces the need for tracking appointments and helps ensure the interviewer is punctual. The scheduler automatically calculates the delivery time of the case in reference to the appropriate time zone.

• Sorting of non-appointment cases according to parameters and priorities set by project staff. For instance, priorities may be set to give first preference to cases within certain sub-samples or geographic areas; cases may be sorted to establish priorities between cases of differing status. Furthermore, the historic pattern of calling outcomes may be used to set priorities (e.g., cases with more than a certain number of unsuccessful attempts during a given time of day may be passed over until the next time period). These parameters ensure that cases are delivered to interviewers in a consistent manner according to specified project priorities.

• Restriction on allowable interviewers. Groups of cases (or individual cases) may be designated for delivery to specific interviewers or groups of interviewers. This feature is most commonly used in filtering refusal cases, locating problems, or foreign language cases to specific interviewers with specialized skills.

• Complete records of calls and tracking of all previous outcomes. The scheduler tracks all outcomes for each case, labeling each with type, date, and time. These are easily accessed by the interviewer upon entering the individual case, along with interviewer notes.

• Flagging of problem cases for supervisor action or supervisor review. For example, refusal cases may be routed to supervisors for decisions about whether and when a refusal letter should be mailed, or whether another interviewer should be assigned.

• Complete reporting capabilities. These include default reports on the aggregate status of cases and custom report generation capabilities.

The integration of these capabilities reduces the number of discrete stages required in data collection and data preparation activities and increases capabilities for immediate error reconciliation, which results in better data quality and reduced cost. Overall, the scheduler provides an efficient case assignment and delivery function by reducing supervisory and clerical time, improving execution on the part of interviewers and supervisors by automatically monitoring appointments and call-backs, and reducing variation in implementing survey priorities and objectives.

4. Survey Instrument Design

The B&B:16/20 panel maintenance address form, provided in appendix C, includes a postage paid, sealable postcard to note address updates. Sample members will first receive an emailed link to the secure address update form on the survey website. Sample members who update their information online before we mail postcards will not receive a postcard.

The interview will employ a web-based instrument and deployment system, which has been in use since NPSAS:08. The system provides multimode functionality that can be used for self-administration, including on mobile devices, CATI, Computer-Assisted Personal Interview (CAPI), or data entry.

In addition to the functional capabilities of the case management system and web instruments described above, our efforts to achieve the desired response rate will include using established procedures proven effective in other large-scale studies we have completed. These include:

• Providing multiple response modes, including mobile-friendly self-administered and interviewer-administered options.

• Offering incentives to encourage response (the incentive structure will be described in the winter 2019-20 request).

• Assigning experienced CATI interviewers who have proven their ability to contact and obtain cooperation from a high proportion of sample members.

• Training the interviewers thoroughly on study objectives, study population characteristics, and approaches that will help gain cooperation from sample members.

• Maintaining a high level of monitoring and direct supervision so that interviewers who are experiencing low cooperation rates are identified quickly and corrective action is taken.

• Making every reasonable effort to obtain an interview at the initial contact, but allowing respondent flexibility in scheduling appointments to be interviewed.

• Thoroughly reviewing all refusal cases and making special conversion efforts whenever feasible (see next section).

5. Refusal Aversion and Conversion

Recognizing and avoiding refusals is important to maximize the response rate. We will emphasize this and other topics related to obtaining cooperation during interviewer training. PTLs will monitor interviewers intensely during the early days of outbound calling and provide retraining as necessary. In addition, the supervisors will review daily interviewer production reports produced by the CATI system to identify and retrain any data collectors who are producing unacceptable numbers of refusals or other problems.

Refusal conversion efforts will be delayed for at least one week to give the respondent time after the initial refusal. Attempts at refusal conversion will not be made with individuals who become verbally aggressive or who threaten to take legal or other action. Refusal conversion efforts will not be conducted to a degree that would constitute harassment. We will respect a sample member’s right to decide not to participate and will not impinge this right by carrying conversion efforts beyond the bounds of propriety.

4. Tests of Procedures and Methods

The B&B:16/20 data collection design will build upon results from the B&B:16/17 data collection as well as results from prior B&B cohort data collections that employed similar strategies of applying different protocols that involved varied incentive amounts, contacting modes and methods, and interventions to groups of sample members based on their previous response behaviors. The full data collection plan for B&B:16/20 full-scale study will be provided in the winter 2019-20 request for clearance.

5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study

The study is being conducted by NCES. The following statisticians at NCES are responsible for the statistical aspects of the study: Mr. Ted Socha, Dr. Tracy Hunt-White, Dr. David Richards, Dr. Elise Christopher, and Dr. Gail Mulligan. NCES’s prime contractor for B&B:16/20 is RTI International (RTI). The following staff members at RTI are working on the statistical aspects of the study design: Dr. Jennifer Wine, Dr. Melissa Cominole, Ms. Jennifer Cooney, Dr. Nicole Tate, Ms. Erin Thomsen, Dr. Antje Kirchner, Dr. Erin Dunlop Velez, Dr. Emilia Peytcheva, and Mr. Peter Siegel.

Subcontractors include HR Directions; Research Support Services; EurekaFacts, LLC; KForce Government Solutions; Activate Research; and Strategic Communications, Inc. The consultants are Dr. Sandy Baum, Dr. Stephen Porter, and Dr. Paul D. Umbach. Principal professional RTI staff, not listed above, who are assigned to the study include Ms. Donna Anderson, Ms. Gayathri Bhat, and Ms. Ashley Wilson.

ii