Volume I

Experimental Sites Initiative

Loan Counseling Experiment

Cognitive and Usability Interviews

OMB# 1850-0952 v.1

Volume I

Supporting Statement

Submitted by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

March 2021

Attachments

Attachment I. Loan Counseling Experiment School Survey

Attachment II. Experimental Sites Initiative (ESI) Data Reporting Tool

Attachment III. Loan Counseling Experiment College Interview Protocol

Table of Contents

1) Submittal-Related Information 3

2) Background 3

3) Design and Context 4

4) Recruitment and Data Collection 4

5) Estimated Respondent Burden 5

6) Estimate of Costs for Recruiting and Paying Respondents 5

7) Cost to federal government 5

8) Assurance of Confidentiality 5

9) Justification for Sensitive Questions 6

10) Project Schedule 6

1. Submittal-Related Information

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct cognitive and usability interviews with colleges participating in the Loan Counseling Experiment to identify ambiguity, misunderstanding, or other difficulties respondents have with two annual data collections being conducted by the Federal Student Aid (FSA) Office. Interviews will begin in April 2021.

2. Background

The Loan Counseling Experiment was started by the U.S. Department of Education under its Experimental Sites Initiative (ESI) in response to a presidential directive to help federal student loan borrowers manage their debt.¹ Through ESI, participating colleges can require undergraduate students to complete additional loan counseling as a condition of receiving a federal loan – beyond the currently-mandated entrance counseling when students first take out a loan and exit counseling when they leave or finish college.

The Department’s Institute of Education Sciences (IES) is evaluating the experiment, by testing the effect of additional loan counseling on students’ borrowing, school progress, and later repayment behaviors in 35 colleges. All students in the participating colleges who previously completed the mandatory entrance counseling are eligible to participate in the study. Eligible students are assigned using a lottery to either complete additional required loan counseling once per academic year, beyond the mandatory entrance and exit counseling, or not complete any additional loan counseling. The study tracks borrowing and progress through college among students in both the treatment and control group over time using college surveys and administrative records, comparing the two groups on these measures to determine the effectiveness of additional loan counseling. In the fall of 2017, colleges identified the first cohort of eligible students and assigned them to receive additional loan counseling or not. Those assigned to additional counseling receive additional counseling each fall, beginning in 2017 and continuing until they leave the college. Borrowing, persistence, credit accumulation, and attainment outcomes were first measured in the fall of 2018, one year after the first cohort of students entered the experiment.

This request is to conduct cognitive and usability interviews with college financial aid administrators at the 35 colleges participating in the Loan Counseling Experiment to identify ambiguity, misunderstanding, or other difficulties respondents have answering questions on two annual surveys. First, participating colleges are asked to describe the implementation of the required additional loan counseling on the Loan Counseling Experiment (LCE) School Survey (See Attachment I). Second, colleges participating in any ESI experiment, including the Loan Counseling Experiment, are asked to use the ESI Data Reporting Tool to identify individual students in the experiment and include information about their assigned group, credits earned, grade point average (GPA), program completion, and federal and non-federal student aid awarded and disbursed (See Attachment II).

Items on the LCE School Survey to be evaluated include questions about the loan counseling approach selected by colleges, topics covered, and challenges that colleges encountered while administering the experiment, particularly related to identifying eligible students, assigning them to receive additional loan counseling or not, and notifying them about the additional loan counseling requirement. In addition, we will ask probing questions about how respondents came up with answers on the ESI Data Reporting Tool to better understand if ineligible students were mistakenly included as participants, why most colleges reported students earning an unexpectedly high number of credits, why discrepancies in information reported about credits earned were common, what scales were used to report grade point average, and why colleges did not report information about program completion. The LCE College Interview Protocol is presented in Attachment III.

The cognitive and usability interviews will be conducted in April 2021 via videoconference with up to two college financial aid administrators from each of the 35 participating colleges. That is, the person or persons responsible for completing the LCE School Survey and the ESI Data Reporting Tool will participate in a single interview lasting no more than 45 minutes.

3. Design and Context

The cognitive and usability interviews for which this clearance is requested will be used to maximize the quality of data collected, clarify misunderstanding and difficulties completing instruments, guide potential modifications to the ESI Data Reporting Tool, and inform school surveys developed for future ESI experiments. Preliminary analyses of both instruments have identified data quality issues and questions that could elicit information more effectively. Responses on the ESI Data Reporting Tool suggest that many colleges report data for the wrong group of students and report implausible and inconsistent data on credits earned and GPA. In addition, the LCE School Survey did not elicit clear information about the selected loan counseling approach and the challenges encountered while administering the experiment, particularly related to identifying and randomly assigning eligible students, and notifying them about the additional loan counseling requirement.

Specifically, cognitive and usability interviews will focus on:

• Determining the extent to which questions elicit accurate information,

• Evaluating the clarity of questions and common misinterpretations,

• Determining appropriate response categories, and

• Examining the thought process used to arrive at answers.

The cognitive and usability interviews will be an intensive, in-depth discussion with financial aid administrators at each participating college. The interviews are intended to maximize the quality of data collected by asking probing questions to better understand respondents’ answers to the LCE School Survey and to determine sources of misunderstanding in responses to the ESI Data Reporting Tool.

In the interviews, respondents will be asked to elaborate on questions originally asked in the two instruments. The interview protocol will have sections covering the following topics:

1. Selection of the additional loan counseling approach, including factors influencing the selection and the content covered.

2. Experiences implementing the experiment, including identifying eligible students, conducting random assignment, notifying students about the loan counseling requirement, and monitoring students’ completion of additional loan counseling.

3. Communication with and assistance from the U.S. Department of Education.

4. Issues and challenges reporting data on students using the ESI Data Reporting Tool.

5. Transition from the Financial Awareness Counseling Tool to the Annual Student Loan Acknowledgement

4. Recruitment and Data Collection

Abt Associates has extensive experience interviewing college financial aid administrators and will be responsible for contacting participants, obtaining their agreement to be interviewed, conducting the interviews, compiling interview responses, and summarizing findings. Interviewers will ask questions in an open-ended manner and encourage participants to elaborate about their understanding and how they formulate their responses. Some questions will be closely aligned with items from the original instruments, and other will probe more generally to elicit respondents’ understanding. Interviewers will code responses into pre-specified categories to the extent possible, while also documenting responses that do not fit into the pre-specified categories.

Abt Associates will obtain the main point of contact (or contacts) already on record for the Loan Counseling Experiment at each of the 35 participating colleges. The Department of Education will provide names and contact information to Abt Associates. The Abt study team will email the contact person at each college and provide information about the interviews, their purpose, and the time commitment. Abt will work with the contact person to identify the correct financial aid administrators to participate in the interview – that is, the person or persons responsible for completing the LCE School Survey and the ESI Data Reporting Tool. Once the correct financial aid administrators are identified, interviews will be scheduled. The interviews are planned to include two respondents from each college.

The interviews will take place over WebEx, a video conference platform, and may be audio and/or video recorded so that researchers can reference recordings to supplement written notes taken during the interview.

5. Estimated Respondent Burden

Tables 1 details the respondent burden for the cognitive and usability interview.

The estimates assume that it will take 15 minutes to screen, recruit, and schedule interviews, that interviews will take 45 minutes, and that 2 administrators will be recruited per college for 35 colleges. The estimates do not consider non-response from colleges. If college staff refuse to participate in the study, then the total hours in Table 1 will be an overestimation.

Table 1. Estimated Participant Burden

Activity	Number of Colleges	Number of Financial Aid Administrators	Minutes per respondent	Total burden hours
Screening, Recruitment, & Scheduling	35	70	15	18
Cognitive and usability interview	35	70	45	53
Study Total	35	70	60	70

Note: Numbers have been rounded to whole numbers.

6. Estimate of Costs for Recruiting and Paying Respondents

No incentives will be provided to interviewees.

7. Cost to federal government

The total cost to federal government for conducting the cognitive and usability interviews will be $108,300. This cost includes recruitment, conducting the interviews, analysis and reporting conducted by Abt Associates.

8. Assurance of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Respondents will be informed that their participation is voluntary and that all the information they provide will be used only for statistical purposes and will not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). At the start of the interview, this information will be conveyed verbally to respondents by the interviewer.

All interviewees will be assigned a unique respondent (ID), created solely for data file management and used to keep all participant materials together. A crosswalk file that connects respondent ID to the interviewee’s name and associated college will be stored on a secure data server and only used for tracking the scheduling and completion of the interviews. All comments made in the interviews will be de-identified and aggregated in any published reports. As a result, none of the interviewees or their associated colleges could be connected to any statement or opinion described in the final report. The recorded files and crosswalk will be secured for the duration of the study and will be destroyed after the final report is completed.

9. Justification for Sensitive Questions

The cognitive and usability interview protocol does not include questions that might be considered sensitive.

10. Project Schedule

Recruiting for the cognitive and usability interviews will begin in April 2021, as soon as OMB approval is granted, and continue through June 2021.

1 Through ESI, authorized under title IV of the Higher Education Act of 1965 (HEA), the Secretary has the authority to grant waivers of certain regulatory requirements to allow a limited number of institutions to test alternative methods of implementing title IV programs.