Attachment III. ESI LC School Interview Protocol 3-22-2021

Red font indicates text that will be pre-programmed in the interview script by Abt. Purple font indicates instructions to the interviewer. Blue boxes are instructions or transitions that the interviewer will read to respondents. Radio buttons indicate only one answer should be selected. Checkboxes indicate the interviewer should check all that apply.

Table of Contents

I. Introduction 2

II. Timing of Loan Counseling Experiment 3

III. Additional Loan Counseling Approach 3

IV. Experiences Implementing the Experiment 6

V. ESI Data Reporting Tool 11

VI. Interactions with the Department of Education 14

VII. Transition from the F.A.C.T. to the ASLA 14

VIII. Wrapping up 15

IX. Closing Statement 16

1. Introduction

2. My name is [say your NAME]. My organization, Abt Associates, is contracted by the U.S. Department of Education’s Institute of Education Sciences to conduct a study of the Experimental Sites Initiative Loan Counseling Experiment. Thank you for making time to speak with me today – we are very interested in learning more about if and how the loan counseling experiment was carried out at your institution. The call should take no longer than 45 minutes.

3. We appreciate all the information you may have already provided to the Department’s Federal Student Aid (FSA) office. Prior to our call, I reviewed the ESI Reporting Tool and School Survey data that you shared with the Department of Education for the 2017-18 school year. I am following up to gather additional information.

4. It’s important that you know your responses today will be used only for research purposes, to describe how the Loan Counseling Experiment operated. None of our study reports will name you as an individual. The study information is being collected under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Personally identifiable information (PII) about individual respondents will not be released, and individual colleges will not be identified in published reports. However, data displayed by college for the Department of Education could be attributed to your institution or possibly an individual respondent. Your identifiable data will not be stored for reuse in future research.

5. If any questions are unclear or you would like additional clarity about a particular question, please let me know.

6. Do you have any questions before we begin?

7. All answers should be recorded verbatim. To avoid signaling to respondents that they should be doing a particular activity or following particular guidelines, do not use response categories as follow-up prompts. If respondents answer a question earlier in the interview, prior to its appearance on the protocol, record their answer when given and skip the question when you come to it on the protocol.

8. Timing of Loan Counseling Experiment

9. We want to understand more about if, and when, your institution participated the Loan Counseling Experiment over the past several years. To begin, I’d like to confirm the years your institution required additional loan counseling for some student borrowers.

1. During which of the following years did you require additional loan counseling for some students?

	2017-18	2018-19	2019-20	2020-21
Check if yes				

2. [If “no” for a given year] Why did you not require additional loan counseling in [YEAR]?

	2017-18	2018-19	2019-20	2020-21
Difficulty obtaining the required additional loan counseling product				
Difficulty developing the counseling curriculum/materials				
Difficulty integrating the counseling into existing school technology				
Difficulty identifying borrowers eligible for additional loan counseling				
Difficulty implementing because of COVID-19 pandemic				
Upcoming change to ASLA instead of FACT				
Change in leadership				
Reduction in financial aid/counseling staff				
Other. Specify:				

10. Additional Loan Counseling Approach

11. Next, I have some questions about how you selected the required additional loan counseling approach for your institution.

3. We want to be sure that we know who developed the required additional loan counseling approach your institution used during each year of the experiment. Who was the developer of the required additional loan counseling approach you used in each year?

	2017-18	2018-19	2019-20	2020-21
U.S. Department of Education Financial Awareness Counseling Tool (F.A.C.T.)				
iGrad				
Other commercial product. Specify other product:				
Your institution				
ASLA				
Other. Specify:				

4. [If changed] Why did your institution change the approach used?

• Students provided negative feedback on the content. Specify:

• Students provided negative feedback about the amount of time it took to complete the required additional loan counseling

• Students provided negative feedback about the in-person counseling. Specify:

• Students provided negative feedback about the online counseling. Specify:

• Required too much staff time to deliver the required additional loan counseling

• Too expensive

• Staff wanted to cover different content

• Staff wanted loan counseling that could be tailored to students’ individual needs

• Other. Specify:

5. [If never used the F.A.C.T.] Were you aware that the U.S. Department of Education offers a free online loan counseling tool, called the Financial Awareness Counseling Tool (or FACT)?

• Yes

• No

6. [If the college ever developed their own approach to required additional loan counseling] What was the most important reason that your institution developed its own required additional loan counseling?

___________________________________________________________________________________________________________________________________________________________ ______

7. How important were each of the following in your selection of the required additional loan counseling approach? Please answer “not at all important,” “somewhat important,” “very important,” “don’t know,” or “not applicable.” How important was the… Read each potential reason.

	Not at all important	Somewhat important	Very important	Don’t know	Not applicable
Free or low purchase price					
Easy to integrate counseling into existing school technology					
Familiarity, because used same approach for entrance counseling					
Ability to tailor topics to students’ specific needs					
Ability to cover topics not included in iGrad or other commercial products					
Ability to cover topics not included in the F.A.C.T.					
Other, please specify: ____________

8. [If “ability to cover topics not included in the F.A.C.T.” is somewhat or very important] What topics did you include in the required additional loan counseling that are not part of the F.A.C.T.?

____________________________________________________________________________________________________________________________________________________________________

If the institution always used an off-shelf-approach (F.A.C.T., ALSA, iGrad, other commercial product, or any combination of these approaches), GO TO Q.13.

9. [If the institution ever developed their own approach] Was the required additional loan counseling developed by your institution offered in-person, online/computer-based, or a combination?

Clarify what is meant by in-person versus online/computer based and select the appropriate response category below (for example, if a student must go to a classroom to complete the required additional loan counseling but the content is delivered through software then select online/computer based).

• In-person.

• Online/computer-based counseling.

• A combination of in-person and online /computer-based counseling.

1. [If the required additional loan counseling is offered in-person] How was the additional loan counseling delivered?

   • One-on-one

   • Small group

   • A combination of One-on-One and Small Group

   • Other, specify

2. [If the required additional loan counseling is offered in-person] Who delivered the additional required loan counseling?

   • Peer-to-peer

   • Financial aid staff

   • College advisor

   • Faculty from a specific department

   • Other, specify

10. What content was included in the required additional loan counseling that was offered [mode of delivery…]? Ask for each mode of delivery identified in response to question 9.

	[If in-person or a combination of in-person and online/ computer-based counseling] a. In-person	[If online or a combination of in-person and online/ computer-based counseling] b. Online/ Computer-based counseling	[If a combination of in-person and online/ computer-based counseling] c. Both in-person and online/ computer-based counseling
Basic information about student loans			
General borrowing recommendations			
Types of federal student loans			
Loan limits			
Student loan amount			
Current loan balance			
Projected balance at completion			
Budget for remaining years of education			
Starting salaries for occupation of interest			
Repayment options			
General financial literacy (e.g., compounding interest)			
Other. Specify:			

11. [If in-person or a combination of in-person and online delivery] What content or topic was most important to offer to students in-person?

____________________________________________________________________________________________________________________________________________________________________

(If the institution developed the approach to required additional loan counseling was always delivered online, GO TO Q. 13.)

12. [If the college ever developed their own approach to required additional loan counseling and delivered it in-person] What was the most important reason your institution decided to deliver the required additional loan counseling in-person?

____________________________________________________________________________________________________________________________________________________________________

13. We would like to understand how the required additional loan counseling differs from entrance counseling. First, what does your institution use for entrance counseling?

______________________________________________________________________________

14. Now, how does the required additional loan counseling differ from entrance counseling?

• Same as entrance counseling

• More tailored to individual student

• Mode of delivery, specify:

• Different topics covered

• Other, specify:

1. [If different topics covered] What topics are covered as part of the required additional loan counseling that are not covered in entrance counseling?

• Basic information about student loans

• General borrowing recommendations

• Types of federal student loans

• Loan limits

• Student loan amount

• Current loan balance

• Projected balance at completion

• Budget for remaining years of education

• Starting salaries for occupation of interest

• Repayment options

• General financial literacy (e.g., compounding interest)

• Other. Specify:

13. Experiences Implementing the Experiment

Now I want to ask you about your experiences implementing the experiment.

Identifying Eligible Students

15. When did your institution identify eligible students for the required additional loan counseling?

• Before financial award notifications are prepared

• After financial aid award notifications are sent

• After the student loan is disbursed

• On a rolling basis throughout the academic year

• Other. Specify:

16. What students were eligible for additional loan counseling at your institution?

• Undergraduate students

• Students who completed entrance counseling already

• Student awarded a federal student loan for the year

• Student enrolled at least half time

• Other. Specify:

17. Were all students who met the eligibility criteria included when determining who would be required to have additional loan counseling and who would not be?

• Yes

• No

18. [If NO] Which eligible students were not included, and why were they not?

____________________________________________________________________________________________________________________________________________________________________

19. What challenges did your institution encounter identifying eligible students?

    • Identifying the appropriate students in the data system

[If noted] What made it challenging to identify the right students?

______________________________________________________________________________________________________________________________________________________

• Identifying students who had already completed entrance counseling

• Identifying students who also met other eligibility criteria (undergraduate student, enrolled at least half time) in addition to being awarded a federal student loan

• Identifying eligible students in later years, after the first year of the experiment (once some students were already receiving additional loan counseling)

• Determining which transfer students were eligible

• The timing of when eligible students are identified and the availability of data

• Other. Specify:

• Did not encounter any challenges

20. [If did not encounter any challenges is not selected] What was the biggest challenge your institution encountered when identifying eligible students?

______________________________________________________________________________

Determining which Students Were Required to Complete Additional Loan Counseling and which Students Were Not

21. Could you describe the process you used to determine the students who would be required to complete additional loan counseling and the students who would not?

    • Half of eligible students to get additional loan counseling, half not

    • Used last digit of SSN to determine who gets additional loan counseling or not

    • Even for additional loan counseling, odd not

    • Odd for additional loan counseling, even not

Other. Specify: _____________________________________________________________________________________________________________ _______________________________________________

22. What about in the years that followed?

    • Used the same process to determine which students would be required to complete additional loan counseling as the first year

    • Kept students in the same condition each year

    • Implemented with a new group of students each year

    • Other. Specify:

_____________________________________________________________________________________________________________ ______________________________________________________

23. How did the process of determining which students would be required to complete additional loan counseling differ for transfer students?

____________________________________________________________________________________________________________________________________________________________________

• The process was the same for transfer students

24. [If used SSN] What challenges did your institution encounter determining the students who were required to complete the addition loan counseling based on social security numbers?

    • Linking SSNs to eligible borrowers across different data systems

    • Unable to automate the process of determining which students were required to complete additional loan counseling and which were not using SSN

    • Needed to change who was required to receive additional loan counseling after identifying errors.

1. [If yes] Did corrections occur before students were required to complete additional loan counseling or after the wrong students were required to complete additional loan counseling?

• Before any students were incorrectly required to complete additional loan counseling

• After the wrong students were required to complete additional loan counseling

2. Were corrections made in the same year as the error or a later year?

• Same year

• Later year

• Other. Specify:

• We did not have any challenges

25. [If used SSN] What was the biggest challenge for your institution in using social security numbers to determine which students were required to complete additional loan counseling and which students were not?

______________________________________________________________________________

26. Were you provided guidance by FSA about how to determine which students would be required to complete additional loan counseling and which students would not?

• Yes

• No

Notifying Students about the Additional Loan Counseling Requirement

27. At what point in the process of obtaining a loan did your institution inform students about the additional loan counseling requirement?

    • Before the financial aid award notification

    • At the same time as the financial aid award notification

    • After the financial aid award notification

    • After the student loan is disbursed

    • Other. Specify:

Monitoring Students’ Completion of Additional Loan Counseling

28. When did students have to complete their required additional loan counseling?

• Before their student loan was disbursed

• Before tuition payments were due

• During the same semester/trimester/quarter that the loan was disbursed

• During the same award year that the loan was disbursed

• Other. Specify:

29. How are you able to tell if students had completed the required additional loan counseling?

    • Online system tracks students’ completion of additional loan counseling

    • Modified an existing system to track completion

    • Maintain a list of students who complete additional loan counseling, by adding names as each student completes counseling (usually when counseling delivered in-person)

    • Students submit documentation of completion

    • Other. Specify:

30. If students did not complete the required loan counseling before tuition payments were due, was there any follow-up to encourage them to complete it?

    • No follow-up

    • Contacted student via email or text

    • Other. Specify:

31. In the first year the additional loan counseling was required, what portion of the students required to complete additional loan counseling who received direct loans actually did so? Read each response option.

• All (100%)

• Nearly all (about 80 – 95%)

• Most (about 56 – 79%)

• About half (about 45 – 55%)

• Less than half (about 15 – 44%)

• A few (1 – 14%)

• None (0%)

32. Was this about the same each year, higher in later years, lower in later years? Read each response option.

• Higher portion (of students required to complete additional loan counseling actually did so) each year

• About the same portion (of students required to complete additional loan counseling did so) each year

• Lower portion (of students required to complete additional loan counseling actually did so) each year

• Other. Specify:

33. In the first year the additional loan counseling was required, what portion of the students who were not required to complete the required additional loan counseling ended up completing it? Read each response option.

• None (0%)

• A few (1 – 14%)

• Less than half (15 – 44%)

• About half (45 – 55%)

• Most (56 – 79%)

• Nearly all (80 – 95%)

• All (100%)

34. Was this about the same each year, lower in later years, higher in later years? Read each response option.

• Lower portion (of students NOT required to complete additional loan counseling actually had the additional counseling) each year

• About the same portion (of students NOT required to additional loan counseling actually had it) each year

• Higher portion (of students NOT required to complete additional loan counseling actually had it) each year

• Other. Specify:

35. When it came to tracking students across multiple years, how challenging were the following? Please answer “not at all challenging,” “a little challenging,” “moderately challenging,” “very challenging,” “don’t know,” or “not applicable.” How challenging was… Read each potential challenge.

	Not at all challenging	A little challenging	Moderately challenging	Very challenging	Don’t know	Not applicable
Keeping track of students who are eligible across multiple years
Keeping track of students who are awarded direct loans across multiple years
Keeping track of students who are required to take the additional loan counseling across multiple years
Making sure students who were required to complete additional loan counseling continued to do it in each subsequent year
Making sure students who were NOT required to complete additional loan counseling did not complete it in any subsequent year
Other. Specify/;

36. What was the biggest challenge for your institution in tracking students over time and making sure they completed the required additional loan counseling, if supposed to, and did not, if not supposed to?

______________________________________________________________________________

______________________________________________________________________________

14. ESI Data Reporting Tool

15. In this next section I’d like to ask about reporting data on students in the experiment using the ESI Data Reporting Tool.

37. When reporting data using the ESI Reporting Tool, what students were included in the required additional loan counseling group (“treatment group”)? Mark in the table below under “Treatment Group.”

	Treatment Group	Comparison Group
Based on federal loan borrowing
Students awarded a federal student loan for the current year		
Students awarded a federal student loan in a prior year (regardless of whether awarded a loan in the current year)		
Students who were first-time federal loan borrowers		
Students who had already completed entrance counseling		
Students who had not completed entrance counseling		
Based on progress in school
Students who were not first-time freshmen		
Students who were first-time freshmen		
Undergraduate students		
Students enrolled in any degree program, including graduate programs		
Based on enrollment status
Students enrolled at least half time		
Students enrolled less than half time		
Other. Specify:		

38. Did the students included in the required additional loan counseling group stay the same or change in later years?

• Stayed the same each year

• Changed in later years. What changed?

______________________________________________________________________________

______________________________________________________________________________

39. When reporting data using the ESI Reporting Tool, what students were included who were not required to complete additional loan counseling (“comparison group”)? Mark in the table above under “Comparison Group.”

40. Did the students included who were not required to complete additional loan counseling stay the same or change in later years?

• Stayed the same each year

• Changed in later years. What were the changes?

______________________________________________________________________________

______________________________________________________________________________

41. [For institutions that reported students earning more than 40 credits per year] I’d like to understand more about the information institutions reported about credits earned. Across institutions, the reported credits earned in one school year ranged from 0 to 180 credits [and Z to ZZ in your college]. Given that a full-time course load is approximately 30-32 credits per year, what might explain the larger number of credits?

____________________________________________________________________________________________________________________________________________________________________

42. For credits attempted and credits earned, did your institution report the number of credits for all students, clock hours for all students, or credits for some students and clock hours for some students?

• Credits for all students

• Clock hours for all students

• Credits for some students and clock hours for some students

43. [If both credits and clock hours]

    1. How did you decide whether to report credits or clock hours?

       • Based on student’s major/program of study

       • Other. Specify:

    2. How is it possible to tell if credits or clock hours were reported for an individual student?

       • You can tell by the reported value.

         • Possible or plausible range for clock hours is________________.

         • Possible or plausible range for credits is____________________.

       • Other. Specify:

       • Not possible to tell the difference

44. [For institutions that reported that cumulative credits increased more than the number of credits earned during the year for any student] Institutions sometimes reported that students’ cumulative credits earned over the year increased more than the number of credits earned in the same year.

    1. Given that we would expect these numbers to be the same, what might explain the difference?

       • Data come from two different systems

       • Transfer credits from another institution

       • Students concurrently enrolled at multiple institutions

       • Other. Specify:

    2. When increases in cumulative credits differ from the number of credits earned during the year, which is the most accurate?

       • Increases in cumulative credits

       • Number of credits earned

45. [For institutions that reported GPAs higher than 4.0] Some institutions reported grade point averages (GPA) above 4.0 and ranging as high as 8.2. Given that GPA is generally measured on a 4-point scale, what might explain the values for GPA over 4.0?

____________________________________________________________________________________________________________________________________________________________________

46. [For institutions that reported any students awarded $0 in federal student aid] Institutions reported that nearly half of the students participating in the Loan Counseling Experiment were awarded $0 in total federal student aid (excluding Federal Work Study). We expected participating students to have been awarded federal student aid. What are the reasons why students included in the ESI Data Reporting Tool were awarded $0 in total federal student aid?

    • Students were identified based on prior year borrowing (before they were awarded a loan for the current year)

    • Data on federal student aid was not reported using the Reporting Tool, because it is reported to the Department of Education using a different data system

    • Other. Specify:

47. [For institutions that reported any students who received a federal loan disbursement that was larger than their award] Across institutions, about 40% of students received a federal loan disbursement that was larger than their award. Given that we expect the amount disbursed to be the same or lower than the award, what might explain the amount disbursed being larger than the award?

____________________________________________________________________________________________________________________________________________________________________

48. [For institutions that reported any students earning Federal Work Study] Across institutions, the average amount of Federal Work Study awarded to students who received an award was $XXX [and $ZZZ in your college]. Given that Federal Work Study awards are $YYY on average, what might explain the low amounts?

____________________________________________________________________________________________________________________________________________________________________

49. Institutions rarely reported information about whether students completed their degree program, although we expect that some students did complete their program. What might have made it difficult to report information about whether students completed their program?

____________________________________________________________________________________________________________________________________________________________________

50. Institutions rarely reported information about whether students completed voluntary loan counseling, although we expect that some students did complete voluntary loan counseling. What might have made it difficult to report information about whether students completed voluntary loan counseling?

____________________________________________________________________________________________________________________________________________________________________

51. How challenging were the following when using the ESI Reporting Tool? Please answer “not at all challenging,” “a little challenging,” “moderately challenging,” “very challenging,” “don’t know,” or “not applicable.” How challenging was… Read each potential challenge.

	Not at all challenging	A little challenging	Moderately challenging	Very challenging	Don’t know	Not applicable
Understanding questions / data fields
Understanding instructions
Asking ED questions
Having enough time to complete
Access required information from institution data system(s)
Other. Specify:

17. Interactions with the Department of Education

18. Next, I am going to ask you about your interactions with the Department of Education.

52. What type of communication did you have with the Department of Education about the Loan Counseling Experiment?

    • Webinar(s)

    • Reference materials

    • Training or guidance for completing the School Survey

    • Training to use the ESI Reporting Tool

    • Contacted the department with questions

    • Received responses to our questions from the Department

    • Other. Specify:

53. On what topics would more guidance have been helpful?

    • Identifying student loan borrowers who were eligible for additional loan counseling

    • Determining which students should be required complete additional loan counseling using their social security number

    • Available approaches to additional loan counseling

    • Completing the ESI reporting tool

    • Other. Specify:

20. Transition from the F.A.C.T. to the ASLA

21. As you have likely heard, the Department of Education will be retiring the Financial Awareness Counseling Tool (F.A.C.T.) and implementing the Annual Student Loan Acknowledgement (ASLA). In this section I’m going to ask about how your institution will transition from the FACT to the ASLA.

54. What will you use for entrance and exit counseling once the F.A.C.T. is retired?

    • We will use ASLA only.

    • We will use ASLA and continue to use the loan counseling approach used for the experiment.

    • We will use ASLA combined with another loan counseling approach (not used in the experiment).

    • Other. Specify:

55. What are the pros and cons of ASLA compared to the F.A.C.T. in terms providing information to students that is likely to help them make financially sound borrowing decisions?

	ASLA	F.A.C.T.
PROS
Tailored to first time borrowers versus current loan borrowers		
Presents general summary information about average costs and borrowing at our school (total estimated student loan debt, estimated monthly payments, and graduation rates)		
Provides general financial literacy information about borrowing student loans		
Presents details on individual student’s borrowing (loan history, outstanding balance, estimated monthly payment based on current balance, and loan limit information)		
Duration of counseling		
Frequency of when it is offered		
Other. Specify:		
CONS
Does not differentiate between first-time and current loan borrowers		
Does not provide summary information about average costs and borrowing at our school		
Does not provide general financial literacy information about borrowing student loans		
Does not provide details on individual student’s borrowing		
Duration of counseling		
Frequency of when it is offered		
Other. Specify:		

23. Wrapping up

24. Finally, I’d like to wrap up our interview by asking you about what you took away from the experience of providing additional loan counseling to students through your institution’s participation in the Loan Counseling Experiment.

56. Is there any additional information that you would like to share about your experience with the Loan Counseling Experiment?

____________________________________________________________________________________________________________________________________________________________________

26. Closing Statement

27. Thank you so much for providing all this information. We really enjoyed learning more about your college’s additional student loan counseling and appreciate your time!

28. The information you provided today will help the Department of Education better understand your needs and students’ needs for student loan counseling.

30. Thank you again!

20