Data Collection for PEP Participant Baseline Report

Windwalker Corporation

OMB Clearance Request

for Contract ED-FSA-12-R-0012

Deliverable 2.3C

Prepared by:

Windwalker Corporation

1945 Old Gallows Road

Tysons Corner, VA 22182

Submitted to:

Christopher T. Lemmie, Director

& Project Manager

FSA Business Practices

[email protected]

202-377-3225

January 21, 2014

Table of Contents

Table of Contents i

Clearance Documentation 1

Documentation for the Generic Clearance of Customer Service Satisfaction Collections 1

Financial Aid Staff Group Protocol 6

Institutional Leader Individual Protocol 10

Clearance Documentation

Documentation for the Generic Clearance of Customer Service Satisfaction Collections

TITLE OF INFORMATION COLLECTION: Data Collection for PEP Participant Baseline Report

[] SURVEY [X ] FOCUS GROUP [ ] SOFTWARE USABILITY TESTING

DESCRIPTION OF THIS SPECIFIC COLLECTION

BACKGROUND

Under Title IV of the Higher Education Act of 1965, which covers the administration of U.S. federal student financial aid programs, Federal Student Aid (FSA) administers the following programs: Pell grants, Stafford loans, PLUS loans, and the “campus-based” programs including Federal Work Study, Perkins Loans, and Federal Supplemental Educational Opportunity Grants. Federal regulation requires schools to have written policies and procedures for the administration of the Title IV student assistance programs. To ensure institutional regulatory compliance, FSA provides training and technical assistance for financial aid administrators, institutional leaders, and other institutional support staff nationwide via its Information for Financial Aid Professionals (IFAP) website and the School Experience Group (SEG). The SEG’s mission is to identify and the unique service needs of all post-secondary education institutions and provide them with tailored resources to meet those needs. Within SEG, the Minority Serving and Under Resourced Schools Division (MSURSD) is responsible for providing support, assessment, and training specifically targeted for Minority Serving Institutions (MSIs).

MSURSD is currently working to enhance the services and resources they provide to the MSIs, with the dual goals of increasing MSI compliance rates and of assisting MSIs in improving student performance outcomes such as increased student retention and graduation rates. Within the past month, Windwalker, who was contracted to review MSURSD’s operations and processes, provided MSURSD leadership with recommendations on how to improve the effectiveness and efficiency of their division’s operations and processes in the form of a Performance Enhancement Pilot (PEP) program. The PEP recommendations stem from insights garnered from the operational review and business process mapping report that Windwalker finalized for MSURSD in late 2013. MSURSD leadership is currently interested in gaining baseline performance assessment information from a sample of the MSIs before many of the PEP recommendations are implemented to measure the impact of the organizational changes made as a result of the PEP. Specifically for the current work, MSURSD is interested in gaining insight into the following:

• Identifying the greatest gaps between MSI leaders and financial aid staff in understanding compliance-related requirements;

• Identifying the greatest gaps between MSI leaders and financial aid staff in implementing aid-related processes; and

• Discovering ways to enhance the financial aid experience for students receiving aid at MSIs.

INTENDED PURPOSE AND NEED FOR THE COLLECTION

As was mentioned in the previous section, MSURSD has contracted Windwalker Corporation to identify areas where the division can increase the efficiency and effectiveness of their operations and processes. Windwalker has already conducted an extensive operational review and has completed detailed maps of MSURSD’s major processes. A Performance Enhancement Pilot (PEP) program was created by Windwalker, identifying the strategies we recommend MSURSD employ to enhance their operations and processes. MSURSD will be testing several of these recommendations by piloting them at 12 MSIs that they have selected to participate in this research phase. The selected schools will be representative of the full MSI population in terms of cluster affiliation (e.g. HBCU, HSI, etc.) and risk level (as determined by MSURSD).

This current phase of the research includes conducting root cause interviews with institutional leaders and financial aid staff from 12 MSIs selected by MSURSD to participate in the PEP. The interviews will provide MSURSD with information to identify where gaps in compliance-related knowledge and process implementation exist to assess the extent to which the recommendations in the PEP decrease gaps.

The two protocols contain between 8 and 15 items to gather the desired information. The goal of collecting data from two different audience groups is to compare the feedback collected from each group to identify gaps in perception that MSURSD can address through their operational and processes changes. Specifically, the interviews will ask respondents questions on the following themes:

• identification of stakeholders involved with compliance-related issues;

• ways in which stakeholders gain information about compliance-related issues, and the utility of each;

• identification of aid-related processes and how each process is developed;

• identification of resources provided to students to facilitate receipt of aid; and

• identification of successes and challenges when developing and implementing aid-related processes.

The onsite qualitative data collection is a component of a larger evaluation effort to guide SEG’s program development and enhancement of services available for MSIs. All interview protocols are provided as appendices in this document.

COLLECTION PROCEDURES

Each onsite data collection visit will entail a single Windwalker researcher experienced in qualitative data collection spending one business day at the selected institution. Two sets of interviews/focus groups will be conducted during each site visit – an interview with institutional leaders and a focus group with financial aid staff. The interviews will be entirely qualitative in nature and will be administered by one of several Windwalker researchers experienced in administering the interview/focus group protocols. It is expected that one institutional leader and up to three financial aid staff will participate in data collection efforts at each institution. Each interview and focus group session will take approximately one hour to complete and will begin with the researcher explaining the purpose of the research and how the results will be used. The researcher will also ask if the interviewees agree to have the conversations recorded using a tape-recorder provided by the researcher. It will be explained to participants that the recording is for data analysis purposes only and will only be accessible to the research team. However, the researcher will not record the session should any of the participants decline to have their comments recorded.

The interviews will include open-ended questions to identify gaps in compliance-related knowledge, as well as to identify areas where processes are successfully being implemented or where they need to be improved. The interviews and focus groups will last approximately 1 hour and will contain between 8 and 15 questions, depending upon which group is participating. Protocols will include both the interview questions and a series of prompts to act as guides for researchers to gather data from respondents who are having difficulty answering the question as intended.

MSURSD will provide the list of MSIs that they want to participate in the PEP pilot to Windwalker. This list will contain the contact information for each designated point of contact with whom Windwalker should coordinate the visits, as well as the contact information for institutional leaders and the financial aid staff who will participate in the interviews and focus groups. Once the list is provided to Windwalker, the Windwalker task lead will contact the point of contact to coordinate a time when the Windwalker researcher can come to the institution to conduct the interviews. Windwalker will work with the point of contact at each institution to schedule the individual interviews and focus groups with the relevant participants.

Upon completing all data collection activities at each institution, the Windwalker researcher will review his or her notes and listen to the recorded session, as available. At this time, the researcher will synthesize all of the themes that emerged from the interviews and focus groups and identify areas where the findings were either corroborated or conflicted across the sessions to note gaps in perception across the respondent groups. These qualitative findings will be added to each school’s reported quantitative data, which will look at each individual school’s performance across a variety of outcome performance measures, as detailed in deliverable 2.3B of the above-mentioned contract. Greater detail on how these results will be used is provided in the “Planned Use of Data” section below.

DATES, LOCATIONS, AND PARTICIPANTS

The proposed timeframe for site visits and data collection is approximately February 3 – March 14, 2014 to accommodate the delivery of the draft baseline reports to MSURSD leadership by April 1. One Windwalker researcher will conduct the interviews for a given site over the course of one business day. No payments, stipends, or incentives are proposed. Windwalker will conduct two separate interviews and/or focus groups: an interview with institutional leaders and a focus group of institutional financial aid staff from the institution who either currently has or in the past had received financial aid. In total, a population of 48 leaders and financial aid staff will participate in the data collection efforts across the 12 schools participating in the PEP. This timeline meets the contractual requirement that the interviews be conducted during the spring 2014 school semester and after OMB clearance is obtained. The interview/focus group instruments have also been shared with MSURSD leadership for approval, and final approval from them has already been granted.

PLANNED USE OF DATA

As was mentioned in the “Collection Procedures” section, the data collected from these efforts will be used to identify gaps in compliance-related knowledge as well as to identify areas where processes are successfully being implemented or need to be improved. As was also mentioned earlier, these findings will be presented in an institution-specific baseline report that couples the findings from the onsite data collection efforts with quantitative benchmark data. The data will assess each institution’s performance across a variety of quantitative outcome metrics, including past compliance, student retention and default rate, which will be gathered through the analysis of secondary data collected by the US federal government, such as from IPEDS. The first drafts of the baseline report for each PEP-participating institution is due to MSURSD leadership by April 1, 2014. The goal of this report is to provide a snapshot of each institution’s performance before many of the PEP initiatives identified in deliverable 2.3a have been implemented to act as a baseline barometer of institutional performance. Quarterly follow-up phone-based conversations will be conducted with each of the institutional leaders and financial aid staff who participated in the baseline data collection efforts detailed here to assess institutional changes that occur as a result of the PEP recommendations. Protocols for those additional rounds of research will be developed and submitted to OMB for approval at a later date.

AMOUNT OF ANY PROPOSED STIPEND OR INCENTIVE

Not applicable.

BURDEN HOUR COMPUTATION (Number of responses (X) estimated response or participation time in minutes (/60) = annual burden hours):

Category of Respondent	No. of Respondents	Participation Time	Burden
Presidents/CEOs and financial aid staff	48	60 minutes	48 hours
Totals	48	60 minutes	48 hours

BURDEN COST COMPUTATION

Category of Respondent	No. of Respondents	Hourly Rate	Response Time	Total
Presidents/CEOs	12	$84.88	1 hour	$1,018.56
Financial aid staff	36	$57.91	1 hour	$2,084.76
Totals	48		1 hour	$3,103.32

STATISTICAL INFORMATION

We expect a 100% response rate in terms of institutional leaders and financial aid staff participating in interviews and focus groups. Given that all data gathered from these data collection efforts will be qualitative, minimal statistical analysis will be conducted. Frequencies of themes that emerged may be presented, but the vast majority of the findings from these data collection efforts will communicate broad qualitative themes rather than present quantifiable results.

REQUESTED APPROVAL DATE: 10 business days past submission date

NAME OF CONTACT PERSON: Chris Lemmie

TELEPHONE NUMBER: (202) 377-3225

MAILING LOCATION: 830 First St. N.E., Washington, DC 20202

ED DEPARTMENT, OFFICE, DIVISION, BRANCH: Chief Customer Experience Office,

Office of Federal Student Aid, U.S. Department of Education

Financial Aid Staff Group Protocol

OMB Approval No.: 1845-0045

Expires: 00/00/0000

FSA MSURSD MSI Support

TASK 2.3C: Development of Baseline Performance Report for Each PEP Participating Institution

Financial Aid Staff Focus Group

PLEASE READ THE FOLLOWING TO RESPONDENTS

Introductory Script for Financial Aid Staff Focus Group: Good morning/afternoon. First of all, thank you so much for being here and for being willing to participate in this study for the US Department of Education. Before we start, I would like to introduce myself, provide a little background on my work, and answer any questions you might have.

INTRODUCTION AND BACKGROUND

My name is name and I am a researcher from Windwalker Corporation, an organization that conducts research in education. I am here today to interview you as part of an effort to provide the Office of Federal Student Aid’s Minority Serving and Under-Resourced Schools Division (MSURSD) with information about your school’s student aid process and how you obtain compliance-related information. They are interested in gaining this information so that they can provide your school with additional resources and services that would assist you with these issues.

As part of this project, we are speaking with your school’s President/CEO about financial aid-related processes, procedures and information regarding federal compliance. Specifically, I’ll be asking you about topics such as how you learn about compliance-related information, the amount of leadership support you receive, what processes your office institutes for students who are receiving aid, and what resources you would like to receive from MSURSD that would enable you to increase your knowledge of compliance-related topics and to improve your processes.

I’ll talk about how this interview will go next, but for now, are there any questions about the study or why we are here?

Potential question by interviewee:

• Why was I/our school/our program chosen?

Note to interviewer - Upon learning more about the program and the basis for selection, insert general reasons that are not too leading

DESCRIPTION OF INTERVIEW PROCEDURE AND CONFIDENTIALITY

Next, I’d like to say a few things about the process for this discussion. During our session today, I'll be asking you some questions and I will be taking notes on what you are saying. I will not record your names.  If you don’t mind, I would also like to record our conversation simply for note-taking purposes. No one outside of our research team would hear the recording; it would just be for our own reference. If you would like me to turn off the recorder at any point, just let me know. Would that be OK?

I want to assure you that all of the information you provide today will be kept confidential and will only be used for the purposes of this study. The recorded interview will be transcribed by me, and portions of this interview may be incorporated into the final report to the Department of Education but your identity will remain confidential. I will not use your name or attribute any quotes to you. Therefore, anyone who reads the study will not know that the statements came from you. The results of this study will be used for FSA use only to determine if the services and resources that they provide to your school increase knowledge of federal compliance-related issues and if they enhance the processes that your financial aid office engages in. Whatever you express in this room stays in this room. We’re asking you to make the same commitment - to keep what is said in this room confidential and not repeat it to others in your school.  That way people can feel safe to be honest, and in turn, we will gain more helpful information to improve the financial aid process at your school. Agreed?

This session will give you a chance to express your opinions.  I want you to feel comfortable in saying what you really think and feel, as your ideas and opinions are important to me. It is OK to disagree with one another. I’d like to hear everyone’s perspective. If you do not feel like responding to a question, you do not have to.

Are there any questions before we get started?

If you agree to the terms mentioned we will accept your verbal consent to participate and proceed with this interview.

Note to interviewer:

Throughout the interview, possible probes are set aside following each general question. Asking these questions may help to move the interview pace or may prompt a less talkative respondent. However, do not feel you need to ask each of these probes; it is likely that the respondent will cover many of these issues when responding to your initial inquiry. Keep the tone conversational and comfortable

I would like to start by learning about each of you and your roles.

1. What is your role and how long have you been working at this school?

1. Have you always worked in the financial aid office? If not, where else have you worked and for how long?

2. Have you had a role in financial aid at another school? If so, what role did you have and for how long?

Thank you for providing me with some information about your backgrounds. Now I would like to begin discussing compliance-related issues.

3. First, how do you learn about federal compliance-related issues?

What to listen for and probe, if necessary:

• What resources and tools are used?

• Who provides this information?

4. Do you think this is sufficient to keep you on top of changes to issues related to compliance? Why or why not?

5. How else would you like to obtain this information? Who do you feel should provide this?

6. Do you feel that institutional leadership is knowledgeable about compliance issues?

If yes:

6a. How so?

If no:

6b. Why not?

7. Would you like for your institution’s leaders to be more engaged with these issues?

If yes:

7a. Why?

7b. How do you think they can become more engaged?

If no:

7c. Why not?

Now I’d like to start talking about the processes and procedures that your office engages in.

8. What are the financial-aid related processes that your office oversees for title IV programs (including federal grants, work study and loans)?

Listen for specific processes and details for federal grants, work study and loans.

9. How are these processes developed?

What to listen for and probe if necessary:

• Who is involved in the development?

• What is the role of each?

10. What aspects of process development work well? For instance, do people collaborate well, are timelines met, are all participants knowledgeable, etc.?

11. What challenges do you experience when developing the processes?

12. How can the development process be made better?

13. What resources do you use to assist you in process development? Why do you use these?

14. Are there additional resources that you think would be beneficial to receive to use in process development?

What to listen for and probe, if necessary:

• Specific resources they would like to receive

• What resources can MSURSD provide that would assist?

15. Do you have anything else that you would like to add before we conclude this interview?

Those are all the questions I have. Thank you for your time and for participating in this study.

Institutional Leader Individual Protocol

OMB Approval No.: 1845-0045

Expires: 00/00/0000

FSA MSURSD MSI Support

TASK 2.3C: Development of Baseline Performance Report for Each PEP Participating Institution

Institutional Leader Interview

PLEASE READ THE FOLLOWING TO RESPONDENTS

Introductory Script for Financial Aid Staff Focus Group: Good morning/afternoon. First of all, thank you so much for being willing to participate in this study for the US Department of Education. Before we start, I would like to introduce myself, provide a little background on my work, and answer any questions you might have.

INTRODUCTION AND BACKGROUND

My name is name and I am a researcher from Windwalker Corporation, an organization that conducts research in education. I am here today to interview you as part of an effort to provide the Office of Federal Student Aid’s Minority Serving and Under-Resourced Schools Division (MSURSD) with information about your school’s student aid process and how you obtain compliance-related information. They are interested in gaining this information so that they can provide your school with additional resources and services that would assist you with these issues.

As part of this project, we are also speaking with your financial aid staff about financial aid-related processes, procedures and how they obtain information regarding federal compliance. Specifically, I’ll be asking you about topics such as the extent to which you are involved with compliance-related issues, what you view as the financial aid office’s successes and challenges and your vision for how knowledge of compliance-related issues and development of financial aid processes can be improved across your institution.

I’ll talk about how this interview will go next, but for now, are there any questions about the study or why we are here?

Potential question by interviewee:

• Why was I/our school/our program chosen?

Note to interviewer - Upon learning more about the program and the basis for selection, insert general reasons that are not too leading

DESCRIPTION OF INTERVIEW PROCEDURE AND CONFIDENTIALITY

Next, I’d like to say a few things about the process for this discussion. During our session today, I'll be asking you some questions and I will be taking notes on what you are saying. I will not record your name.  If you don’t mind, I would also like to record our conversation simply for note-taking purposes. No one outside of our research team would hear the recording; it would just be for our own reference. If you would like me to turn off the recorder at any point, just let me know. Would that be OK?

I want to assure you that all of the information you provide today will be kept confidential and will only be used for the purposes of this study. The recorded interview will be transcribed by me, and portions of this interview may be incorporated into the final report to the Department of Education but your identity will remain confidential. I will not use your name or attribute any quotes to you. Therefore, anyone who reads the study will not know that the statements came from you. The results of this study will be used for FSA use only to determine if the services and resources that they provide to your school increase knowledge of federal compliance-related issues and if they enhance the processes that your financial aid office engages in.

Are there any questions before we get started?

If you agree to the terms mentioned we will accept your verbal consent to participate and proceed with this interview.

Note to interviewer:

Throughout the interview, possible probes are set aside following each general question. Asking these questions may help to move the interview pace or may prompt a less talkative respondent. However, do not feel you need to ask each of these probes; it is likely that the respondent will cover many of these issues when responding to your initial inquiry. Keep the tone conversational and comfortable

I would like to start by learning a bit about your background.

1. How long have you been working at this school?

2. Have you always been the President/CEO? If not, what role did you have before becoming the President/CEO

Thank you for providing me with some information about your background. Now I would like to turn our discussion toward compliance-related issues and financial aid processes.

3. How involved are you with compliance-related issues and financial aid processes?

What to listen for and probe if necessary:

• What tasks do they do?

• How much time do they spend on such tasks?

If they are involved:

• 3a. What is your role in these?

• 3b. How do you stay engaged with these issues?

• 3c. What resources do you use to stay engaged? Who provides them?

If they are not involved:

• 3d. Why aren’t you involved?

• 3e. Who is involved and what are their roles?

4. To what extent do you want to be involved in these process and compliance-related issues? Why?

5. Are you aware of ways in which the financial aid office is being successful? How?

6. Are you aware of the challenges faced by the financial aid office?

If yes:

6a. What are these challenges?

6b. What are your thoughts on how these challenges can be remedied?

If no:

6c. Who else, other than the financial aid team, know about the challenges faced by the office?

7. Are there any resources that you would like to be provided to you or your financial aid staff to assist with compliance and/or implementation of aid-related processes?

If yes:

7a. What resources?

7b. Why do you think they would be helpful?

7c. Who do you think should provide them?

If no:

7d. Why wouldn’t you like to be provided with resources?

8. Do you have anything else that you would like to add before we conclude this interview?

Those are all the questions I have. Thank you for your time and for participating in this study.