Carried over App H

This document is being carried over from October 2015 clearance

Appendix H
Student Records Institution Staff Focus Group Final Report

October 2015

NPSAS/NCES STUDENT RECORDS

FOCUS GROUPS

Prepared for: RTI ON BEHALF OF NPSAS

Prepared by: SHUGOLL RESEARCH 7475 Wisconsin Avenue Suite 200 Bethesda, Maryland 20814 301-656-0310 www.shugollresearch.com

August 2015

TABLE OF CONTENTS

Section		Page
1.0	Background and Research Methodology	4
2.0	Research Objectives	5
3.0	Conclusions and Implications	7

1.0 BACKGROUND AND RESEARCH METHODOLOGY

Background And Purpose

• NPSAS contracted with Shugoll Research to bring together various institution staff experienced with the NPSAS:16 field test to assess and better understand ways to improve the procedures that NCES uses to collect student records data.

• Feedback from this research will help guide development of the revised data elements and data abstraction instrument for NPSAS:16 and possibly other NCES studies.

Research Methodology

• A total of three, 90-minute focus groups were conducted virtually or remotely from respondents’ own computers as follows:

• The virtual remote platform allowed RTI and NCES project staff to observe each session via online video streaming.

• Respondents were recruited by RTI project staff using its list of institutional contacts, and then rescreened and confirmed by Shugoll Research recruiting staff.

• Respondents had to meet the following recruiting specifications:

• Must be familiar with NPSAS:16 field test or Student Records Data Collection process

• Must be involved in completing the NPSAS reports

• Must have access to high speed internet

• Must represent a mix of institution types if possible

• For documentation purposes, the recruiting disposition is included below:

Limitations

• A qualitative research methodology seeks to develop direction rather than quantitatively precise or absolute measures. The limited number of respondents involved in this type of research means the results should be regarded as directional in nature and be used to generate hypotheses for future decision making.

• The non-statistical nature of qualitative research means the results cannot be generalized to the population under study with a known level of statistical precision.

• In some instances, it was difficult for focus group participants to recall specific aspects of the NPSAS Student Records Data Collection process.

2. RESEARCH OBJECTIVES

The specific study objectives are to:

1. Discuss Proposed Enrollment List Eligibility Items

2. Discuss Collection of Term Information

3. Obtain Reactions to Revised Definitions

4. Evaluate the Mode of Data Collection (i.e., Excel, CSV or Web Modes)

5. Discuss the Postsecondary Data Portal

6. Discuss the NCES Data Lab

A series of slides as well as live pages from the NPSAS website provided content and served as stimuli for the study participants when discussing their experiences with the NPSAS Data Collection process.

2. CONCLUSIONS AND IMPLICATIONS

Enrollment List of Eligibility Items

The decision to change the process of determining student eligibility to a "yes" and "no" system should be re-evaluated and perhaps re-considered. The new system would undoubtedly increase the amount of time needed to complete this section of the data collection process and make the task more challenging for schools with less flexible information systems or for certain questions where it is difficult to answer "yes" or "no".

• No one expresses significant concerns regarding the current process to assess student eligibility so they wonder why NCES wants to change it to a "yes" or "no" format.

• There is some concern that having to answer "yes" or "no" for multiple questions per student will take the schools longer to complete the student eligibility section and require more collaboration between departments. Also, it is important to note that some schools simply do not have "yes" or "no" answer capabilities in their information systems so they would have to spend time manipulating the data. Finally, some questions are not necessarily black and white so they don't always lend themselves well to "yes" or "no" answers.

• Schools prefer not to have to return to the same student record repeatedly. Therefore, they would prefer to know upfront all the information they are being asked to gather about each student so they can do it all at one time.

• Duplicative data requests should ask that the data be presented in the same way. In other words, the data should be requested in a consistent format so it can be copied and pasted easily in multiple places or so that the data automatically populates from the first time it was entered.

Collection of Term Information

It appears as if the majority of schools can map their terms into three terms, while some may have to add approximately two more terms. Rarely is there a need to map terms in the double digits. In general, there does not seem to be a great need to change this section of the data collection process.

• Schools do not seem to have too much difficulty reporting term information unless they are being asked to provide data for a term that has not yet begun.

• Most schools do not appear to have the need to account for more than five terms and most say they report three terms: fall, spring and summer. Mini sessions are typically imbedded in one of the larger terms.

• There seems to be the possibility of a greater number of terms for online programs as compared to more traditional in-person education.

Reactions to Revised Definitions for Degree Program

The revised definitions and accompanying answer options are clear and comprehensive for degree program. In general, they work well. Perhaps some effort should be made to provide instructions on how schools can help their double major and dual degree students maximize financial aid. Additionally, possibly consider revising the order of the answer options so that all the "non-degree" options are grouped together.

• Schools are familiar and comfortable with the revised definitions and response options for both the degree programs.

• Schools express some concern over how to ensure that their students get the appropriate and needed financial aid for their major, especially for students who end up changing their major along the way.

• Generally speaking, when students pursue double majors or dual degrees, they are told to declare as their major or degree, the most expensive one to optimize their financial aid they are due.

• Some study participants suggest it would make more sense to reorder the answer options so that all the non-degree options are grouped together.

Reactions to Revised Definitions for Class Level

It would be beneficial to provide more guidance on how schools should classify levels for their various types of graduate students.

• Students have no difficulty selecting or mapping class levels for their undergraduate students and say the options provided are comprehensive. The definitions and response options are clear and complete.

• Schools recommend that the request for class level, particularly for undergraduates, be based on credits completed and not credits enrolled. This would mitigate any issues related to the timing of the data collection and how it could influence class level designation.

• Schools are less confident about how to classify class level for graduate students and often use "graduate unclassified" or classify their professional degree students as first year…, second year…, third year… and so on.

Identifying Individuals Involved in the NPSAS Student Records Data Collection Process

• Typically, the NPSAS Student Records Data Collection Process is a collaborative effort. Individuals completing the data collection process primarily work in Institutional Research & Assessment, Financial Aid, the Business Office and the Registrar.

• Coordination between departments is relatively easy as long as those involved meet upfront to discuss the information needs of the initiative. Knowing the type of information required for the NPSAS Student Records Data Collection process is key to successful and timely completion of the request.

• Schools would like the data collection process to be more flexible in terms of what type of information is needed and how the data can be formatted. They do not want to be asked for the same data multiple times, they want consistency regarding the type and format of the information, and they want to minimize add-ons or changes. These modifications would expedite completion of the NPSAS Student Records Data Collection Process.

• Finally, schools would like to see the Student Records Data Collection Process reflect the times so that response options such as "Hispanic", "multi-racial" and "non-resident alien" are included.

Feedback from Users of Excel Mode, CSV Mode and Web Mode

Excel Mode - The Excel Template is not always easy to use primarily because it is not very flexible. In particular, there is a lack of formatting flexibility including the naming and sizing of the columns. Therefore, those who complete the Student Records Data Collection Process in Excel would like to see the template disabled. Also, although there are few problems associated with rectifying simple error messages, it is much more difficult to rectify errors that are caused by empty cells or entirely empty columns. Better instruction is needed to help Excel users who complete the data collection process avoid these kinds of error messages.

Excel Mode - Review Your Data Page - Few use this page. However, those who use Excel would like to see it accommodate Macros and UBScripts so they can validate their information as it is being entered.

CSV Mode - In general, using the CSV file specifications seems to be relatively easy. Problems come up when data is unknown or when categories and nomenclature in the field test do not match what the school uses. These problems center on race designations, financial aid information and the fact that there is no "unknown" or "default" answer option to use. Users do not seem to have much trouble correcting simple error messages, but again when entire columns are missed or empty, they usually have to lean on tech support for help. Finally, schools would like assurance that their "final" or "official" file is the one that gets submitted, since they can't figure out how to delete previous versions that might either be incorrect or incomplete.

CSV Mode - Review Your Data Page - Few use this page. Some schools could not even open the page. One complaint among those who did use it is that it is difficult to verify or validate answers because "codes" are used rather than actual responses.

Web Mode - The Web Mode Grid is easy to use. Data entry is simple and they particularly like seeing the progress status i.e., the green check marks. Users also like the auto-populate feature and have few, if any, concerns about the text boxes, drop downs, date pickers and data formats. They would prefer not having to fill in place holder responses e.g., "not enrolled" for students not enrolled in various terms. Instead, they'd prefer to be allowed to leave the space blank. There is also a strong preference for combining information requests that relate. For example, if schools are being asked to provide both term information and budget information, it would make more sense to ask these things together since they are related. Users say it is generally easy to fix errors. The errors are easy to find and correct.

The enrollment calendar is a focus of some concern among web users because it asks for the year first rather than the month, which is counter intuitive. Another problem is when NCES asks the schools to provide personably identifiable information or other types of personal information on their students that they are not required to collect or maintain. Some web mode users also think it is difficult to activate the "save" function when they need it.

Post-Secondary Data Portal

Too few have experience with the Post-Secondary Data Portal to provide significant feedback. Among the one or two with some experience, their comments are generally positive suggesting that greater use of the portal could be advantageous to schools that participate in the NPSAS/NCES Student Records Data Collection process.

NCES Data Lab

The majority of schools are unfamiliar with the NCES Data Lab. After a quick cursory review of the NCES Data Lab there is keen interest in the tool and the information it provides. NCES should consider increasing awareness of its Data Lab and the benefits of the data it can provide.