2019-20 National Postsecondary Student Aid Study (NPSAS:20) Pretesting

National Center for Education Statistics (NCES)

Attachments XIV – XVII

2019-20 National Postsecondary Student Aid Study (NPSAS:20) Pretesting

OMB# 1850-0803 v. 243

December 2018

Attachments

Attachment XIV – Usability Testing of the PDP: Recruitment Procedures and Materials

Attachment XV – Usability Testing of the PDP: Screening Questions and Intake Form

Attachment XVI – Usability Testing of the PDP: Consent to Participate in Research

Attachment XVII – Usability Testing of the PDP: Usability Testing Protocol

Contents

Attachment XIV. Usability Testing of the PDP: Recruitment Procedures & Materials 3

Recruitment E-mail to Potential Participants 3

Frequently Asked Questions (FAQs) 4

Confirmation E-mail Usability Testing 5

Thank You E-mail/Letter 6

Attachment XV. Usability Testing of the PDP: Screening Questions & Intake Form 7

Recruitment Screening Questions 7

Recruitment Website Intake Form 9

Attachment XVI. Usability Testing of the PDP: Consent to Participate in Research 11

Attachment XVII. Usability Testing of the PDP: Usability Testing Protocol 13

Sample Probes 14

Attachment XIV. Usability Testing of the PDP: Recruitment Procedures & Materials

Recruitment E-mail to Potential Participants

Subject: NCES needs your feedback on NPSAS:18-AC

<DATE>

Dear <NAME>,

EurekaFacts, a research company based in Rockville, MD, is administering an important study on behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES) in collaboration with RTI International. The purpose of this research is to improve the 2019–20 National Postsecondary Student Aid Study (NPSAS:20), which NCES uses to collect important information about how students and parents finance postsecondary education. As a recent participant in NPSAS:18-AC, we need your help.

We are looking for staff who recently completed the NPSAS:18-AC collection to participate in one-on-one usability interviews to determine ways to improve the Postsecondary Data Portal (PDP) and other materials used by NCES to collect data.

This study is crucial to the continued success of our education system as data from NPSAS:20 will be used by the federal government to inform public policy.

The interview will be administered online and will last no more than 90 minutes. You may participate remotely from your own computer. A EurekaFacts moderator will lead the discussion on behalf of NCES and all participants should be comfortable and willing to contribute their thoughts and opinions. Because we care about your privacy, we will assign you a unique identifier (ID) and your responses will be kept separately from your name and the name of your institution.¹

Your feedback will help NCES revise the data collection instruments so they are clear and can be easily used by thousands of other postsecondary administrators across the country. You will receive a $50 Amazon gift card at the end of the session as a thank you for your time and effort in participating.

I have included a list of Frequently Asked Questions with additional information about the study. Please call <CONTACT PHONE> or email <CONTACT EMAIL> with any questions regarding this project.

If you are interested in participating, please fill out the following online form: <URL/LINK TO INTAKE FORM>

We look forward to speaking with you soon!

Sincerely,

<CONTACT NAME>

<JOB TITLE>

EurekaFacts, LLC <CONTACT PHONE>

ENCLOSED: Frequently Asked Questions (FAQ)

Frequently Asked Questions (FAQs)

2019–20 National Postsecondary Student Aid Study (NPSAS:20) PDP Usability Testing

1. What is this study about?

On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES) and in collaboration with RTI International, EurekaFacts, a research company based in Rockville, MD, is administering one-on-one interviews with staff at various postsecondary educational institutions for the editing of the National Postsecondary Student Aid Study 2020 (NPSAS:20). The purpose of the interview is to help NCES better understand the usability of the Postsecondary Data Portal (PDP) website. Feedback from the interview will be used to improve student records collection instruments for NPSAS:20 and other NCES studies.

2. Where will the interview take place?

Interviews will be virtual and take place online. You will receive a confirmation email with the link and login information.

3. What will happen during the interview?

During the interview, participants will complete a series of tasks using the Postsecondary Data Portal website and will be asked to provide feedback on the process. All sessions will be recorded to make sure we don’t leave out or misrepresent anything you say. Please see question 5 for information regarding the protection of your privacy.

4. Do I need to do anything special for this online interview?

For online interviews, you will need to have a reliable internet connection, a webcam and a quiet private space where you can engage in the session for its duration. You will need to return your electronically signed consent form prior to the start of the session. Further instructions will be provided in a confirmation email.

5. Will information be kept private?

Each participant will be assigned a unique identifier (ID) so your name will not be linked to your answers. All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Audio from the session will be recorded only for analysis. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

6. How long will the interview last and when will it take place?

The interview will last approximately 90 minutes. Interviews can be scheduled at your convenience.

7. Will the results go to the participant’s place of work?

Participants’ responses will not be disclosed to their place of work and will not be linked to any personally identifiable information.

8. Is there an incentive for participating?

Yes. Upon completion of the interview, each participant will receive a $50 Amazon gift card. This will be sent via email. You may decline the gift card if required by your institution.

9. What are the possible risks of being in the study?

There are no known risks to participating in this voluntary study.

10. Who can I contact with questions or for further information?

If you have any questions about the study, you can call <NAME>, the <TITLE> at <CONTACT PHONE>. If you have questions about your rights as a study participant, you can call the EurekaFacts’ Office of Research Protection at <PHONE NUMBER> (a toll-free number).

11. Can participation end early?

Participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to withdraw from this study at any time and can leave the interview early if you wish.

Confirmation E-mail Usability Testing

Subject: Confirmation for NPSAS usability testing

<DATE>

Dear <NAME>:

Thank you for agreeing to participate in an online usability interview for the National Postsecondary Student Aid Study. The session will last about 90 minutes.

This is a confirmation that your one-on-one usability interview has been scheduled at the following location, date, and time:

LOCATION: Your virtual <<focus group, interview>> will take place online. The link for participation is <<LINK.>> The log in is <<LOGIN>>.

DATE: Day of Week, Month Day, Year

TIME: XX:XX AM/PM <TIMEZONE>

GIFT CARD: At the conclusion of the session you will be emailed a $50 Amazon gift card as a thank you for participating. You may decline the gift card if required by your institution.

According to our records, the email address and phone number where you will be participating are <<EMAIL>> and <<PHONE>>. If this information is inaccurate, please contact us using the information below.

PLEASE READ THE FOLLOWING INSTRUCTIONS:

Please use these directions below for participating in your interview:

• Please be in a quiet private location with good quality internet service and a working webcam.

• Locate this email.

• Open the consent form, type your name and date, save it. Please reply all to this email when you attach the consent form and send it to us. You will have to return your electronically signed consent form before beginning the session.

• Please be logged into the link 5 minutes before> your session is scheduled to begin.

We look forward to speaking with you. If you have any questions or concerns or if your above contact information is incorrect, please contact <CONTACT NAME> at <CONTACT INFORMATION>.

<CONTACT NAME>

EurekaFacts, LLC

<CONTACT INFORMATION >

ATTACHED: Consent Form

Thank You E-mail/Letter

Subject: Thank you from NCES

<DATE>

Dear <Name>,

On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES) and RTI International, EurekaFacts would like to thank you for participating in our study.

Your participation provided invaluable feedback about the forms and procedures NCES uses to collect institutional education data. The insightful input we received from you, combined with input from other professional educators, is incredibly informative and is helping us develop the best possible procedures and systems for the 2019–20 National Postsecondary Student Aid Study.

Again, thank you for your assistance.

Sincerely,

<CONTACT NAME>

<CONTACT TITLE>

EurekaFacts, LLC

Attachment XV. Usability Testing of the PDP: Screening Questions & Intake Form

Recruitment Screening Questions

(Script when calling potential participants from a compiled list)

Good morning/afternoon/evening. Is this <TITLE AND LAST NAME FROM CALL LIST>? Good <MORNING/AFTERNOON/EVENING> My name is <CALLER 1^st AND LAST NAME>, and I am calling from EurekaFacts, a Maryland-based research company. We are administering an important national study in collaboration with RTI International on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education, and we need your help.

We are looking for staff who recently completed the NPSAS:18-AC collection to participate in research sessions to determine ways to improve the Postsecondary Data Portal (PDP) and other materials used by NCES to collect data. All participants will receive a $50 Amazon gift card as a ‘Thank you’ for their time and participation.

I’d like to tell you a little bit more about the study, but first, if now is a good time, can I ask you a few questions to see if you are eligible to participate? [YES or NO and call back later]

1. Do you currently work at [institution name]?

   • Yes  CONTINUE TO Q2

   • No  THANK AND TERMINATE

2. Were you one of the people responsible for providing data for NPSAS:18-AC?

   • Yes  CONTINUE TO Q3

   • No  ASK FOR THE PERSON RESPONSIBLE FOR COMPLETING NPSAS:18-AC

3. Do you feel comfortable expressing and voicing your opinions to our team of researchers?

   • Yes  CONTINUE TO STUDY DESCRIPTION USABILITY TESTING

   • No  THANK AND TERMINATE

Study Description Usability Testing

Thank you. As I mentioned, the study is being administered on behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES) and is a study about post-secondary educational professionals’ experiences using and interacting with the Post-Secondary Data portal. Participants will be asked to work one-on-one with the interviewer, through various tasks within the PDP. Participants will also be asked to provide feedback about the PDP tasks, their own experiences using the PDP’s data entry features, as well as any barriers to successfully and efficiently utilizing the PDP. This study will help improve the experiences of thousands of other post-secondary staff across the country when using the Postsecondary data portal each year.

The interview will last no more than 90 minutes and can be scheduled at your convenience and will be conducted over the internet at a time convenient to participants. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). At the conclusion of your session you will receive a $50 Amazon gift card.

4. Are you interested in participating in this important study?

   • Yes  CONTINUE TO Q5

   • No  THANK AND TERMINATE

If YES: Great, thank you very much. To ensure that we interview a broad mix of individuals, I have a few brief screening questions to ask you. It should only take a few more minutes of your time.

5. How long have you worked in postsecondary education?

• Less than 1 year

• 1-4 years

• 5-10 years

• 10-15 years

• 15-20 years

• 20+ years

6. Do you have personal experience using the PDP?

• Yes

• No

7. To participate in the interview and receive the $50 Amazon gift card we must receive a signed consent form. You must return an electronically signed consent form before the remote session can begin. It is extremely important that we have your signed consent form and that you have joined the session 5 minutes PRIOR to the scheduled start time. Participation will be closed 5 minutes after the scheduled start time of the session. Do you understand that you must have a completed and signed consent form to participate?

• Yes

• No  THANK AND TERMINATE

8. Thank you for your interest. If you qualify, we will follow-up with you to schedule the interview. Could you please provide us with your contact information? (All required except “Alternate phone number”)

RECORD FIRST NAME: _____________________________________

RECORD LAST NAME: _____________________________________

RECORD EMAIL ADDRESS: _____________________________________

BEST PHONE NUMBER: _____________________________________

ALTERNATE PHONE NUMBER: _____________________________________

[Termination Language: Based on the requirements of this study, we are not able to include you in the interviews at this time. We greatly appreciate your time speaking with us. Have a good <day/evening>.

Recruitment Website Intake Form

Paid Research Study Opportunity for Education Professionals

Thank you for your interest in our study!

EurekaFacts, a research company based in Rockville, MD, is administering an important national study in collaboration with RTI International on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education.

We are looking for administrators and administration staff at institutions offering postsecondary educational programs to participate in educational research to determine ways to improve the Postsecondary Data Portal (PDP) and other materials used by NCES to collect data. All participants will receive a $50 Amazon gift card as a thank you for your time and participation.

The one-on-one usability interview will last no more than 90 minutes and can be scheduled at your convenience and will be conducted over the internet at a time convenient to participants. All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

If you qualify and participate in the study, you will receive a $50 Amazon gift card as a thank you at the end of the session.

To ensure that we interview a broad mix of individuals, please fill out the screening questions below.

If you have any questions, please contact <NAME> at <PHONE> or <EMAIL>.

1. Do you currently work at an institution that offers a post-secondary degree or certificate?

   • Yes

   • No  THANK AND TERMINATE

2. Were you one of the people responsible for providing data for NPSAS:18-AC?

   • Yes  CONTINUE TO Q3

   • No  ASK FOR THE PERSON RESPONSIBLE FOR COMPLETING NPSAS:18-AC

3. Which of the following best describes your role or job title?

   • Administration

   • Institutional Research/Institutional Effectiveness

   • Registrar

   • Financial Aid

   • IT/Networking or Support Staff

   • None of the Above  THANK AND TERMINATE

4. Would you be interested in participating?

   • Yes

   • No  THANK AND TERMINATE

5. What is the name of the institution you work for and where is it located?

Name: _________________________________________

City: ______________________

State: ____________________

6. How long have you worked in postsecondary education?

• Less than 1 year

• 1-4 years

• 5-10 years

• 10-15 years

• 15-20 years

• 20+ years

7. We are conducting interview sessions between <DATE> and <DATE>. Please select three (3) days and times that would work for you to participate in the research interview.

RECORD MOST PREFERRED DATE AND TIME: _____________________________________________

RECORD 2^nd ALTERNATE DATE AND TIME: ________________________________________________

RECORD 3^rd ALTERNATE DATE AND TIME: ________________________________________________

8. To participate in the interview and receive the $50 Amazon gift card we must receive a signed consent form. You must return an electronically signed consent form before the remote session can begin. It is extremely important that we have your signed consent form and that you have joined the session 5 minutes PRIOR to the scheduled start time. Participation will be closed 5 minutes after the scheduled start time of the session.

Do you understand that you must have a completed and signed consent form to participate?

• Yes

• No  THANK AND TERMINATE

9. Thank you for your interest. If qualifications are met, we will call to follow-up with you about additional questions we may have and potentially scheduling an interview. Could you please provide us with your contact information? (All required except “Alternate phone number”)

RECORD FIRST NAME: _____________________________________

RECORD LAST NAME: _____________________________________

RECORD EMAIL ADDRESS: _____________________________________

BEST PHONE NUMBER: _____________________________________

ALTERNATE PHONE NUMBER: _____________________________________

[Termination Language: Based on the requirements of this study, we are not able to include you in the interviews at this time. We greatly appreciate your time speaking with us. Have a good day.]

Attachment XVI. Usability Testing of the PDP: Consent to Participate in Research

CONSENT FORM

FOR PARTICIPATION IN RESEARCH

Thank you for your interest in participating in the study described below. This consent form explains the study. Please read it carefully and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them later as they occur to you.

1. What is this study about?

On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES) and in collaboration with RTI International, EurekaFacts, a research company based in Rockville, MD, is administering one-on-one interviews with staff at various postsecondary educational institutions for the editing of the 2019–20 National Postsecondary Student Aid Study (NPSAS:20). The purpose of the interview is to help NCES better understand the usability of the Postsecondary Data Portal (PDP) website. Feedback from the interview will be used to improve student records collection instruments for NPSAS:20 and other NCES studies.

2. Where will the interview take place?

Interviews will be virtual and take place online. You will receive a confirmation email with the link and login information.

3. What will happen during the interview?

During the interview, participants will complete a series of tasks using the Postsecondary Data Portal website and will be asked to provide feedback on the process.

4. Do I need to do anything special for this online interview?

For online interviews, you will need to have a reliable internet connection, a webcam and a quiet private space where you can engage in the session for its duration. You will need to return your electronically signed consent form prior to the start of the session. Further instructions will be provided in a confirmation email.

5. Will information be kept private?

Each participant will be assigned a unique identifier (ID) so your name will not be linked to your answers. All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Audio from the session will be recorded only for analysis. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

6. How long will the interview last and when will it take place?

The interview will last approximately 90 minutes. Interviews can be scheduled at your convenience.

7. Will the results go to the participant’s place of work?

Participants’ responses will not be disclosed to their place of work and will not be linked to any personally identifiable information.

8. Is there an incentive for participating?

Yes. Upon completion of the interview, each participant will receive a $50 Amazon gift card. This will be sent via email. You may decline the gift card if required by your institution.

9. What are the possible risks of being in the study?

There are no known risks to participating in this voluntary study.

10. Who can I contact with questions or for further information?

If you have any questions about the study, you can call <NAME>, the <TITLE> at <CONTACT PHONE>. If you have questions about your rights as a study participant, you can call the EurekaFacts’ Office of Research Protection at <PHONE NUMBER> (a toll-free number).

11. Can participation end early?

Participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to withdraw from this study at any time and can leave the interview early if you wish.

____ Yes, I agree to participate in this study

____ No, I do not want to participate in this study

-----------------------------------------------------------------------------------------------------------------------------------------------

By signing below, you agree that you, _______________________________, will participate in this study and that we may record your conversation with the researcher. You also agree that you will not use or disclose any information provided by your fellow focus group participants in identifiable form. You will receive a $50 Amazon gift card via email after your session has ended.

Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

We thank you for your voluntary participation and ask you to sign the form below.

Print Name: _____________________________________________

Signature:______________________________________________ Date:________________________

To be completed by EurekaFacts staff:

Payment Card Transaction Number: _______________________________________________________

Attachment XVII. Usability Testing of the PDP: Usability Testing Protocol

Usability Testing Protocol

Interviewer instructions

This guide will be used to steer discussion to the topic areas to be covered, and the specific questions of interest within each topic area. For the purposes of qualitative research, it is not necessarily intended that these questions be asked exactly as they are worded here. Usability Interviews are more like a conversation than a set of structured questions. The discussion should be as informal as possible, and participants should be encouraged to speak openly and freely. Because each interview participant is different, a responsive and flexible approach should be used for the focus group discussion.

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Participants will be administered different items and probes based on the Mode selection appropriate for their institution. The duration of each interview will be 90 minutes.

** START RECORDER**

Introduction to the Study

Hello, my name is ______. Thank you for participating in this study today. We are currently in the process of evaluating the Postsecondary Data Portal website, which is used to collect data for the National Postsecondary Student Aid Study (NPSAS). In a few minutes I will show you the PDP website, and then I will ask that you use it to complete several tasks. I will ask for your opinion about various features on the website and the information being presented to you.

Some key things to remember as we move along:

• There is no right or wrong answer. We are not testing you or your knowledge. We are here to evaluate the website.

• What we are concerned about is the usability of this website, its features, functionality, and information presentation preferences. This will help provide us with information that will help users like you have a better overall experience.

• We are also very interested in your thoughts and opinions as we proceed. So please “think aloud” as you try to find information. During a “think aloud” I would like you to verbalize everything that you are thinking, including your impressions and thoughts while you work through a given task or respond to a certain question. Tell me your reasoning, your understanding of the information we are dealing with, and also if there is anything that you are not sure of or are confused by.

• We are interested in both positive and negative feedback you might have. I did not design the material we will review together so your comments do not affect me personally.

• When you would like to talk about something in particular, please indicate and point to the section you are talking about (either with your finger or the mouse). That way we can capture on video exactly what aspect you’re talking about.

• As you know, we are videotaping so that we can review what happened. Neither your name nor any other identifying information about you will be associated with the video or any other information you provide.

• A colleague is observing to help make sure we capture everything that happens.

Do you have any questions?

Sample Probes

1. If the respondent is quiet/not answering (this is usually during a Think Aloud), offer a verbal nudge to get them talking, such as:

   1. What is going on in your head right now?

   2. I see you’re looking at the <item>. What are you thinking?

   3. You seem to be thinking hard – can you tell me what’s in your mind right now?

   4. What are you thinking?

   5. Any other thoughts?

2. When the respondent says something, but it is difficult to tell exactly what they mean, or if they do not say very much, urge the respondent to elaborate:

   1. Tell me a bit more about that.

   2. That’s interesting – can you say more about it?

   3. Go on…

   4. Okay, and what else are/were you thinking?

3. If more clarity or a more explicit description is desired, encourage further explanation, such as:

   1. Can you explain what you mean by that?

   2. Can you say a bit more to help me understand what you mean/are thinking/were thinking?

All Groups

Introduction & Home Page

Take a look at the PDP homepage (present portal page to participant).

Question/Prompt	Response
How recently have you visited the PDP website?
Tell me about your experience using the PDP website for NPSAS:18-AC.

1. Direct participant’s attention to the study icons at the bottom of the screen.

Question/Prompt	Response
Have you ever noticed the study icons at the bottom of the screen? Have you ever clicked on them?
Have you ever visited the Datalab website? IF YES: Tell me about your experience. IF NO: Would you be interested in receiving materials from NCES about how to use it?
Direct the participant’s attention to the FAQ link at the top of the page. Have you ever reviewed the FAQs on this page? How useful were they?

Task Menu

Take a look at the ‘Postsecondary Data Portal Task Menu’ page (present portal page to participant).

Question/Prompt	Response
What is your first impression of this page?
Can you describe the resources available to you on this page?
Did you find any words or part of the page confusing? IF YES: What did you find confusing? What could we do to make the page less confusing?
Take a look at the tasks listed on this page, in your own words, what do each of these tasks mean to you?
Based on what you see here, which tasks do you think are completed, and which still need to be complete?
Based on what you see here, please describe what you would do next.
Record observations and comments here

Resources

Take a look at the sidebar to the right of the Task Menu.

Question/Prompt	Response
When you participated in NPSAS:18-AC, do you remember clicking on any of these resources? IF YES: Which one(s)? Did you find the information helpful?
Direct the participant’s attention to the FAQ link in the sidebar. Have you ever reviewed the FAQs on this page? [If no, ask the participant to click on the FAQ link.] How useful is this information to you? What information would be most helpful to you?
Direct the participant’s attention to the Schedule link in the sidebar. Have you ever reviewed the schedule on this page? [If no, ask the participant to click on the link.] How useful is this information to you? What information would be most helpful to you?
Direct the participant’s attention to the Instructions link in the sidebar. Have you ever reviewed the instructions on this page? [If no, ask the participant to click on the link.) How useful is this information to you? What information would be most helpful to you?
Direct the participant’s attention to the Contact Help Desk area in the sidebar. Did you reach out to the help desk during NPSAS:18-AC? IF YES: Did you contact them by phone or email? How easy or difficult was it to find the Help Desk information? IF NO: If not, which of these methods are you most likely to use?
If NCES offered a live chat option, would you be more or less likely to use that option?
If you needed to get access to the PDP for someone else at your institution, how would you go about doing that? If not already mentioned by the participant, direct his or her attention to the Manage My Users link in the sidebar. Ask the participant to add a user. How easy or difficult was it to add a user?
If NCES were to add an “alerts” area, where they would post updates about the project, would you find that useful? What types of information would you like to receive alerts about? Probe: What tasks you should complete next? Upcoming deadlines? Anything else?
Record observations and comments here

Institution Registration Page

Now, please click on the “Complete the Institution Registration Page.”

Question/Prompt

Response

1. Based on what you see here, please describe what you would do on this page.

2. When your institution participated in NPSAS:18-AC, were you the person who completed this page?

IF YES: Did you need input from other offices at your institution? How did that go? How long did it take you to complete this step?

IF NO: Which person/office completed this step?

3. This page asks school staff to report when they can provide the list of students enrolled at their school. Based on this example, how would you decide which option to choose?

4. Did you find any words or part of the page confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

5. Is there any information about your institution not collected here that you think we should know to improve your experience completing the study?

Prompt: Your preferences for how we contact you (phone/email)? What time of day we contact you?

Record observations and comments here

____________________________________________

_____________________________________________

_____________________________________________

Please click on the “Provide your Student Enrollment List.”

Student Enrollment List

Question/Prompt

Response

1. Based on what you see here, please describe what you need to do on this page.

Note if participant identifies two submission sections necessary to complete the form (section 3 and section 4). ______________________________________________________

______________________________________________________

______________________________________________________

2. When your institution participated in NPSAS:18-AC, were you the person who completed this step?

IF YES: Did you need input from other offices at your institution? How did that go? How long did it take you to complete this step?

IF NO: Which person/office completed this step?

3. Now please go ahead and complete this page using this information (provide participant with sample data).

Remember to “Think Aloud” as you work on this page.

Record participant’s comments and think aloud:

Probe as necessary while participant completes the task and thinks aloud.

Note if participant clicks on:  Instructions

4. Did you find any words or part of this page confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

5. I noticed you <did, did not> click on the “instructions” link <why, why not>?

6. How easy or difficult was it to understand the enrollment list preparation instructions?

7. How easy or difficult was it to complete this page?

IF DIFFICULT: What could we do to make it easier?

Record observations and comments here

Institution Information Page

Please return to the task menu and select the next task “Provide Institution Information for Student Records.”

Question/Prompt	Response
Based on what you see here, please describe what you need to do on this page.
When your institution participated in NPSAS:18-AC, were you the person who completed this step? IF YES: Did you need input from other offices at your institution? How did that go? How long did it take you to complete this step? IF NO: Which person/office completed this step?
How likely are you to watch the video on this page? Why or why not?
Please watch the video. PROBE: In your own words, how informative is the video? PROBE: Is there any other information that you would like us to include in the video?
For section 2, how would you decide which option to choose (status by month or status by term)?
For section 3, please enter a term. How easy or difficult was it to enter a term?
Did you find any words or part of this page that were confusing? IF YES: What did you find confusing? What could we do to make the page less confusing?
Record observations and comments here

Student Records – Mode Selection

Now, please go to the final task from the Task Menu (should be Provide Student Record Information). Review the “Mode Selection” page and, while thinking out loud, please explain which mode you would select and why. Please do not make the selection yet.

Question/Prompt

Response

Record participant’s comments and think aloud:

Probe as necessary while participant completes the task and thinks aloud.

If not addressed in the think-aloud, ASK:

1. Take a look at the Web Mode / Excel Mode / CSV Mode selection; in your own words, describe this mode and who should use this mode.

If not addressed in the think-aloud, ASK:

2. Did you notice the recommendations for institutions at the bottom of each mode?

Is this information helpful to you?

If not addressed in the think-aloud, ASK:

3. Why do you think the [web mode] is highlighted in blue?

4. How likely are you to watch the video on this page?

5. Please watch the video.

PROBE: Is the video helpful to you? Would it change your decision about which mode to select?

PROBE: Is there any other information that you would like us to include in the video?

6. Did you find any words or part of this page/video that were confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

Record observations and comments here

Now select the mode you would use for your institution and follow the instructions. Remember to think out loud as you complete the task.

Record participant’s comments and think aloud: Probe as necessary while participant completes the task and thinks aloud.

Interviewer: Go to the set of probes appropriate for the mode selected.

Student Records – Excel Mode (Excel Mode Group Only)

Please take a look at the excel mode page.

Question/Prompt

Response

1. Based on what you see here, please the steps necessary to complete the data request.

2. How likely are you to watch the video on this page?

3. Please watch the video.

PROBE: Is the video helpful to you?

PROBE: Is there any other information that you would like us to include in the video?

 Yes

 No

4. Did you find any words or part of this page/video that were confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

[Ask the participant to download the Excel template, save the file on their computer, and open the file.] Please remember to think out loud as you complete the task.

Question/Prompt

Response

Record participant’s comments and think aloud:

Probe as necessary while participant completes the task and thinks aloud.

1. Note if participant clicks on:

 Video  Instructions  Item Codebook  Show Errors  Download error list

 Review Data (in Excel or separate tab)

 Finalize all Student Record Information

2. Does the participant experience any usability issues?  Yes  No ___________________________________________________________________________________________________________________________

3. Check if participant displays difficulty that is:

 construct – relevant (content)

 construct – irrelevant (e.g. format, instructions, usability, etc.)

__________________________________________________________________________________________________________________________

4. Did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

Ask the participant to click on the Instructions tab.

5. Did you review the instructions tab when you completed NPSAS:18-AC?

6. Did you find any words or part of the instructions tab that was confusing?

7. Did you use the “Tips” or “What’s New” documentation when you completed NPSAS:18-AC? Was it helpful to you?

Ask the participant to click on the Codebook tab.

8. Did you review the codebook tab when you completed NPSAS:18-AC?

9. Did you find any words or part of the codebook tab that was confusing?

10. Is there anything that we could change to make the codebook more helpful to you?

Ask the participant to click on the Student Information tab and begin entering data.

Note if the participant uses the “Help” icons (red triangles in corner of some cells).

11. I noticed you <did, did not> use the help icons. <Why or Why not>?

IF DID: How did you know to use them?

IF DID NOT: What would need to change to make it more likely that you might use them?

 Did use help icon

 Did not use help icon

12. In the Student Information tab, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

13. Students are sorted by last name/first name. How easy or difficult is it for you to provide data with the students sorted this way? Would you prefer that they were sorted differently?

14. In the Enrollment section, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

15. In your institution, who (or what title/role) typically completes the Budget information tab?

PROBE: How easy or difficult do you think it would be for multiple people to complete different sections of this file?

16. Does your institution create individualized budgets for all students?

IF YES: How easy or difficult is it to complete the Budget section?

IF NO: How easy or difficult is it to create an estimated cost of attendance?

 Yes

 No

17. In your institution, who (or what title/role) typically completes the Financial Aid information tab?

18. In the Financial Aid section, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

If not addressed in the think-aloud, ASK:

19. How easy or difficult was it for you to navigate from section to section?

If not addressed in the think-aloud, ASK:

20. When your institution participated in NPSAS:18-AC, did you need input from other offices at your institution?

IF YES: How did that go?

How long did it take you to complete this step?

If not addressed in the think-aloud, ASK:

21. Is there a better way to group data elements together (instead of the four sections in the current file)?

If not addressed in the think-aloud, ASK:

How easy or difficult is it for you to tell which data elements are most critical?

Ask the participant to return to the PDP and upload the sample Excel template.

21. How easy or difficult was it to upload the file?

21. How easy or difficult is it to understand the error messages?

PROBE: What do you think [X] error means? How would you go about resolving the error?

Step 4: Review Data

21. Did you find any words or parts of this section confusing?

IF YES: What was confusing? What can we do to make it less confusing?

If not addressed in the think-aloud, ASK:

21. I noticed you <did/did not> review the data. <Why/why not>?

IF DID: How helpful was the review output?

IF DID NOT: What would need to be changed to make it more likely that you would review the data?

Record observations and comments here

Student Records – CSV Mode (CSV Mode Group Only)

Please take a look at the CSV mode page.

Question/Prompt

Response

1. Based on what you see here, please the steps necessary to complete the data request.

2. How likely are you to watch the video on this page?

3. Please watch the video.

PROBE: Is the video helpful to you?

PROBE: Is there any other information that you would like us to include in the video?

 Yes

 No

4. Did you find any words or part of this page/video that were confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

5. How familiar are you with using CSV files? Would you say you are Very Familiar, Familiar, Unfamiliar, or Very Unfamiliar?

 Very Familiar  Familiar  Unfamiliar  Very Unfamiliar

6. Would you create the CSV files yourself, or would you work with someone else at your institution to create them?

[Ask the participant to download the CSV file specifications and open the file.] Please remember to think out loud as you complete the data entry.

Question/Prompt

Response

Record participant’s comments:

Probe as necessary.

1. Note if participant clicks on:

 Video  Show Errors  Download error list

 Review Data (in Excel or separate tab)

 Finalize all Student Record Information

2. Does the participant experience any usability issues?  Yes  No ________________________________________________________________________________________________________________________

3. Check if participant displays difficulty that is:

 construct – relevant (content)

 construct – irrelevant (e.g. format, instructions, usability, etc.)

________________________________________________________________________________________________________________________

Ask the participant to click on the Instructions tab.

7. I noticed you <did, did not> review the <Tips and/or What’s New> sections of the instructions. <Why, why not>?

 Tips

 What’s New

7. Did you use the “Tips” or “What’s New” documentation when you completed NPSAS:18-AC? Was it helpful to you?

9. Did you find any words or part of the “Instructions” tab (CSV Specs) that were confusing?

IF YES: What did you find confusing? What could we do to make the instructions less confusing?

 Yes

 No

Ask the participant to click on any of the four data sections tabs and review the file layout specifications.

In your own words, can you describe each of the columns in the file layout?

9. Did you find any words or part of the specifications confusing?

IF YES: What did you find confusing and how can we make it better?

 Yes

 No

If not addressed in the think-aloud, ASK:

9. What does the column “valid values” mean?

If not addressed in the think-aloud, ASK:

9. Now that you have reviewed the instructions and the file specifications, could you tell us about the process of creating CSV files at your institution for NPSAS:18-AC?

9. Were there any data elements that you could not provide? If so, how did you handle that when creating your files?

9. How easy or difficult is it for you to tell which data elements are most critical?

Ask the participant to return to the PDP and upload the sample Excel template.

16. How easy or difficult was it to upload the file?

If not addressed in the think-aloud, ASK:

17. What do you think “View upload history” does?

PROBE: What information does it display?

PROBE: Is any information displayed in this table confusing?

IF YES: What did you find confusing and how can we make it better?

17. How easy or difficult is it to understand the error messages?

PROBE: What do you think [X] error means? How would you go about resolving the error?

Step 4: Review Data

17. Did you find any words or parts of this section confusing?

IF YES: What was confusing? What can we do to make it less confusing?

If not addressed in the think-aloud, ASK:

17. I noticed you <did, did not> review the data. <Why, Why not>?

IF DID: How helpful was the review output?

IF DID NOT: What would need to be changed to make it more likely that you would review the data?

If not addressed in the think-aloud, ASK:

17. How easy or difficult was it for you to navigate from page to page?

If not addressed in the think-aloud, ASK:

17. How easy or difficult was it to upload your completed files?

If not addressed in the think-aloud, ASK:

23. When your institution participated in NPSAS:18-AC, did you need input from other offices at your institution?

IF YES: How did that go?

How long did it take you to complete this step?

Record observations and comments here

Student Records – Web Mode (Web Mode Group Only)

Please take a look at the web mode page.

Question/Prompt

Response

1. Based on what you see here, please describe what you need to do on this page.

2. How likely are you to watch the video on this page?

3. Please watch the video.

PROBE: Is the video helpful to you?

PROBE: Is there any other information that you would like us to include in the video?

 Yes

 No

4. Did you find any words or part of this page/video that were confusing?

IF YES: What did you find confusing? What could we do to make the page less confusing?

 Yes

 No

Now, go ahead and begin entering your data using the mode you selected (Provide data set to participant). Remember to think out loud as you work on this task.

Question/Prompt

Response

Record participant’s comments and think aloud:

Probe as necessary while participant completes the task and thinks aloud.

Note and probe if the participant views the video.

______________________________________________________________________________________________________________________________________________________________________________________________________

1. Note if participant clicks on:

 Help Icons  Video

2. When participant finalizes a section, do they “check for errors?”

 Yes  No

3. Does the participant experience any usability issues?  Yes  No ___________________________________________________________________________________________________________________________

4. Check if participant displays difficulty that is:

 construct – relevant (content)

 construct – irrelevant (e.g. format, instructions, usability, etc.)

_________________________________________

_________________________________________

_________________________________________

Note whether the participant clicks on the student name or if they complete the page by section.

5. I noticed you entered the information <by student, by section>, why did you make that choice?

 By Student

 By Section

If not addressed in the think-aloud, ASK:

6. In the General Student section, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

If not addressed in the think-aloud, ASK:

7. In the Enrollment section, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

8. In your institution, who (or what title/role) typically completes the Budget information section?

9. In your institution, who (or what title/role) typically completes the Financial Aid information section?

10. In the Financial Aid section, did you find any words or part of this page that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

If not addressed in the think-aloud, ASK:

11. How easy or difficult was it for you to navigate from page to page?

If not addressed in the think-aloud, ASK:

12. I noticed you <did, did not> click on the “Help” icons. <Why, Why not>?

13. At the bottom of each of the 4 sections are two “Finalize” buttons. What do you think would happen if you clicked on these buttons?

PROBE: Please explain, in your own words, the purpose of each button.

PROBE: What, if anything, would make it easier to understand these buttons?

If not addressed in the think-aloud, ASK:

14. How easy or difficult was it for you to navigate from page to page?

If not addressed in the think-aloud, ASK:

15. When your institution participated in NPSAS:18-AC, did you need input from other offices at your institution?

IF YES: How did that go?

How long did it take you to complete this step?

Record observations and comments here

Debriefing (All groups)

Question/Prompt	Response
Before we finish, I’d like to hear any other thoughts you have about the NPSAS Study and PDP Website we worked on today.
Are there any important resources/instructions missing that you think should be included?
On a scale of 1 to 5, where 1 is not easy at all and 5 is the easiest task ever, how easy was this task to complete? PROBE: What would it take to improve your score?
Is there anything else that you think could make the PDP Website clearer or easier to use?
Is there anything else you would like to tell me about working on the PDP Website?
If contacted by NCES about providing the requested data, how likely are you to do so? Would you say Very Likely, Somewhat Likely, Somewhat Unlikely, or Not at all Likely? IF Unlikely: Why not?	 Very Likely  Somewhat Likely  Somewhat Unlikely  Not At All Likely

Thank participant for his/her time and provide instructions for gift card, as appropriate.

1 All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Though sessions are audio recorded for analysis, no personally identifying information will be recorded. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).