2019-20 National Postsecondary Student Aid Study (NPSAS:20) Pretesting

National Center for Education Statistics (NCES)

Attachment VII

Student Focus Group/Audience Assessment Protocol

2019-20 National Postsecondary Student Aid Study (NPSAS:20) Pretesting

OMB# 1850-0803 v. 243

December 2018

Contents

Section I: Introduction (5-10 Minutes) 3

Section II: Contacting Survey Respondents (35 Minutes) 6

Section III: Invitation Letter and Reminders Messages (35 Minutes) 7

Section IV: E-Mail Header (15 Minutes) 8

Section V: Conclusion (2 Minutes) 8

Focus Group Moderator’s Guide

MODERATOR: THIS IS A GUIDE TO HELP YOU LEAD THE DISCUSSION. YOU CAN MODIFY AND ADAPT THESE QUESTIONS TO MEET SITUATIONAL NEEDS DURING EACH SESSION.

Section I: Introduction (5-10 Minutes)

Good [morning/afternoon]. My name is _______________ and I work for StratComm, a research company located in Natick, Massachusetts. We are carrying out a comprehensive research study on behalf of the National Center for Educational Statistics, part of the U.S. Department of Education.

This group discussion will focus on materials we send to people when we ask them to participate in a survey. We want to come up with the best strategies that will attract people like you to become survey respondents. I will introduce a topic and may ask you to pull out one the materials that was included in your packet. I will ask a series of questions and allow time for each of you will enter your responses in to the Illumination Lab System (I-lab portal). Once all responses are provided, I may ask follow-up questions to the group.

Your views and experiences are important, so we would like you to tell us what you think and feel about the contacting material and content that we share with you. It is important that you provide us with your honest opinions and that you understand that we will not make any judgments of you for your opinions. Also, as you share your personal opinions and experiences, it is not necessary for everyone to agree with each other. Please remember that there are no right or wrong answers to the questions I will be asking. Everyone’s input is equally important and helpful. We are interested in all your ideas, comments, and suggestions.

You should have received a packet in the mail that includes a copy of the consent form that explains your participation in the study. You have already acknowledged that you read and understood the information provided on the consent form and you have agreed to participate in today’s session. However, if you have changed your mind, please let me know at this time.

The packet that you received in the mail also includes copies of the contacting materials that we will discuss today. Please make sure you have those materials in front of you. You will need them for the discussion that will start in just a few minutes

Do you have any questions before we discuss the Illumination Lab system in more detail?

Instructions: The following points will introduce you to our Illumination Lab System and the approach that we’ll be using for the session:

1.	LOCKSTEP PROCESS: For today's session we'll be using a concept called 'lockstep' that allows everyone to be on the same electronic page at the same time. We'll move you to each activity on the agenda and will let you know when we are making the transition to the activity.
2.	CONFERENCE CALL: We'll also be on a simultaneous voice conference call so that you may hear the moderator’s instructions. Normal conference call rules apply. Please mute your phones to avoid background noise (do not place phones on hold...). If you do not have a mute button on your phone you may use *6 on your phone keypad to mute your individual line and *6 again on your keypad to unmute your line.
3.	REFRESH/RECONNECT: The system is enabled on the Internet and is generally very stable. If you do encounter a problem on the session, try to REFRESH your browser first. If not resolved, simply RECONNECT by going back to the URL www.gowiq.com/WIQ then enter your userid, and password, rejoin the session and you'll be connected to the current activity.
4.	HELP DESK: We will also have a help desk active during the entire session should you have any issues that can't be immediately resolved. The help desk can be reached at 515.739.1033, confirmation number 112410#.
5.	TECHNICAL NOTE: We would ask that you use a standard browser for the session today (Internet Explorer or Google Chrome for a PC and Safari, Chrome or FireFox for a MAC). If you are using Internet Explorer version 10 or 11 (IE10/IE11), please add the base URL www.gowiq.com to your compatibility view list.
6.	SESSION CONTEXT: Your responses to today's session will be used only to improve the survey and we want you to be as candid as you can. There are no wrong answers – just different points of view. Say what you are thinking. As you know, we will be using your input to help recruit students like you to participate in our future survey. Let's get started.

Now, let’s start with some questions about you (others will not be able to see your responses):

1. What is your sex?

• Male

• Female

• Prefer not to answer

2. What is your age?

• Under 18  THANK AND TERMINATE (Ineligible)

• 18-24

• 25-29

• 30-34

• 35-39

• 40-49

• 50-59

• 60 or older

3. Are you of either Hispanic or Latino origin?

• Yes

• No

• Prefer not to Answer

4. What is your race? (Select all that apply)

• American Indian or Alaska Native

• Asian

• Black or African American

• Native Hawaiian or Other Pacific Islander

• White

• Prefer not to answer

5. What is the name of the college, university, or trade school you have attended at any time between July 1, 2018 and today? (If more than one: What is the name of the school you consider to be your primary school?)

   [NPSAS SCHOOL]

6. Which of the following are you working on at [NPSAS SCHOOL] in the 2018-2019 academic year (July 1, 2018 - June 30, 2019)?

Degrees

• Associate’s degree (usually a 2-year degree)

• Bachelor’s degree (usually a 4-year degree)

• Master's degree

• Doctoral degree--research/scholarship (for example, PhD, EdD, etc.)

• Doctoral degree--professional practice (including: chiropractic, dentistry, law, medicine, optometry, pharmacy, podiatry, or veterinary medicine)

• Doctoral - other

Certificates

• Undergraduate certificate or diploma (usually less than 2 years), including those leading to a certification or license (example: cosmetology)

• Post-baccalaureate certificate

• Post-master's certificate

Classes only (not in a degree program)

• Undergraduate level classes

• Graduate level classes

7. What is your current level at [NPSAS SCHOOL]?

• First year / Freshman

• o Second year / Sophomore (Skip to Q9)

• o Third year / Junior (Skip to Q9)

• Fourth year / Senior (Skip to Q9)

8. Was [NPSAS SCHOOL] the first college, university, or trade school you attended after completing your high school requirements?

• Yes

• No

9. Other than the degree or certificate you are working on at [NPSAS SCHOOL], have you earned any degrees or certificates since high school? (Only include degrees or certificates earned through a college, university, or trade school.)

• Yes

• No

Section II: Contacting Survey Respondents (35 Minutes)

Let’s begin our discussion. I’d like to start off by learning a bit about how you communicate (5 minutes).

1. Let’s brainstorm all the different ways you can communicate with your friends, family, businesses and the official offices (university offices, DMV, your local town or county, etc.).

2. [BACK ROOM CATEGORIZATION] Now for each of the following types of communication, how often do you use/check them?

   1. Multiple times a day

   2. Once a day

   3. Multiple times a week

   4. Once a week

   5. A few times a month

   6. Once a month or less often

3. What is your preferred method of communication? Why? What about that method makes it the most convenient for you?

4. What is your preferred method of communication for official offices such as your school’s administration, the DMV, or your local town government? Is it the same or different as your overall preferred method of communication? Why?

Now, as I mentioned, we will be discussing some recruitment materials for a web survey. We will refer to materials that were mailed to you ahead of this session. Please refer to those materials as we move through the next set of questions. I will also show them on the screen to make sure we all stay together.

Imagine that you were selected to take part in the web survey conducted by the National Center for Education Statistics (NCES). This means that you received an invitation letter and/or a reminder postcard in the mail, addressed to you, informing you that you have been selected to participate in the study and providing you with log in information for the web survey.

Now, let’s talk in general about the best way to remind you about a survey you did not complete when you received the original invitation (15 min):

1. What is the best method to remind you about a survey?

   1. [IF NEEDED, “For example, by email, mail, or phone?]

2. Do you consider multiple contacts from a survey to be helpful reminders to complete the survey or not?

   1. How many contacts would you consider to be too many?

   2. Which method of contact would be the most helpful as a reminder?

   3. Which method of contact would be the most intrusive as a reminder?

   4. IF NOT ANSWERED IN QUESTION 5, would you consider a text message from a survey to be intrusive? Why or why not?

3. Do you answer calls from an unrecognized phone number or not?

   1. Would it make a difference if it is a local number or not?

   2. [IF AT LEAST ONE PERSON SAYS YES FOLLOW-UP WITH, “What are your criteria for answering unrecognized calls? Would you pick up if you were expecting a certain call, or is it based on the area code you see, or do you answer all unrecognized phone numbers?”]

4. How often do you check voicemails left on your phone?

On the screen there are 4 reminder postcards that you have also received in the mail. Please look at them carefully and consider which ones you are most likely to act on and why.

Now, let’s talk about the postcards (15 min):

1. Which of the 4 postcards do you like best and why?

2. Which one is more attention-getting? What specifically catches your attention?

3. What are some features of this postcard that may prompt you to take the survey?

4. Are there any features that are missing, but you would find attractive and necessary to prompt you to open a postcard?

Section III: Invitation Letter and Reminders Messages (35 Minutes)

Here are a few examples of statements we use in our invitation and reminder letters and emails [PROVIDE EXAMPLE STATEMENTS TO EACH PARTICIPANT]. Please take a minute to read over these statements. We will discuss each section separately, but as you read them, think about what makes a particular statement(s) stand out from the rest. Let us begin with the section called Introduction Sentences:

1. Were there any statements that stood out to you and in what way?

2. Were there any statements that motivated you to take the survey and why?

3. Were there any statements that would turn you off from taking the survey and why?

4. [IF NEEDED PROBE ON:

   1. How about statements on the purpose of the survey or how the data will be used?

   2. And statements about how long the survey will take?

   3. What about the incentive being offered?

   4. Would statements about confidentiality procedures turn you off from taking the survey?]

[REPEAT QUESTIONS 1-3 FOR EACH SUBSECTION OF THE PROVIDED MATERIALS]

Some letters were sent out with the attached leaflet – NPSAS in the News:

5. What is your overall impression of this leaflet?

6. Would it influence your decision to participate in the survey or not? Why?

Now, here are three letter examples we have used in past studies [SHOW 3 EXAMPLE LETTERS TO EACH PARTICIPANT]. Please take a minute to look at these letters; don’t worry about reading them.

Now let’s talk about them:

7. What do you like about these letters visually?

8. What do you dislike about these letters visually?

9. There is a QR code on these letters…

   1. Do you know what a QR code is?

   2. Would you know how to use a QR code? What do you expect to happen?

   3. Would you scan the QR code if you received such a letter invitation in the mail?

Section IV: E-Mail Header (15 Minutes)

Now we want to focus on e-mail messages you might receive as reminders to complete the survey assuming you had not previously done so:

1. If you were to receive an email from someone you do not know personally, what elements in the subject line would make you decide to open the email?

   1. [IF NEEDED PROBE ON:

      1. What about the subject line being personalized (i.e., including your name)?

      2. And the inclusion of an incentive (i.e., dollar amount)?

      3. How about expressions of excitement (e.g., exclamation points or emojis) in the subject line?]

   2. What makes an email look like a scam, junk, or something you would delete without opening?

Here are a few example subject lines we have used in past studies [SHOW EXAMPLES TO EACH RESPONDENT]. Please have a look and let’s talk about these examples:

2. Which email would you open and why?

   1. Are there any subject lines that would make you immediately delete the email without opening it? Why?

Section V: Conclusion (2 Minutes)

Are there any final comments or questions?

I want to thank you all for joining our focus group and sharing your opinions and suggestions. The $50 check or gift card payment we promised will be provided to you in the next few days. If you do not receive it in that timeframe, please contact Taylor Groo at 508-907-7000 x238 or [email protected].