Gainful Employment Focus Groups 2017

Westat Moderator’s Guide

2017 Gainful Employment Focus Group

For Students Enrolled In Proprietary Schools
and Community Colleges

Revised October 19, 2016

Welcome

Good evening. My name is _______ and I am a researcher at Westat. I will moderate tonight’s focus group. Thank you for taking time to participate in the group. First, let me tell you about the purpose of this study.

Introduction to the Study

Westat, a social science research company in Rockville, MD, is working with the
U.S. Department of Education on a project to help schools provide better information to current and prospective students about their post-high school career and training programs that lead to a certificate or degree. Most of the discussion will focus on reviewing and discussing information contained in a few draft web pages developed by the Department of Education. Schools that offer these certificate or degree programs are required to provide specific information about their programs on their web sites.

Therefore, the Department of Education would like to hear from students, like you, to determine if this information is clear and easy to understand before finalizing the web pages. The information you share will be used by the Department to help students make informed decisions about the programs and schools before and during enrollment.

Focus Group Guidelines

1. There are no right or wrong answers. We value a variety of opinions and experiences.

2. Please talk one at a time.

3. Talk in a voice as loud as mine.

4. Avoid side conversations with your neighbors.

5. I need to hear from everyone; however, you do not have to answer every question.

6. It is OK to piggyback on someone else’s comments.

7. Work for equal “air time” so that no one talks too little or too much.

8. Information shared in the group will remain confidential; the report will only summarize the responses and will not identify you by name or school. Help us protect your confidentiality by only stating your first name and NOT mentioning the name of your school.

9. Please do not share the information discussed with anyone outside of the group until all the data have been collected and the web pages are finalized by the Department of Education, which will not occur until 2017. [Get verbal agreement from everyone.]

10. Staff from the Department of Education and Westat are observing and taking notes on the discussion behind the one-way mirror and the session will be audio recorded.

11. Please turn off your cell phones, etc. or turn them to vibrate. And please, refrain from texting or checking email during the focus group.

Any questions before we begin?

I. Warm-Up and Opening Questions

OK, let’s get started by going around the table and introducing ourselves.

Please give your first name only and tell me:

• What career or vocational program you are currently enrolled in, and

• How long you’ve been enrolled in the program.

Please do not share the name of the institution you are attending or your last name.

Now, let’s begin by talking a bit about how you learned about your current program and school.

1. What type of information did you look for when selecting a program of study?

PROBE IF NOT MENTIONED:

• type of career or vocational program or certificate offered

• cost

• geographic location

• length of program

• reputation of school (probe on how determined?)

• job placement (services, record?)

• other

2. Where or how did you get information about your school or program?

PROBE FOR:

• Website

• Walk in

• Phone call

• Family member, friend, etc.

3. When did you first contact the school you currently attend (i.e., how many months or days before you applied?)

3a. What was your main reason for contacting them (enrollment, cost information, question about financial aid, application process/requirements question [about what?], etc.)?

4. What in particular made you choose the program at the school you currently attend?

Thank you very much. For the remainder of the discussion, we will focus our attention on reviewing draft screen shots of a web page. I will ask you specific questions about each of the items in the screen shots.

For purposes of our discussion, let’s assume that all schools that offer career or training degree or certificate programs would be required to provide some or most of this information on their college web sites.

Please note that we will be using made up examples from one school for our discussion. The information and data used in the example are not real, but resembles what students might see on school websites.

I want to start by briefly displaying what the full web page for a program would look like. If everyone could please take a moment to look at the screen, this is roughly what would appear on the web page.

II. Students Graduating on Time

Please take a moment to look at what is displayed on the screen.

[MODERATOR READS TEXT ALOUD]

1. Please describe, in your own words, what you think these statements mean?

a. IF NOT MENTIONED: What is the difference between the two statements?

2. What, if anything, is confusing about the statements?

3. What, if anything, would you like to change about the information presented to make it easier to understand?

4. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the screen again. You will see a slightly different version.

[MODERATOR READS TEXT ALOUD]

5. What do you think about this information?

a. [IF NEEDED]: Does the additional information make it easier or harder to understand the statement (about 65% of students)? What makes you say that?

PROBE IF NOT MENTIONED:

b. What does “share” mean here?

6. What, if anything, would you change to make the information in the box easier to understand?

Now, please look at the screen again. This time, we have added a graphic.

7. What does this graphic mean to you?

PROBE IF NOT MENTIONED: What do the colors of the figures represent?

8. Show of hands: How many people think this graphic makes it easier to understand the ‘students graduating on time’ section? How many think the graphic makes it harder to understand the section? What makes you say that?

a. How many of you think the section works fine without the graphic? Tell me more about this.

9. What, if anything, would you change about the graphic to make it easier to understand?

III. Program Costs

Now we’d like to talk about the Program Costs section briefly. I’m going to display the whole section, so please take a moment to read it.

1. Looking at the graphic, what does it mean to you?

PROBE IF NOT MENTIONED: Please explain the bar line, “national average.”

PROBE IF NOT MENTIONED: How does the bar line relate to the “below average” arrow?

PROBE IF NOT MENTIONED: Which cost(s) would you say are represented in the graphic? (i.e., individual costs or total cost of attendance)?

2. Show of hands: How many people think this graphic makes it easier to understand the ‘program costs’ section? How many think the graphic makes it harder to understand the section? What makes you say that?

a. How many of you think the section works fine without the graphic? Tell me more about this.

3. What, if anything, would you change about the graphic to make it easier to understand?

IV. Debt and Borrowing

Please take a moment to look at what is displayed on the screen.

Now, let’s focus on each line one by one.

A. Students Borrowing Money

[MODERATOR READS TEXT ALOUD]

1. Please describe, in your own words, what you think this statement means.

2. What, if anything, is confusing about the statement?

3. What do you think about the use of green for the numbers?

4. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the screen again. You will see a slightly different version.

[MODERATOR READS TEXT ALOUD]

5. What is your reaction to this information?

a. Does the additional information make it easier or harder to understand the statement? What makes you say that?

b. PROBE IF NOT MENTIONED: What is the difference between Federal, private, and institutional loans?

c. Is it helpful or not helpful to have this difference described here? Which of these terms, if any, would you like explained or described? Would examples help?

6. How would you use the additional information? What makes you say that?

7. What, if anything, would you change to make the information box text clearer?

Now, please look at the screen again. This time, we have added a graphic.

8. Looking at the graphic, what does it mean to you?

9. Show of hands: How many people think this graphic makes it easier to understand the ‘students borrowing money’ section? How many think the graphic makes it harder to understand the section? What makes you say that?

a. How many of you think the section works fine without the graphic? Tell me more about this.

10. How would you use the information the graphic gives you? What makes you say that?

11. What, if anything, would you change about the graphic to make it easier to understand?

B. Typical Graduate Debt

Now, please look at the next display framed in red on the screen, “The typical graduate leaves with…”

[MODERATOR READS TEXT ALOUD]

12. Please describe, in your own words, what you think this statement means?

a. PROBE IF NOT MENTIONED: What do you think a “typical graduate” means?

13. What, if anything, is confusing about the statement?

14. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the screen again. You will see a slightly different version.

[MODERATOR READS TEXT ALOUD]

15. What is your reaction to this information?

a. Does the additional information make it easier or harder to understand the statement? What makes you say that? [count numbers]

16. Are there any words or phrases that are unclear?

a. PROBE IF NOT MENTIONED: What does the term “median debt” mean? Would a brief definition be helpful here or not?

b. PROBE IF NOT MENTIONED: What do you now think the term “typical student” represents?

c. Which of these terms, if any, would you like explained or described? Would examples help?

17. How would you use the additional information? What makes you say that?

18. What, if anything, would you change to make the information box easier to understand?
    
    
    

Now, please look at the screen again. This time, we have added a graphic.

19. Looking at the graphic, what does it mean to you?

PROBE:

How does the information in this graphic differ from the previous slide? [SHOW ELEMENT WITHOUT GRAPHIC]

20. Show of hands: How many people think this graphic makes it easier to understand the ‘typical graduate debt’ statement? How many think the graphic makes it harder to understand the statement? What makes you say that?

a. How many of you think the statement works fine without the graphic? Tell me more about this.

21. How would you use the information the graphic gives you? What makes you say that?

22. What, if anything, would you change about the graphic to make it easier to understand?

C. Monthly Student Loan Payments

Now, please look at the next highlighted display on the screen, “The typical graduate pays…”

[MODERATOR READS TEXT ALOUD]

23. Please describe, in your own words, what you think this statement means?

PROBE IF NOT MENTIONED: What do you think a “typical graduate” means?

24. What, if anything, is confusing about the statement?

a. What, if anything, would you change to make the statement less confusing?

25. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the screen again. You will see a slightly different version.

[MODERATOR READS TEXT ALOUD]

26. What is your reaction to this information?

a. Does the additional information make it easier or harder to understand the statement? What makes you say that? [count numbers]

27. Are there any words or phrases that are unclear?

PROBE FOR MEDIAN AND 5.0% INTEREST RATE.

PROBE:

Would it be useful to have a definition of the word “median?” Where?

28. How would you use the additional information? What makes you say that?

29. What, if anything, would you change to make the information box easier to understand?

Now, please look at the screen again. This time, we have added a graphic.

30. Looking at the graphic, what does it mean to you?

31. Show of hands: How many people think this graphic makes it easier to understand the ‘typical graduate pays’ statement? How many think the graphic makes it harder to understand the statement? What makes you say that?

a. How many of you think the statement works fine without the graphic? Tell me more about this.

32. How would you use the information the graphic gives you? What makes you say that?

33. What, if anything, would you change about the graphic to make it easier to understand?

D. Earnings

Now, please look at the next highlighted display on the screen, “The typical graduate earns…”

[MODERATOR READS TEXT ALOUD]

34. Please describe, in your own words, what you think this statement means.

PROBE:

a. What do you think ‘typical graduate” means here?

b. [IF NEEDED]: To you, does ‘typical graduate’ mean all graduates or only certain types of graduates?

35. What, if anything, is confusing about the statement?

36. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the screen again. You will see a slightly different version.

[MODERATOR READS TEXT ALOUD]

37. What is your reaction to this information?

a. Does the additional information make it easier or harder to understand the statement? What makes you say that? [count numbers]

38. Are there any words or phrases that are unclear?

PROBE:

a. What does the term “typical graduate” mean? (i.e., see if they notice it’s just for GRADUATES who received FEDERAL AID).

b. Would it be useful to have a definition of the term “typical graduates?” Where?

39. How would you use the additional information? What makes you say that?

40. What, if anything, would you change to make the information easier to understand?

Now, please look at the screen again. This time, we have added a graphic.

41. Please tell me what you think this graphic means?

a. What do you think about the horizontal layout of the graphic?

[IF NEEDED]:

b. What do you like about the appearance of this graphic?

c. What do you dislike about it?

[IF NEEDED]:

d. What, if anything, would you suggest changing about the graphic?

42. Show of hands: How many people think this graphic makes it easier to understand the ‘typical graduate earns’ statement? How many think the graphic makes it harder to understand the statement? What makes you say that?

a. How many of you think the statement works fine without the graphic? Tell me more about this.

V. Licensure Requirements

1. Please describe, in your own words, what you think this statement means.

2. What, if anything, is confusing about the statement?

a. IF NOT MENTIONED: What are licensure requirements?

3. If you clicked on the “i” here [POINT TO IT], what would you expect or want to see?

Now, please look at the new version of that section.

4. What is your reaction to this information?

a. Does the additional information make it easier or harder to understand the statement? What makes you say that?

5. Are there any words or phrases that are unclear?

6. How would you use the additional information? What makes you say that?

7. What, if anything, would you change to make the information easier to understand?

VI. Loan Repayment Rate

Now, we’re going to look at four different ways to talk about students repaying loans. These are presented in no particular order, but we’d like to hear from you about whether they are easy to understand, what should be changed, and which versions you like best.

Please look at the screen for Option #1.

1. What does this statement mean to you?

2. What, if anything, is confusing about the statement?

PROBE for principal [i.e., does not include interest]

a. What, if any, additional information would be helpful to have here? Why?

Now, please look at the Option #2.

3. What does this statement mean to you?

4. What, if anything, is confusing about the statement?

a. What, if any, additional information would be helpful to have here? Why?

Now turn your attention back to the screen for the third option.

5. What does this statement mean to you?

6. What, if anything, is confusing about the statement?

PROBE for original principal balance [i.e., does not include interest]

7. What, if any, additional information would be helpful to have here? Why?

Now let’s look at the fourth option.

8. What does this statement mean to you?

9. What, if anything, is confusing about the statement?

PROBE for very low, loan repayment rate.

10. Is it easier or harder to understand the statement about loan repayment without a number? What makes you say that?

a. Do you want to know how we determined that the repayment rate is low, or is that not important to you? What makes you say that?

11. What, if any, additional information would be helpful to have here? Why?

Now, looking back at the screen, we have placed all four versions together.

12. Which of the four versions do you prefer? [Want to hear from everyone]

13. What is the main reason you prefer that one? [Want to hear from everyone on this question.]

14. Which of the three versions is easiest to understand? What about this version makes it easiest to understand?

15. [IF NOT ALREADY DISCUSSED]: What was confusing about the other versions?

VII. Additional Header Language

Next, we are going to discuss three versions of language that could appear on your program’s website, above the information we have already discussed, if there were issues with the program.

It would appear on the website similar to this [POINT TO THE WARNING LANGUAGE]:

1. What are your first impressions of the page now that this information is showing on it?

2. [IF NEEDED]: What do you think about the way the information is presented here? What do you like about it? What do you dislike about it? [PROBE on the red color, if not mentioned] What do you think about the triangle with an exclamation point in it?

3. [IF NEEDED]: Does the way it’s presented here make you want to read it, or not read it? Why?

Now, we are going to discuss three different versions of the message that could appear on this page.

Please take a moment to read the first option.

4. What is your reaction to this version of the warning language?

   1. What is this text saying about the program?

5. What questions, if any, does this warning raise for you?

   1. What kind of standards do you think it is talking about?

   2. Is it important or not important to you to know what standards the program did not pass? What makes you say that?

6. If this appeared on your program’s website, how would you use the information?

[IF NEEDED]: What affect, if any, would it have on your decision to enroll in the program? [If no effect on decision]: Why not? [If would affect decision]: In what ways?

Now let’s look at a second version of the warning language.

[MODERATOR READS TEXT ALOUD]

7. After reading this text, what do you think it is saying about the program?

8. Is it easy or difficult to understand this warning language?

a. [If not easy], what about the language is confusing?

b. What changes, if any, would make it easier to understand the warning language?

9. If this appeared on your program’s website, how would you use the information?

[IF NEEDED]: What affect, if any, would it have on your decision to enroll in the program? [If no effect on decision]: Why not? [If would affect decision]: In what ways?

Now, please look at these two versions together. Focus on the first sentence of each warning.

Version #2

Version #3

[MODERATOR READS TEXT OF VERSION #1 ALOUD]

10. What is your reaction to the differences in the first sentence between versions 2 and 3?

11. After reading version #3, what do you think it is saying about the program?

12. Which version do you prefer? What are some reasons you prefer this version?

Now, let’s look at all three versions together.

Version #1

Version #2

Version #3

13. Which of the three versions do you prefer? What are some reasons you prefer this version? [Want to hear from everyone on this question.]

14. [IF NEEDED] Which version is easiest to understand right away? Why? [Want to hear from everyone on this question.]

15. Who do you think this message is intended for? What do you think the U.S. Dept of Ed intends students to do, based on this message? What makes you say that?

a. What do you think about adding header that says “U.S. Department of Education Warning:” before this text? [IF NEEDED]: Would that be helpful or not helpful? Why?

b. [IF NEEDED]: Are you interested or not interested in this type of information about your program? What makes you say that?

16. Do you have any other comments or questions about the warning language?

VII. General

Please look at the screen again. Now, we are displaying all of the different topics that we’ve talked about over the course of the past hour or so. This is how it would appear on your college’s webpage.

1. What are your overall impressions of how it looks?

1. Which topics are you most interested in learning about, for your program? Why?

2. How easy or difficult is it to find that information that is most important to you?

PROBE:

3. What information is most important to you?

2. If you were using this website, how likely would you be to click on the “i” buttons for more information?

3. What important information would you want to know about your program?

PROBE:

1. What important information is missing from the example we are viewing?

4. Thinking about all of the different types of information you have seen that will be provided to students, when would it be most useful to view this information?

[MODERATOR NOTE: record answers on flip chart]

PROBE FOR:

• Before enrolling/applying

• Before the first time you take out a loan

• If the program status changes (i.e. if a warning is applied)

• Regularly

• Different time (specify)

False Close [If time]

Thank you so much for being here tonight. Before we conclude, I need to step out for a few minutes to make sure we have everything ready for you.

[Moderator checks with observers for up to two additional questions to pose, time permitting.]

Final Close

[Upon return, moderator will pose questions from observers, if any.]

This concludes our discussion for tonight. Again, thank you so much for taking time out of your very busy lives to be here with us tonight. We have your incentive ready; we just need everyone to sign this form and we will hand you an envelope. Please enjoy the rest of your evening and travel home safely.

32