Protocol for 2015 National Survey of College Graduates
Participant ID #: |_N__|_S__|_C__|_G__|___| ___|___|___|
Interview Date: |__|__| / |__|__| / |__|__|__|__| (mm/dd/yyyy)
Interviewer initials: |__|__|
Start Time: ____________ AM / PM End Time: ____________ AM / PM
Section 1: Interviewer: Read/ Paraphrase the following text:
Greeting: Hello. My name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study.
What: In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys.
How: [FOR PAPER TESTING: SHOW RESPONDENT NSCG FORM] This is a questionnaire called the National Survey of College Graduates. If this were the real survey, you would receive this questionnaire in the mail. In a couple of minutes I am going to ask you to fill out this survey just as you would if you had received the questionnaire at your home but with one major difference.
[FOR CATI TESTING]: I want you to answer the questions exactly the way you would if an interviewer had come to your home for an official interview, but with one major difference.
Think Aloud: I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.
Practice: Let’s do a practice question before we start: How many windows are in your home?
Interruptions: [FOR PAPER TESTING]: I am going stop you at a couple of points in the questionnaire and ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.
[FOR CATI]: From time to time, I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.
Do you have any questions before we begin?
Confidentiality: Our session today is completely confidential. Any names you provide will never be used in our reports. Your participation in this study is completely voluntary and you can decline to answer any particular question.
Recording: So I don’t have to rely on my memory later on, I’d like to tape record this interview. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes. Is that ok with you?
** HAVE PARTICIPANT SIGN CONSENT FORM**
**Begin: **TURN ON TAPE RECORDER**
OK, let’s begin. Please remember to think aloud as you answer the questions.
INTERVIEWER INSTRUCTIONS: [FOR PAPER TESTING:] AS THE RESPONDENT FILLS OUT THE FORM, RECORD OBSERVATIONS ASSOCIATED WITH EACH QUESTION. AFTER THE RESPONDENT HAS COMPLETED EACH SECTION, ASK THE PROBES ASSOCIATED WITH EACH QUESTION AND SECTION.
[FOR CATI TESTING:] NO NEED TO TAKE NOTES, BUT AS YOU ASK THE QUESTIONS, RECORD ANSWERS WHEN REQUESTED IN THE CHECK BOXES SO YOU WILL NOT NEED TO SCROLL BACK THROUGH THE QUESTIONNAIRE.
[FOR WEB TESTING ONLY]:
Greeting: Hello. My name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study.
What: In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys.
Confidentiality: Our session today is completely confidential. Any names you provide will never be used in our reports. Your participation in this study is completely voluntary and you can decline to answer any particular question.
Recording: So I don’t have to rely on my memory later on, I’d like to video record this interview. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes. Is that ok with you?
** HAVE PARTICIPANT SIGN CONSENT FORM**
**Begin: **TURN ON Media Recorder**
Today, we will be working on a few projects together.
1: First, we’ll collect some information about you and your computer and Internet experience.
2: Second you will help us evaluate the design of the online National Survey of College Graduates by completing it online. I’ll ask you some satisfaction questions and debriefing questions about the survey as well.
[Eye tracking]
When you complete the college graduate survey, we are also going to record where you look on the screen. In a moment, we will do a simple task that will allow the computer to find your eyes. I am going to have you position yourself in front of the screen so that you can see your nose in the reflection at the bottom of the monitor. To calibrate your eyes, please follow the dot across the screen with your eyes.
[Calibrate the participants’ eyes]
Okay, you can look away from the screen. Here is a short questionnaire so you can tell us about yourself. [Hand the demographic questionnaire to participant and wait in the room while they fill it out and return it to you]
Ok great. Now I’d like you to complete an online form about your computer and Internet usage. I’ll leave the room for this part of the session, but we can communicate with the microphones.
[Bring up background questionnaire, enter their participant number and select begin, leave room]
Ok great. Now you will complete the college graduate survey. I want you to answer the questions exactly the way you would if you got on your computer at home, but with one major difference.
I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.
Practice: Let’s do a practice question before we start: How many windows are in your home?
From time to time I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.
Do you have any questions before we begin?
Ok great. For the survey, you will need a username and password. I’ve left your user name and password on the piece of paper on the table. Please use the highlighted information. You will also have to pretend that your name is the highlighted name on the piece of paper. So, when you come to the screen asking you to confirm this name, please say yes. Other than that, please complete it as you would at home. I would like you to think aloud as you complete the survey. I will remind you to “keep talking” if you get quiet. I will interrupt you during the survey to ask you some questions as you complete it.
If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.
Now, let me bring up the survey for you. [Bring up survey in Tobii]
Ok you can begin. Please remember to think aloud as you answer the questions.
Legal text and introductory information (Paper and Web only):
Observe
[] Respondent read information
Probe: What are your thoughts about the information you just read?
Would you be more or less likely to complete the survey after reading that information?
Why would you be more/less likely to complete the survey?
[] Did not read information
Probe: I noticed you read/did not read that information. Please read it now and let me know what your thoughts after reading it
Would you be more or less likely to complete the survey after reading that information?
Why would you be more/less likely to complete the survey?
A1.
Probe: What were you doing at this time?
Probe: What does this question mean in your own words? What does working mean to you?
If someone was working part time around this time, but not necessarily that week, how should that person answer the question?
A2.
A3.
Probe: Was it easy or difficult to select a response?
Note: Pay attention to any response or comment associated with did not need or want to work.
A4.
A5.
A6.
A7.
Probe: How did you come up with your answer?
Probe: Was it easy or difficult to select a code?
Probe: Was there an answer you thought about but did not select?
A8.
A9.
If R mentions more than one job, Probe: How did you decide which job to choose as your principal employer?
Note: For anyone who is self-employed or an independent contractor, who do they list. Probe to make sure you understand whether they list themselves or their client.
A10.
A11.
Probe: How did you come up with this estimate?
Note: For anyone who is self-employed or an independent contractor, probe to make sure you understand who they were referring to when answering this question.
A12.
A13.
Probe: Tell me more about your employment situation
A14.
A15.
A16.
A17.
A18.
Probe: How did you come up with your answer?
Probe: Was it easy or difficult to select a code?
Probe: Was there an answer you thought about but did not select?
A19.
□ Did the respondent choose the option for “some other field, specify”?
CATI response: ________________
If yes, Probe: Do you think that your response could have fit within any of the provided categories?
Probe: Which field were you thinking about when you gave your answer?
Probe: What does the term “technical expertise” mean to you in this question? Would the term “knowledge and skills” mean the same thing to you?
A20.
Note: For anyone who is self-employed or an independent contractor, probe to make sure you understand the date they provide to this question—is it the date they started the client job or the date they started working for themselves.
A21.
*What does the phrase ‘related to your highest degree’ mean to you in this question?
A22.
Probe: What do the terms pay and promotion mean to you in this question? Do they mean the same thing or are they different?
A23.
A24.
Probe: How did you come up with your answer?
Probe: Was there an answer you thought about but did not select?
Probe: Was there any category you found confusing? Which one?
A25.
A26.
A27.
A28.
Probe: Do any of these items not apply to your job situation?
Probe: Tell me more about how you came up with your answer for each of these items
A29.
A30.
Probe: Can you tell me how you came up with your answer?
A31.
A32.
A33.
□ If answer to A32 is less than 30 hours per week:
Probe: Would your answer have changed if I asked you if you want to work 30 or more hours per week instead of 35 hours?
A34.
A35.
A36.
A37.
A38.
Probe: How did you determine your earned income?
A39.
If yes, Probe: Can you tell me a bit more about your certification or license?
Probe: Please give me some [other] examples of the types of certifications or licenses that would apply to this question.
[] Check this box if they mentioned examples from the instructional text, or referred to the italicized examples
A40.
Probe: Tell me about how you came up with your answer.
Probe: What are they?
Note: See if it is a license in order to operate a business.
Probe: If the instruction [read or point to instruction] was not present, would your answer change?
A41.
A42.
Probe: If the instruction [read or point to instruction] was not present, would your answer change?
A43.
Probe: Can you tell me a bit more about your answer?
Probe: If the instruction [read or point to instruction] was not present, would your answer change?
A44.
Probe: Tell me more about how you came up with your answer.
A45.
A46.
Probe: What does “required by employer” mean to you?
□ Did the respondent choose the option for “required by employer”?
If Yes: Tell me more about how the license is required by your employer.
Probe: Would your answer change if the item said Required or expected by employer.
A47.
A48.
Probe: Tell me what you were thinking as you answered this question?
Probe: Did you have any difficulty answering it?
A49.
Probe: In your own words, what is this question asking?
Probe: What certificate or license were you answering about as you answer the previous questions? [Verify that it is the most recent one if the person has several.]
Tell me about all the certifications and licenses you have, which ones are active? When did you get them? Why did you get them?
[FOR PAPER TESTING: STOP AND ASK R ALL SECTION A PROBES]
B1.
B2.
Probe: What were you thinking about when you were answering this question?
B3.
C1.
C2.
[] Respondent did not choose the option “required or expected by employer”
Probe: What does “required or expected by employer” mean to you?
□ Respondent choose the option for “required or expected by employer”?
Tell me more about how the license is required or expected by your employer.
Would your answer have changed if the category only said “Required by employer”?
C3.
C4.
C5.
C6.
[FOR PAPER TESTING]: STOP AND ASK R ALL SECTIONS B AND C PROBES
D1.
D2.
□ Did
R go to high school in a foreign country?
D3.
D4.
□ Did R seem to read or have trouble with the response categories?
If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?
D5.
D6.
D7.
D8.
D9.
If
D2 = foreign country,
Probe:
Can you tell me more about
that?
D10.
Probe: Can you tell me more about that?
D11.
Probe for part d: What were you thinking about when you were answering this question?
Probe for part d: Do you think minors should be included here?
D12.
Note: observe the order they review the categories and whether they go backwards
Probe: Can you tell me more about your answer?
Probe: Were any of the categories confusing? Which ones?
Probe: What is financial assistance from the Veterans Educational Assistance Act (i.e., the G.I. Bill) mean to you?
Probe: What education were you thinking about in this question?
Probe: Did you include room and board in your response, or just tuition?
D13.
Probe: How did you come up with your answer?
Probe: How difficult was it for you to answer this question?
If $1-10,000 or higher for one degree or more, Probe: Did you include interest in your estimate(s)?
D14.
Probe: Tell me what you were thinking about when you chose your answer.
Note for paper reporters: observe skip sequence
D15.
D16.
Probe: Tell me more about the resources you received.
D17.
If yes, probe: Can you tell me a bit more about your certificate?
D18.
Probe: Can you tell me more about your answer?
Probe: What does the question mean in your own words?
Probe: What are some examples of this type of certificate?
D19.
D20.
Probe: Can you tell me more about your answer?
Probe: What does the question mean in your own words?
Probe: What are some examples of this type of certificate?
D21.
D22.
Probe: Tell me more about your answer and your certificate.
D23.
[] Respondent did not choose the option “required by employer”
Probe: What does “required by employer” mean to you?
Would your answer have changed if the category said “Required or expected by employer”?
□ Respondent choose the option for “required by employer”?
Tell me more about how the license is required by your employer.
Would your answer have changed if the category said “Required or expected by employer”?
D24.
D25.
Probe: Please describe to me all your certificates, when you received them (before or after your undergraduate or graduate degrees), and the school which awarded them.
Probe: What certificate were you answering about as you answer the previous questions? [Verify that it is the most recent one if the person has several.]
□ Did R seem to have trouble with the response categories?
If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?
If applicable, probe about university employer/graduate assistant situation
If not asked after section A, Probe: If you had to pay out of pocket for a certificate but were later reimbursed for your expenses, how do you think you would have answered this question?
[FOR PAPER TESTING: STOP AND ASK R ALL CERTIFICATE PROBES]
D26.
D27.
D28.
D29.
D30.
D31.
Probe: What does “required or expected by employer” mean to you?
□ Did the respondent choose the option for “required or expected by employer”?
If Yes: Tell me more about how the license is required or expected by your employer.
Probe: Would your answer change if the item only said Required by employer.
D32.
[FOR PAPER TESTING: STOP AND ASK R PROBES AND OTHER SPONTANEOUS PROBES]
E1.
E2.
Probe: What does “full-time” work mean to you?
If necessary, Probe: How many hours do you think full-time work is?
E3.
*If not asked after section A, Probe: What does the term “technical expertise” mean to you in this question? Would the term “knowledge and skills” mean the same thing to you?
E4.
E5.
E6.
Observe any difficulty with this question : it does not ask about themselves, but instead about parents.
E7.
E8.
E9.
E10.
E11.
E12.
E13.
Probe: What does this question mean to you?
Probe: about respondent’s history of entrance and exit to U.S.
E14.
E15.
E16.
E17.
E18.
E19.
E20.
E21.
E22.
E23.
If yes, Probe: Can you tell me more about that?
E24.
E25.
E26.
E27.
Probe: How do you feel about providing this information?
E28.
Probe: Why did you choose that answer?
[FOR PAPER TESTING: STOP AND ASK R ALL SECTION E PROBES]
DEBRIEFING:
Overall, would you say the survey questions were easy or difficult to respond to? Which ones? Why?
Do you think there are questions some people would find difficult? Sensitive? Which ones?
Did you think any of the questions were asking the same thing?
(WEB ONLY) Did you notice the italicized instructions in the survey?
Do you have anything else you would like to tell us that you haven’t had a chance to mention yet?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Aleia Y Clark |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |