Cognitive interviews/respondent debriefing protocol
Locate up to 10 qualified subjects (answer YES on Q3 or Q5, YES on Q6 of current draft)
“Hi. I’m a researcher from the University of Washington. Do you have a moment?”
[Show flier] “We’re here today doing research on how people use the free Internet computers in the library. Do you use the computers in the library to connect to the Internet?”
[If yes] “Great. As part of our research, we will be administering telephone survey to library users across the counter. But before we do that, we need to validate the survey questions by walking though them with people like you in a face-to-face setting. The interview will take about 60-90 minutes. I have a room to use inside the library. Would you be willing to participate in an interview about how you use the computers in the library?”
[If no] “Ok. We’re looking to talk to people who use the library computers. Thank you for your time.”
Escort subject to interview room.
Introduce the survey.
“Thank you for agreeing to participate in this survey about how you and your family use your public library’s computers. This research is sponsored by the Institute for Museum and Library Services and the Bill & Melinda Gates Foundation and conducted by the University of Washington. Your responses are confidential and will help us evaluate and improve library computer services all over the country.
“During this interview, we will also be testing the survey to make sure that when we call other people on the phone they will understand and be able to answer the questions. I am going to record our conversation so that I can fully pay attention to what you say. This should take no more than 1½ hours. Do you have any questions before we get started?”
Explain the role of the respondent and interviewer
“First, I’m going to read the survey questions to you and want you to answer just as you normally would if you got a phone call or were approached at the mall for a survey. If I ask any questions that you are uncertain about or that require you to think, please let me know. After we get through the survey, I’m going to go back over some of the questions and ask you to talk about your answers.
Read each survey question to the subject and record the answers.
If respondent appears to be having a difficult time responding to specific questions, probe with think-aloud questions. For example, “You seem to be having a hard time answering this question. Can you tell me what you’re thinking about as you try to answer?”
Note major difficulties for follow-up at the end of the debriefing section. Minor difficulties will be accounted for in the analysis of the interview transcripts.
Introduce follow-up questions.
“Thanks for answering those questions. Now I’m going to go back over some of them and ask you to talk about some of your answers. I’m not ‘testing’ you about your answers; rather, I want to make sure that the survey is as clear and easy to answer as possible. So, please feel free to be critical!
Probe specific target questions.
Screening questions. The main purpose of the probes in this section is to ensure that the respondent can differentiate between public library computer terminals that access the Internet and those that only access the library’s catalog or other resources like electronic magazines or reference books. Respondents will be asked to rephrase questions or define terms in their own words and to provide examples of the targeted behavior from their own experience. For example, in follow-up to a positive response to Q4, “Have you used a computer in the public library to access library resources, such as looking up books or placing holds, or to use online resources available through the library’s website like digital articles or books?” respondents will be asked what kinds of resources they used the last time they accessed library resources through a computer in the library.
Questions in this section will also test information retrieval and response category selection, where appropriate. In follow-up to positive responses, interviewers will ask how the respondent recalled the frequency of their use of library resources, what timeframe they were thinking of, and how difficult it was for them to choose from among the response choices provided in the survey.
Q3: One of the first questions I asked you was “Have you visited a public library or bookmobile in the last year?”
Can you tell me what you think the term visited meant?
When I said in the last year were you thinking of the past 12 months from today, or all of 2008, or something else?
Q4 [If YES]: When I asked, “Have you used a computer in the public library to access library resources, such as looking up books or placing holds, or to use online resources available through the library’s website like digital articles or books?” you said that you had.
When was the last time you used a computer in the public library to access library resources?
Can you tell me what kinds of resources you used?
Q5 [If YES]: You said that you used a computer in the public library to access the Internet [insert frequency from Q5.1] last year.
Can you tell me what kinds of things you did on the Internet the last time you visited the library to use the computer?
Thinking about the library you used, can you tell me if there were different computers for using the Internet than for accessing library resources? How did you know the difference?
Q6: I asked you if you had used the Internet from outside the library to access library resources.
In your own words, what does it mean to access library resources using the Internet?
[If YES] Can you tell me about the last time you accessed library resources through the Internet?
Q7: I asked if you had connected to the Internet using a public library wireless service.
Can you tell me what wireless service means to you?
How would you go about connecting to it?
General use questions (U1-U5). These questions are to gather general information about library computer use, with particular attention to issues important to library researchers and policy makers. Probing questions for this section will focus on the respondents’ comprehension of the survey questions and concepts. For example, in follow-up to U4 which is intended to gauge the extent of Lay Information Mediator Behavior (LIMB) respondents will be asked to rephrase the question and to provide examples of how they have used library computers on behalf of others.
Debriefing questions for U5.1.1 will also be used to assess respondent comfort with answering a potentially sensitive question about parenting behavior.
U1: In the next section of the survey, I asked if you had used a public library computer or Internet connection while travelling.
In your own words, can you repeat the question?
U2: When I asked if you had participated in any technology-related training or classes at your public library, what kinds of training or classes came to mind when you were answering?
[If YES] What was the last technology-related training or class you took at the library?
U3: I asked if library staff or volunteers had helped you use a computer at the public library.
How did you feel about answering this question? [e.g. was it embarrassing?]
U4: Next I asked you “Have you used a public library computer or Internet connection on behalf or someone else in the past year?” Can you rephrase that question for me?
[If answered yes to LIMB] After you told me that you had used the library computers for someone else, I asked you how many people you had done this for. Can you tell me how you remembered the number of people you had used the computers for? [e.g. are these people who you regularly use the library’s computers for?]
U5.1.1: [If yes U5.1] When you told me that you have children under 18 in your household, I asked if they use the library computers and you said they did. Next I asked if you allowed them to use the library computers without your supervision. Can you tell me what you “supervision” means to you?
How did you feel about answering that question?
Domain screening questions (M1, C1, B1, H1, S1, G1, V1, D1). Probing questions for this section will focus on the respondents’ interpretation of the scope of the domain. Each domain screening question will be revisited with respondents being asked to rephrase the question and give examples of the types of activities they associate with key words in the survey questions.
Debriefing in follow-up to negative responses on domain screening questions will probe for respondent sensitivity and adequacy of response codes for interviewers.
“I asked you a number of questions about specific ways you might have used the library’s free computers. I’m going to go back to some of those questions now.”
M1: I asked if you had used the public library computer for employment or career purposes and you said [insert answer]. Can you tell me what you kinds of things you might do using a public library computer that are related to employment or career purposes?
[If answer was no, probe] When you answered that you hadn’t used the library computers for employment purposes, I asked you if there was a specific reason why you didn’t use the library for these purposes. Was that question difficult or uncomfortable to answer? Why?
C1: When I asked you if you had used the library computers for commercial needs or money management, I gave you the following examples: banking, investments, or buying something. What other kinds of activities related to commercial needs or money management come to mind when I ask that question?
B1: I asked you is you had used a public library computer to start or manage a business. Can you tell me the kinds of things people might do on the public library computers to start or manage a business?
H1: Next, I asked about whether you had used the library computers for health or wellness needs. Can you give me a few examples of health and wellness needs that you think people might use the library computers for?
S1: Can you tell me which of the following you would consider social or entertainment purposes for using the library computers? [If these activities aren’t social or entertainment purposes, ask “What kind of use do you think XXXX is?”]
check email
find a support group
read the news
promote your business
find a date
play a game
research your genealogy
maintain a blog
get a fishing permit
learn how to do something new?
G1: When I asked you if you had used the library computers for getting government or legal information or accessing government services, one of the examples I gave was “learning about regulations.” Can you tell me what kinds of regulations someone might use the library computers to learn about?
Can you give me some other examples of how people might use the library computers for getting government of legal information?
What types of government services do you think people might access through a library computer?
V1: The next question I asked you concerned using the library computers for participating in community life or learning about political or social causes. Can you tell me in your own words what participating in community life means?
D1: The last question I asked you about specific ways people use the library computers was about education or learning. Can you give me some examples of how people might use the library computers for education or learning?
If I told you that I had used the library computer to learn how to arrange flowers, would you consider that an educational use?
[If no] What type of use do you think learning about flower arranging is?
What if I told you I used the library computer to watch a video about how to use a word processing program? Would you consider that an educational purpose?
Can you tell me what you were thinking about when you were deciding whether flower arranging or watching a video about word processing are educational purposes for using the library’s computers?
Open-ended questions (E1-E5). debriefing questions for this section are to gauge the respondents’ comprehension of the survey questions and their reaction to open-ended and scale questions. For example, in follow-up to E2, respondents will be asked to describe in their own words what regular access means.
E1: In the next section, I asked you if you had used the library’s computers for any other purposes than the ones we had already asked you about. When I asked you that, were you able to remember the other purposes I had already asked you about?
[If provided additional uses] Besides the specific purposes I asked you about, you told me that you used the library computers for [read all]. Are there any other ways that you use the library computers that you can think of now?
E2: I asked you if you currently have regular access to a computer for your personal use somewhere other than the public library. In your own words, can you tell me what it means to have regular access to a computer?
E3: When I asked you how important the library’s computers and Internet connection are to you, I gave you 5 choices: very important, important, moderately important, of little importance, and unimportant. You answered [insert answer]. Can you tell me how you decided on your answer? What makes the library’s computers [insert answer] to you rather than [insert next lower answer]?
E4: Next I asked how satisfied you are with the access to computers and the Internet at the public library and gave you these 5 choices: very satisfied, satisfied, neither satisfied nor dissatisfied, dissatisfied, and very dissatisfied. You answered the you are [insert answer] with the library’s computers and Internet connection. Can you tell me what specific kinds of things would make you feel more satisfied about the library’s computers and Internet connection ?
General debriefing questions. Respondents will be asked to provide general feedback on their reaction to the survey, including identifying questions they felt uncomfortable with or had a difficult time answering and revisiting questions they refused or had major difficulty answering.
“Okay. I’m all done with the questions on my list. Before we finish, though, do you have any general feedback on the survey? For example, were there any questions that you felt uncomfortable with or had a hard time knowing how to answer?”
Close interview.
Thank respondent for participation.
Pay incentive.
Revised February 3, 2009 |
| File Type | application/msword |
| File Title | Cognitive interviews/respondent debriefing protocol |
| Author | Samantha Becker |
| Last Modified By | Thorstein Veblen IV |
| File Modified | 2009-02-19 |
| File Created | 2009-02-19 |