Cognitive Interviews for New Items to Measure Professional Certifications and Educational Certificates among Adults in the United States

Cognitive Interview Guide
May 20, 2010

Introduction and Warm-Up
•
•
•
•
•

Who I am
What I do
Length
Purpose of interview
Read confidentiality statement:

“The American Institutes for Research is conducting this study for the National Center for
Education Statistics of the U.S. Department of Education. This study is authorized by law under
the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your
responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec.
183). All responses that relate to or describe identifiable characteristics of individuals may be
used only for statistical purposes and may not be disclosed, or used, in identifiable form for any
other purpose, unless otherwise compelled by law."
•
•
•
•

Be candid; no right or wrong answers
Gratuity for your time and opinions
Colleagues listening in on the interview (if any)
As you are answering questions, “think out loud” when responding. That is, if a question is
unclear, tell me and tell me why it is unclear. Tell me what is going through your mind as
you answer a question

1

INTERVIEWER INSTRUCTIONS:
Round 3 interviews will consist of concurrent think-aloud and follow-up probes and retrospective
probing after the certification section and again after the certificate section. The purpose is to let the
interview flow naturally unless the respondent has a problem answering a question. At the end of each
section, retrospective probing will be used to follow-up on other possible problem areas.
We will also ask questions about the screener and cover letter which will be mailed to participants in
advance.
Cover letter and mail survey instructions
First, I’d like to talk to you about the letter and short survey we sent you in the mail. Did you receive
those?
Do you have them with you? Could you get them out?
Let’s look at the letter first.
In your own words, what is this letter asking you to do?
Is it clear who sponsoring this survey?
Was anything in the letter confusing to you?
Did you have any other comments about the letter?
Now, let’s look at the short survey we sent.
Did you fill it out?
How easy or difficult was it for you to fill it out?
Was anything in the short survey confusing for you?
Let’s go over your answers.
What did you put for (Q1, Q2, Q3 etc..)
What do you think is the purpose of the short survey?
Did you look at the FAQs on the back?
What do you think of the cover?
2

Okay, now I’d like to ask you some additional questions. Stop me and let me know if anything I ask you
is confusing, difficult to understand, or just doesn’t sound right.
BEGIN PHONE INTERVIEW from Question IN4.
Survey instructions
Concurrent think-aloud
Allow the respondent to proceed through the questions until they make a comment or ask a question.
Use non-directive probing to inquire about item problems.
Non-directive probes:
What do you think this question is asking?
What does that mean to you?
Tell me more about that.
Retrospective probes
If you feel there was uncertainty about a respondent’s answer, probe retrospectively on those items.
Ask all respondents:
Let’s go back and review some of the answers you just gave me.
Certifications
CNINTRO, CN1. Was it easy or difficult for you to answer this question?
CN2. How did you come up with your most recent certification?
CN3. [Interviewer instruction] Pay attention to respondent behavior here. Do they hesitate? Ask
questions? Make remarks that indicate the question is awkward? If yes, probe.

3

CN4. Was it easy or difficult for you to answer this question? Probe if respondent says “both” –
Which term, license or certification, do you normally use?
CN6. What does the term “work-related” mean to you? Do you feel that accurately describes your
situation?
CN7-CN9A. [Interviewer instruction] Pay attention to whether or not their problems with people
confusing the application process for a license versus the courses or training.
Were any of these questions about your training and education difficult for you to answer?
CN16. Was it easy or difficult for you to answer this question?

Certificates
CTINTRO, CT1. Was this question easy or difficult for you to answer? In your own words, what is
this question asking?
CT2B. How did you come up with your most recent certificate?
CT12. How did you come up with your answer for this question?
CT15. Was it easy or difficult for you to answer this question?

4