Protocol

Protocol for National Survey of Psychiatrists Questionnaire Pretesting

Participant ID #: |___|___|___|___|___| ___|___|___|

Interview Date: |__|__| / |__|__| / |__|__|__|__| (mm/dd/yyyy)

Interviewer initials: |__|__|

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Section 1: ** Read/ Paraphrase the following text**
Greeting: Hello. My name is ________________, and I work for the Census Bureau.

Thank you for agreeing to participate in our study.

What: Let me start by telling you a little about what we will be doing today. The National Survey of Psychiatrists (NSP), is a new survey that the U.S. Census Bureau will conduct on behalf of the Substance Abuse and Mental Health Services Administration (SAMHSA). This survey collects information about the work experiences of psychiatrists and the services they provide. Our goal is to design a questionnaire that collects high quality data without placing excessive burden on our respondents. Before we finalize the questions in the survey, we want to test them with people like yourself to make sure the questions are easy to understand and answer.

We’re asking you to read through the survey and complete it as you would if you received it in the mail. We are interested in your feedback so we can know what people think of the questions in the survey and how we can potentially improve them. We will be looking at two items. I’ll explain each one to you as we go along. I’m going to ask you to read through and respond to the survey questions, and then ask you some questions about them: what you think certain questions are asking, and what your reactions to them are. There are no right or wrong answers. We are interested in what people think about the survey you will experience today. Please give me your honest impressions, whether good or bad. We appreciate your help so we can make the survey work well for everyone.

How: **SHOW PARTICIPANT THE QUESTIONNAIRE**

This is the questionnaire for the National Survey of Psychiatrists. If this were the real survey, you would either receive this questionnaire in the mail or an invitation to complete the survey online. In a couple of minutes, I am going to ask you to look at this questionnaire just as you would if you had received it in the mail but with one major difference.

Think Aloud: I would like you to think aloud as you read and answer the questions in the survey. I am interested in your feedback on the questions, but I am also interested in the process you go through in your mind as you come up with answers to the questions in the survey. I would like you to tell me everything that you are thinking and feeling as you read come up with your responses to the survey questions. You might have some questions about the survey that come up as we go. You can still ask these questions, but I will wait until the end to answer them. I want to use this time to get your thoughts and opinions.

Practice: Let’s do a practice before we start. Please think aloud as you answer this question: How many windows are in your home?

Interruptions: I might stop you at a couple of points and ask you some questions about your feedback, or about the materials themselves. I am also going to ask you some questions at the end. There are no right or wrong answers, because only you know what you are thinking.

Do you have any questions before we begin?

Confidentiality: Our session today is completely confidential. Your participation in this study is
completely voluntary, and you can decline to answer any particular question.

Recording: So I don’t have to rely on my memory later on, I’d like to record this interview.
That way, I can focus today on what you’re saying rather than having to concentrate on taking notes. Is that ok with you?

**HAVE PARTICIPANT SIGN CONSENT FORM**

**Begin: **TURN ON RECORDER**

OK, let’s begin. Please remember to think aloud as you go through the survey.

Interviewer: Note any confusion or difficulties participants have with the questionnaire.

If participant is not being talkative despite reminders, ask them to underline things they don’t like or find confusing in the questionnaires and circle things they do like.

______________________________________________________________________________

Initial Reactions – Cover Page

Assigning the participant to condition A, B

INTERVIEWER: IF THE PARTICIPANT ID ENDS IN:

ODD NUMBER – ASSIGN THE PARTICIPANT TO CONDITION A – PRESENT THE STANDARD VERSION OF THE COVER PAGE/SURVEY FIRST (QUESTIONS BEGIN ON COVER)

EVEN NUMBER – ASSIGN THE PARTICIPANT TO CONDITION B – PRESENT THE ALTERNATIVE VERSION OF THE COVER PAGE/SURVEY FIRST (QUESTIONS BEGIN ON NEXT PAGE)

**HAND THE PARTICIPANT THE FIRST VERSION OF THE NSP PAPER SURVEY THAT THEY WILL SEE**

**STOP THE PARTICIPANT AFTER SECTION A AND PROVIDE THE OTHER VERSION**

Section A: Eligibility

**HAND THE PARTICIPANT THE SURVEY**

SET-UP: This is the survey you would receive if you were selected to participate in the NSP. I’d like for you to go ahead and read through and complete the survey as you would if you received it in the mail. Please remember to think aloud as you read the questions and answer them.

“START HERE”

INTERVIEWER: NOTE WHETHER THE PARTICIPANT READ THE TEXT UNDER THIS SECTION.

FOCUS THE PARTICPANT ON THE FIRST PART REGARDING RESPONDING ONLINE OR COMPLETING THE PAPER FORM.

Standard Probes:

1. If you received this survey do you think you would prefer to complete it online or using the paper questionnaire and mailing it back? (IF NEEDED) Why?

Question A1. Are you currently licensed to practice medicine in the U.S.?

INTERVIEWER: NOTE ANY FEEDBACK ON QUESTION WORDING OR CUES TO HESITATION.

Standard Probes:

1. (IF NEEDED) What do you think is meant by “to practice medicine”?

Question A3. Are you currently providing direct clinical care as a psychiatrist?

Standard Probes:

1. How would you define “direct clinical care”?

INTERVIEWER: note the features mentioned to ensure that the interpretation is as intended.

Question A4. Have you provided direct clinical care within the PAST TWO YEARS?

Standard Probes:

1. In your own words, what is this question asking?

INTERVIEWER: note if the participant mentions their residency as the situation in which they provided direct clinical care

2. (IF NEEDED – if participant references a temporary situation) For how long did you provide direct clinical care during that time?

** RECORD WHICH SECTION THE PARTICIPANT SHOULD GO TO NEXT _______**

This is another design for the survey that we are considering using. Please take a look at the first couple pages **HAND THE PARTICIPANT THE SECOND VERSION OF THE NSP PAPER SURVEY**

Standard Probes:

1. How would you describe the difference?

2. Is there one design that you prefer over the other?

INTERVIEWER: note any feedback the participant provides

.

**REFOCUS THE PARTICIPANT ON THE STANDARD VERSION OF THE SURVEY BEFORE GOING ON TO THE NEXT SECTION**

Section B: Current Employment in Direct Clinical Care

Question B1. Which areas of clinical care do you specialize in? Mark all that apply.

Standard Probes:

1. What do you think is meant by “specialize in”?

2. (IF NEEDED) How easy or difficult is it for you to answer this question?

DID pARTICIPANT indicate they had a speciality?

Question B2. During a typical week, how many different employment settings do you work in?

Standard Probes:

1. In your own words, what is meant by “setting”?

HOW IS PARTICIPANT DEFINING EMPLOYMENT SETTINGS?

2. (IF NEEDED) Is the number of settings the number of locations? Is it by the types of services provided? If you worked in 2 clinics, how would you answer this question?

INTERVIEWER: note IF PARTICIPANT MENTIONS “LOCATIONS”

DID pARTICIPANT DESCRIBE THE WORKPLACE/FACILITY AS A SETTING OR SOMETHING ELSE

(if so, what)?

Question B3. What settings do you provide direct clinical care within? Mark all that apply.

Standard Probes:

1. Would you describe these response options as “settings”?

2. (IF NEEDED) How would you describe them?

Question B4. What percentage of time do you spend on each of the following types of work in a typical week?

Standard Probes:

1. How confident are you in your answer?

INTERVIEWER: note ANY CONFUSION OR DIFFICULTY. NOTE QUESTIONS OR FEEDBACK ON RESPONSE OPTIONS

Was the participant able to calculate a percentage?

2. What do you think is meant by response option C – “Programs/services leadership”?

(IF NEEDED) Can you give me an example?

Question B5. During a typical week, what percentage of time do you spend providing services to clients in each of the following age groups?

Standard Probes:

1. What do you think this question is asking?

2. How would you calculate this?

3. (IF P STRUGGLD WITH QUESTION) What was difficult about trying to answer this question?

Question B9. Are you able to provide psychiatric services to patients in a language other than English USING an interpreter or translator?

Standard Probes:

1. Can you tell me what is meant by “able to provide”?

Question B10. In which languages are you able to provide psychiatric services to patients using an interpreter or translator? Mark all that apply.

Standard Probes:

1. Were you thinking of languages you currently provide services through or ones that you could although not necessarily doing so at this point in time?

Question B12. Do you have any patients in your current caseload for whom you are providing specific services related to gender identity?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) What do you think is meant by “specific services”?

Question B13. Do you have any patients in your current caseload for whom you are providing specific services related to sexual orientation?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) What are examples of “specific services” that you may provide related to sexual orientation?

Questions B14a – B20

INTERVIEWER: Note any confusion or difficulty. Note any questions or feedback.

Question B20. How many individual patients do you treat using Buprenorphine in a typical week?

INTERVIEWER: Note whether the participant interprets as number of individual patients rather than doses administered.

Note any confusion or difficulty. Note any questions or feedback.

Question B21b. Which of the following treatment methods are you currently providing? Mark all that apply.

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) Is there anything missing from the list of response options?

Question B22. Are you providing court-mandated outpatient treatment to any patients in your current caseload?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. What do you think this question is asking?

2. Is there another word you would use to describe “court-mandated outpatient treatment”?

Question B23. Which of the following best describes your current employment arrangement?

Mark all that apply.

INTERVIEWER: Note any feedback to the word “arrangement”

Standard Probes:

2. (IF NEEDED) What do you think is meant by “arrangement”?

Question B24. How satisfied or dissatisfied are you with each of the following dimensions of your work in direct clinical care?

INTERVIEWER: Note any difficulty or confusion and any feedback.

Question B25. Do you work in an integrated primary care and behavioral health model in a typical week?

An Integrated Care Model is defined as tightly integrated, on-site teamwork with a unified care plan as a standard approach to care for designated populations. Connotes organizational integration as well, often involving social and other community services.

INTERVIEWER: Note any difficulty or confusion and any feedback.

Question B26. Do you use Electronic Health Records for the following tasks? Mark Yes or No for each of the below.

INTERVIEWER: Note any difficulty or confusion and any feedback.

Standard Probes:

1. How would you describe the difference between storing records and maintaining records?

2. Is there a difference between patient health records and patient treatment records?

Question B29. On average, how long are sessions with existing patients? Please include any time required for documentation in addition to patient time.

Standard Probes:

1. Were you thinking of the time you spend or the time that the patient spends?

Question B31. Does the setting in which you provided care most recently accept insurance?

INTERVIEWER: Note any difficulty or confusion and any feedback to the word “setting”

Standard Probes:

1. (IF NEEDED) What do you think is meant by “setting” in this question?

Question B32. Which of the following are reasons that setting does not accept insurance?

Standard Probes:

1. (IF P SELECTS “OTHER”) Can you tell me more about that?

NOTE OTHER REASON P PROVIDES

Section C: Previous Employment in Direct Clinical Care

INTERVIEWER: NOTE ANY QUESTIONS IN SECTION C THAT WERE DIFFICULT FOR THE RESPONDENT TO ANSWER.

NOTE ANY INSTANCES WHEN PARTICIPANT PROVIDES A RESPONSE THAT ADDRESSES THEIR CURRENT SITUATION

NOTE PROBLEMS WITH RECALL AND HOW RECENT THE JOB WAS.

Question C1. Which area(s) of clinical care did you specialize in? Mark all that apply.

Standard Probes:

1. What do you think is meant by “specialize in”?

2. (IF NEEDED) How easy or difficult is it for you to answer this question?

DID pARTICIPANT indicate they had a speciality?

Question C2. During a typical week, how many different employment setting(s) did you work in?

Standard Probes:

1. In your own words, what is meant by “setting”?

HOW IS PARTICIPANT DEFINING EMPLOYMENT SETTINGS?

2. How easy or difficult was it to come up with your answer?

3. (IF NEEDED) Is the number of settings the number of locations? Is it by the types of services provided? If you worked in 2 clinics, how would you answer this question?

INTERVIEWER: note IF PARTICIPANT MENTIONS “LOCATIONS”

DID pARTICIPANT DESCRIBE THE WORKPLACE/FACILITY AS A SETTING OR SOMETHING ELSE

(if so, what)?

Question C3. What setting(s) did you provide direct clinical care within? Mark all that apply.

Standard Probes:

1. Would you describe these response options as “settings”?

2. (IF NEEDED) How would you describe them?

Question C4: What proportion of time did you spend on each of the following types of work activities in a typical week?

Standard Probes:

1. How easy or difficult was it to come up with your answer?

2. How long ago was the job you were answering in reference to?

INTERVIEWER: note ANY CONFUSION OR DIFFICULTY. NOTE QUESTIONS OR FEEDBACK ON RESPONSE OPTIONS

3. What do you think is meant by response option C – “Programs/services leadership”?

(IF NEEDED) Can you give me an example?

Was the participant able to calculate a percentage?

Question C5. During a typical week, what percentage of time do you spend providing services to clients in each of the following age groups?

Standard Probes:

1. What do you think this question is asking?

2. How would you calculate this?

3. (IF P STRUGGLD WITH QUESTION) What was difficult about trying to answer this question?

4. How confident are you in your answer?

Question C9. Were you able to provide psychiatric services to clients/patients in a language other than English USING an interpreter or translator?

Standard Probes:

1. Can you tell me what is meant by “able to provide”?

Question C10. In which language(s) were you able to provide psychiatric services to clients/patients using an interpreter or translator? Mark all that apply.

Standard Probes:

1. Were you thinking of languages you actually provided services through or ones that you could have?

Question C12. Did you have any patients in your current caseload for whom you are providing specific services related to gender identity?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) What do you think is meant by “specific services”?

Question C13. Did you have any patients in your current caseload for whom you are providing specific services related to sexual orientation?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) What are examples of “specific services” that you may provide related to sexual orientation?

Questions C14a – C20

INTERVIEWER: Note any confusion or difficulty. Note any questions or feedback.

Question C20. How many unique patients do you treat using Buprenorphine in a typical week?

INTERVIEWER: Note whether the participant interprets as number of individual patients rather than doses administered.

Note any confusion or difficulty. Note any questions or feedback.

Question C21b. Which of the following treatment methods were you providing? Mark all that apply.

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. (IF NEEDED) Is there anything missing from the list of response options?

Question C22. Were you providing court-mandated outpatient treatment to any patients in your current caseload?

INTERVIEWER: Note any feedback to thIS QUESTION

Standard Probes:

1. What do you think this question is asking?

2. Is there another word you would use to describe “court-mandated outpatient treatment”?

Question C23. Which of the following best describes the employment arrangement of your most recent clinical care work? Mark all that apply.

INTERVIEWER: Note any feedback to the word “arrangement”

Standard Probes:

1. (IF NEEDED) What do you think is meant by “arrangement”?

Question C24. How satisfied were you with each of the following dimensions of your work in direct clinical care?

INTERVIEWER: Note any difficulty or confusion and any feedback.

Question C25. Did you work in an integrated primary care and behavioral health model in a typical week?

An Integrated Care Model is defined as tightly integrated, on-site teamwork with a unified care plan as a standard approach to care for designated populations. Connotes organizational integration as well, often involving social and other community services.

INTERVIEWER: Note any difficulty or confusion and any feedback.

Question C28. On average, how long were sessions with existing patients? Please include any time required for documentation in addition to patient time.

Standard Probes:

1. Were you thinking of the time you spent or the time that the patient spends?

Question C29. Did the setting in which you provided care most recently accept insurance?

Standard Probes:

1. How easy or difficult is it for you to answer this question”? (IF NEEDED) Why?

2. (IF NEEDED) How long ago were you trying to remember?

INTERVIEWER: Note ANY RECALL DIFFICULTIES

Question C30. Which of the following are reasons that setting did not accept insurance? Mark all that apply.

INTERVIEWER: Note ANY RECALL DIFFICULTIES

Question C31. What types of insurance were accepted by the setting in which you most recently provided clinical care? Mark all that apply.

Standard Probes:

1. How easy or difficult is it for you to answer this question”? (IF NEEDED) Why?

INTERVIEWER: Note ANY RECALL DIFFICULTIES

Section D: Current Employment NOT in Clinical Care

Question D1. Do you work in healthcare or a healthcare related field?

INTERVIEWER: NOTE ANY FEEDBACK OR QUESTIONS THE PARTICIPANT ASKS.

Question D3. During a typical week, how many different employment settings do you work in?

INTERVIEWER: NOTE IF THE PARTICIPANT MENTIONS “LOCATION”

INTERVIEWER: WE WANT TO KNOW WHETHER THE PARTICIPANT DESCRIBES THEIR FACILITY TYPE (E.G. AN ADDICTION CLINIC, ETC) AS A “SETTING” OR SOMETHING ELSE.

Standard Probes:

1. In your own words, what is meant by “setting”?

2. If a person works in two addiction clinics, do you think this question intends that he/she answer 1 or 2 (settings)?

Question D4. In what field are you currently employed?

INTERVIEWER: NOTE ANY FEEDBACK OR QUESTIONS THE PARTICIPANT ASKS.

Standard Probes:

1. (IF NEEDED) How easy or difficult was it to select your response?

Section E: Education and Credentials

Question E6. How much debt did you have upon completing your residency?

INTERVIEWER: THE GOAL OF THE PROBES HERE ARE TO FIND OUT WHETHER RESPONDENTS ARE LIKELY TO OVERLOOK ANY SOURCES THAT MAY BE IMPORTANT FOR MEASURING DEBT AMOUNT (SUCH AS CREDIT CARDS, LOANS FROM FAMILY, ETC).

Standard Probes:

1. What sources of debt were you thinking of? INTERVIEWER: NOTE IF THE PARTICIPANT MENTIONS CREDIT CARDS OR ANYTHING OTHER THAN STUDEN LOANS. RECORD ALL SOURCES MENTIONED.

2. DID P MENTION DEBT AFTER MEDICAL SCHOOL AS OPPOSED TO AFTER RESIDENCY?

3. (IF NEEDED) Is it easier to remember the amount of debt you have after medical school or after your residency?

Question E7. Are you currently ABPN board certified?

Standard Probes:

1. Which boards were you thinking of when you answered this question?

Question E8. Which specialties are you currently APBN board certified in?

INTERVIEWER: IF RESPONDENT DOES NOT SELECT “PSYCHIATRY” ASK:

1. DID YOU NOTICE “PSYCHIATRY” IN THIS LIST?

2. (IF NEEDED) IF “PSYCHIATRY” HAD BEEN AT THE TOP OF THE LIST WOULD YOU HAVE SELECTED IT?

INTERVIEWER: NOTE ANY OTHER SPECIALTIES THE PARTICIPANT PROVIDES OR ASKS ABOUT.

Section F: Demographics

The next couple of questions may be sensitive for some people. You don’t need to tell me your actual answer to these questions. I’m interested in what you think the question is asking you and any feedback you might have.

Question F6. What sex were you assigned at birth, on your original birth certificate?

INTERVIEWER: Note any feedback PARTICIPANT PROVIDES.

Question F7. Do you currently describe yourself as male, female or transgender?

INTERVIEWER: Note any feedback PARTICIPANT PROVIDES.

Question F8. Which of the following best represents how you think of yourself?

INTERVIEWER: Note any feedback PARTICIPANT PROVIDES.

Cover Page Images

I’d like to take a moment to look at the images on the cover page of the survey. These are some options that we may choose from.

**HAND THE PARTICIPANT THE ADDITIONAL COVER PAGE IMAGES**

Standard Probes:

1. What do you think of these images? What do they make you think of?

2. (IF NEEDED) Is there one that you like better than the others?

3. (IF NEEDED) Is there another kind of image that you would prefer instead of these options?

______________________________________________________________________________

GENERAL DEBRIEFING PROBES:

Overall, what would you say about the survey you looked at today? Tell me more about that.

Do you think most psychiatrists would find the questions appropriate? Useful?

Why/Why not?

Do you think there are questions some people would find confusing? (IF YES) Which ones?

Were there any questions that you found sensitive? (IF YES) If you were taking the survey would you answer those questions?

Is there anything else you would like to tell us that you haven’t had a chance to mention yet?

______________________________________________________________________________

Thank you for your feedback today. Your participation is greatly appreciated and your input has been really helpful.

**PROVIDE INCENTIVE AND HAVE PARTICIPANT SIGN VOUCHER FORM**

20