Cognitive Testing

Physician Survey of Research Data about Prescription Drugs

Cognitive Testing Moderator Guide Round 2

Introduction

Hello, my name is ______________ and I work for Westat, a survey research company in Rockville, Maryland. Thank you for agreeing to participate in this interview today. Let me start by telling you more about today’s interview. The U.S. Food and Drug Administration (FDA) is conducting this study to better understand physicians’ opinions and preferences about the type of information that is provided about prescription drugs. We’d like you to answer some questions about how you learn about uses that are not FDA-approved, and to review some sample pharmaceutical promotional materials. 

Informed Consent

Before we get started, there are a few things I want to mention. Your participation in this research study is voluntary. You may choose not to answer any question, and you can stop the interview at any time.

As described in the informed consent document you received earlier, there are no direct benefits to you from participating in this research. Your responses are very important because they will help researchers at FDA understand how physicians make decisions about medications. We also believe the risks to you from participating are low. There is a slight risk that your personally identifiable information may be shared, but this is unlikely because we only have your first name.

I want to assure you that we take your privacy very seriously. Although we may include quotes in our report, we will never link your name or any identifying information to the quote. We will also create recordings of the interview and prepare transcripts for purposes of analysis and reporting. For this reason, we ask you use only your first name during the interview. The recordings will be saved in password protected files and stored on secure servers at Westat and FDA locations. All recordings will only be accessible to project team members and will be destroyed no later than three years after the study is completed.  

The interview will take about an hour and you will receive $200 as a token of appreciation. As a reminder, today’s interview will be audio-recorded.

[IF OBSERVERS ARE PRESENT] Finally, I want to let you know that other staff involved in the project are here today listening to our interview, but will not be asking questions.

Do you agree to participate in this interview and to have the interview recorded?

IF RESPONDENT ANSWERS NO, TERMINATE INTERVIEW.

MAKE SURE NO ONE’S WEBCAM IS ON.

Turn on audio recorder and start Zoom recording. The date and time is _______. Now that the recorder is running, let me ask again, do you agree to participate in this interview. Is it okay with you if we record this interview?

Thank you. Let’s get started.

Think-Aloud Completion of Questionnaire

Now, I’d like you to complete the survey. It’s here on the screen, and I’m going to give you control so that you can answer the questions.

IF NEEDED: The information at the beginning of the survey is for the main study, so it gives a different estimate of the time it will take and the incentive amount.

As I mentioned before, we’re interested in what you’re thinking while you’re filling out the survey. So, I’m going to ask you to think out loud. It would be helpful if you could tell me about any reactions—good or bad—that you have to the survey and the questions in it. You don’t have to read the questions themselves out loud—just tell me what you’re thinking as you go through the survey. I may sometimes interrupt you with a question.

While the respondent completes the questionnaire, make notes on:

• Any comments spoken aloud;

• Any difficulty following skip items;

• Any hesitation answering;

• Any questions left blank, and

• When respondent changes an answer.

if respondent asks a question: For now, please answer it the way you would if I wasn’t here listening. I’m making a note of your question, and after you finish the survey, we’ll talk about it.

General Retrospective Probing

Thank you for completing the survey. Now I’m going to ask you some questions.

Overall Impressions

Research Questions

• What items are difficult for respondents to understand? What makes them difficult?
  Signs of difficulty include: re-reading the question stem, voicing that a question is difficult to answer

• What items are difficult for respondents to answer? What makes them difficult?
  Signs of difficulty include: taking a long time to answer, difficulty mapping answer to response options

First, just tell me your overall impressions of the survey.

Was there anything in the survey that was unclear or confusing? What was unclear?

Targeted Retrospective Probing

Now I’d like to talk about some of the questions in the survey. Let’s start back at the beginning.

TAKE CONTROL BACK ON ZOOM AND SCROLL THROUGH QUESTIONNAIRE AS PROBING.

ASK SCRIPTED PROBES. SPONTANEOUSLY PROBE ON ANY ISSUES OBSERVED OR VERBALIZED BY THE RESPONDENT DURING THE THINK-ALOUD.

Physician Survey of Research Data about Prescription Drugs

Thank you for participating in this survey. On the next screen, you will be shown some informational material about a prescription drug.

Please read the material. After you finish, you will be asked some questions about it.

Q1_STIUMULI_confirm view

1. Were you able to view the material?

• Yes

• No  END

Q2_ STIUMULI _recall

NOTE: Respondents will not be able to go back to material to answer

PROGRAMMING: Randomize ordering of items a-e.

1. Please answer the following questions based on what you remember from the material.

The following information may or may not have been in the material you just viewed. Do you remember seeing this information?

CORRECT ANSWERS ARE PROVIDED ONLY AS Information FOR INTERVIEWERS; DO NOT SHARE WITH RESPONDENT DURING PROBING.

	Yes	No	I’m not sure
SHOW FOR ALL CONDITIONS: A brief report on a study that supports use of [DRUG] for [DISEASE]			
SHOW FOR ALL CONDITIONS: A discussion of the limitations of a study that supports use of [DRUG] for [DISEASE]			
SHOW FOR ALL CONDITIONS (cancer only): c1. A description of an experimental study design			
SHOW FOR ALL CONDITIONS (insomnia only): c2. A description of a retrospective study design			
SHOW FOR ALL CONDITIONS: Outcomes for a placebo group			
SHOW FOR CONDITON 1 ONLY: A brief report on a study that does not support use of [DRUG] for [DISEASE]			
SHOW FOR CONDITION 2 ONLY: A summary of results from a study that does not support use of [DRUG] for [DISEASE]			
SHOW FOR CONDITION 3 ONLY: A citation for a study that does not support use of [DRUG] for [DISEASE]			
SHOW FOR CONDITION 4 ONLY: A general statement, provided without any citations, that other study results may not support use of [DRUG] for [DISEASE]			
SHOW FOR CONDITION 5 ONLY: Information about whether the described use of [DRUG] was off-label			

RESEARCH QUESTIONS

• Can respondents map items to what they read?

• Does including an “I’m not sure” option encourage respondents to choose it?

How easy or difficult was it to answer this question?
IF DIFFICULT, What made it difficult?
PROBE ON ANY DIFFICULTIES RELATED TO THE WORDING OF THE ITEMS.

for any ITEM where the respondent incorrectly answered “no”:
Tell me about your answer to row #.

IF participant SELECTS “I’M NOT SURE” FOR AN ITEM:
Tell me about your answer to Question #.
If not sure was not listed as an option, what would you do?

Q3_contrary data_validity

3. How confident are you in the validity of the conclusion that the authors draw about the use of [DRUG] for [DISEASE]?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

RESEARCH QUESTIONS

• How do respondents interpret “validity” in this context?

• What factors contributed to the respondent’s assessment of validity?

In your own words, what is this question asking?

Tell me about your answer.

What information did you consider when deciding whether you felt the study’s conclusion was valid?

For conditions 1-4, Did the information {on the second page/in the box} influence how you answered this question? Why or why not?

Meaning, Influence, and Opinion Items

NOTE: We will not refer to the text as a “disclosure statement,” but will reference it descriptively (i.e., “the information in the box”).

CONDITION 1:

The next question asks specifically about the box at the top of the first page. This is the same material you viewed earlier.

CONDITIONS 2-4:

The next questions ask specifically about the information in the box at the top of the page. This is the same material you viewed earlier.

CONDITION 5:

The next questions ask about the same material you viewed earlier.

CONDITIONS 1-4:

Q4_disclosure statement_meaning

NOTE: ASKED ONLY OF CONDITIONS 1-4

4. In your own words, how would you explain the information in the box to a colleague?

RESEARCH QUESTIONS

• Do respondents have difficulty paraphrasing the disclosure statement?

How easy or difficult was it to answer this question?
IF DIFFICULT, What made it difficult?

NEW PROBE
What do you think the purpose of the box is?

CONDITIONS 1-5

Q5_disclosure statement_likely

5. After reading the information in the [DISPLAY FILL], how likely are you to consider prescribing [DRUG] to a patient with [DISEASE]?

• Not at all likely

• A little likely

• Somewhat likely

• Very likely

• Extremely likely

CONDITION 1:

The next question asks specifically about the second page of the material. This is the same material you viewed earlier.

Q6_disclosure statement_likely 2nd page

NOTE: ASKED ONLY OF CONDITION 1

6. After reading the information on the second page, how likely are you to prescribe [DRUG] to a patient with [DISEASE]?

• Not at all likely

• A little likely

• Somewhat likely

• Very likely

• Extremely likely

RESEARCH QUESTIONS

• Are respondents answering based on the material, or their own experience with the drug?

Tell me about your answer{s} to Question{s} 5 {and 6}.

Have you ever prescribed [DRUG] to a patient with [DISEASE]?
IF YES, Do you think the information {in the box/on the second page} would make more, less, or the same difference if this was a drug you had never prescribed?

NEW PROBE!
Is [DISEASE] a condition that you often prescribe off-label for?
Are there other conditions for which you prescribe off-label more often?

Conditions 1-5

Q7_disclosure statement_useful

7. If you were considering prescribing [DRUG] for [DISEASE], how useful would the information [DISPLAY FILL] be?

• Not at all useful

• A little useful

• Somewhat useful

• Very useful

• Extremely useful

Q8_disclosure statement_clear

8. How clearly is the information presented [DISPLAY FILL]?

• Not at all clear

• A little clear

• Somewhat clear

• Very clear

• Extremely clear

Q9_disclosure statement_credible

9. How credible is the information presented [DISPLAY FILL]?

• Not at all credible

• A little credible

• Somewhat credible

• Very credible

• Extremely credible

RESEARCH QUESTIONS

• In question 9, do respondents understand the term “credible” in this context?

In your own words, what is Question 9 asking?
IF NEEDED, What makes information credible or not credible?

Q10_disclosure statement_more info

10. After reviewing the information [DISPLAY FILL], how likely are you to search for additional information about use of [DRUG] for [DISEASE]?

• Not at all likely

• A little likely

• Somewhat likely

• Very likely

• Extremely likely

RESEARCH QUESTIONS

• How are respondents assessing likelihood of searching for additional information?

Tell me about your answer.
IF NEEDED, How did you decide on {ANSWER}?

CONDITIONS 1-4:

The next questions ask about the entire material.

CONDITIONS 1-5:

Q11_ stimuli_missing info specify

11. What additional information, if any, did you need in order to consider prescribing [DRUG] for [DISEASE]?

RESEARCH QUESTIONS

• Do respondents expect this missing information to be included in the disclosure?

IF ADDITIONAL INFORMATION NEEDED, Where would you expect to find the additional information?

Would you expect the additional information to be included {in the box/ on the second page}?

Stimuli Assessment

Please note that this section (Questions 13-14) will only be included in cognitive testing and the PREtest. this will allow us to identify issues with the stimuli before the actual study.

Q12_ stimuli_difficulty

NOTE: ONLY INCLUDED IN TESTING (cognitive & PRETEST)

12. How difficult was it for you to answer these questions about the material you read?

• Not at all difficult

• A little difficult

• Somewhat difficult

• Very difficult

• Extremely difficult

Q13_ stimuli_difficulty specify

NOTE: ONLY INCLUDED IN TESTING (cognitive & PRETEST)

13. Please describe what made it difficult.

RESEARCH QUESTIONS

• What made it difficult to answer the questions about the stimuli? Elicit additional information for any issues that are specific to the stimuli, e.g., what information is included.

AS NEEDED, Tell me more about your answer to Question 13.

Prescribing Decisions

The next questions ask about how you decide to prescribe an approved drug for an indication that has not yet been approved by FDA, often called “off-label use.”

Q14_OFF-LABEL USE: freq offlabel

14. How often do you prescribe a drug for an off-label use?

• Once a week or more often

• Several times each month

• Several times each year

• Less than once a year

• I have never prescribed a drug for an off-label use

RESEARCH QUESTIONS

• Can respondents map their frequency of prescribing to the provided response options?

How easy or difficult was it to answer Question 14? What made it {easy/difficult}?

Q15_OFF-LABEL USE: practice_freq

PROGRAMMING: Ask only of participants who indicated prescribing off-label in Q14

15. Compared to you, how often do others in your practice prescribe a drug for off-label use?

• More often

• Less often

• About the same

• No other HCPs with prescribing authority in practice

Q16_OFF-LABEL USE: practice_yes/no

PROGRAMMING: Ask only of those who indicated having never prescribing off-label in Q14

16. Do others in your practice prescribe drugs for off-label use?

• Yes

• No

• I don’t know

• No other HCPs with prescribing authority in practice

RESEARCH QUESTIONS

• Can respondents answer questions about the prescribing habits of others in their practice?

How confident are you in your answer? IF CONFIDENT, What makes you confident?

Q17_OFF-LABEL USE: Info sources

PROGRAMMING: Ask only to participants who indicated prescribing off-label in Q14

17. Where do you typically first hear or learn about an off-label use for a prescription drug?

RESEARCH QUESTIONS

• Do respondents understand the item as intended?

• How easy or difficult is for respondents to describe where they typically first hear about an off-label use?

In your own words, what is this question asking?

Tell me more about your answer.

Are there any other places where you might hear of an off-label for the first time?
IF OTHERS NOT LISTED IN RESPONSE, How did you decided which sources to list here?

Overall, how easy or difficult was it to answer this question?

Q18_OFF-LABEL USE: Info sources_freq use

PROGRAMMING: Ask only of participants who indicated prescribing off-label in Q14

PROGRAMMING: Randomize ordering of items a-g, leaving “other specify” (h) last

18. How often do you use the following sources to learn about off-label uses for a drug?

	Never	Rarely	Sometimes	Often	Very Often
Colleagues					
Medical journals					
Google or other online search engines					
Medical reference websites such as UpToDate or Epocrates					
Professional medical associations (e.g., conferences, email, website)					
FDA					
Pharmaceutical companies					
Other, please specify					

Q19_OFF-LABEL USE: Infosources_ likely use

PROGRAMMING: Ask only to participants who indicated having never prescribing off-label in Q14

PROGRAMMING: Randomize ordering of items a-g, leaving “other specify” (h) last

19. How likely are you to use the following sources to learn about off-label uses for a drug?

	Not at all likely	A little likely	Somewhat likely	Very likely	Extremely likely
Colleagues					
Medical journals					
Google or other online search engines					
Medical reference websites such as UpToDate or Epocrates					
Professional medical associations (e.g., conferences, email, website)					
FDA					
Pharmaceutical companies					
Other, please specify					

RESEARCH QUESTIONS

• Can respondents map their sources of information to the sources provided here?

IF OTHER SPECIFY, Tell me about your answer to the “Other” category.
Is this a pretty common source of information on off-label use for PCPs?

Were there any sources missing from the list?
Were there any sources you feel could be deleted from the list?

Q20_OFF-LABEL USE: Infos ources_ seen support

20. Thinking of the studies you have seen in the past, how often have you seen study findings that support an off-label use of a drug?

• Never

• Rarely

• Sometimes

• Often

• Very often

Q21_contrary data_seen CONTRADICTORY

21. Thinking of the studies you have seen in the past, how often have you seen study findings that contradict an off-label use of a drug?

• Never

• Rarely

• Sometimes

• Often

• Very often

Q22_CONTRARY DATA_SEEN INCONCLUSIVE

22. Thinking of the studies you have seen in the past, how often have you seen study findings that have inconclusive support for an off-label use of a drug?

• Never

• Rarely

• Sometimes

• Often

• Very often

RESEARCH QUESTIONS

• Do respondents consider “contradict” and “inconclusive support” to be similar or different concepts?

• How do respondents interpret the response options?

In your own words, what is Question 21 asking?
Tell me about your answer to Question 21.
IF NEEDED, When you say {ANSWER}, how often is that?
ELICIT ABSOLUTE FREQUENCY DESCRIPTION

In your own words, what is Question 22 asking?
Tell me about your answer to Question 22.
IF NEEDED, When you say {ANSWER}, how often is that?
ELICIT ABSOLUTE FREQUENCY DESCRIPTION

What’s the difference, if there is one, between study findings that “contradict” an off-label use and study findings that have “inconclusive support” for an off-label use?

Q23_contrary data_support study aspects

PROGRAMMING: Randomize ordering of items a-h, leaving “other specify” (i) last

23. How important is it to you to know about the following aspects of studies that support an off-label use of a drug?

	Not at all important	A little important	Somewhat important	Very important	Extremely important
Study population					
Study design, e.g., randomized controlled study, observational study					
Sample size
Findings related to safety of the off-label use					
Findings related to side effects of the off-label use					
Study sponsor					
Number of studies that contradict or are inconclusive about an off-label use of a drug					
Findings related to effectiveness of the off-label use
Other, please specify:					

RESEARCH QUESTIONS

• Is the list of study components presented complete?

• Do respondents consider the length of time the study is conducted to be part of the study design?

• Are the sources listed the most common sources?

IF OTHER SPECIFY, Tell me about your answer to the “Other” category.
IF NEEDED, Why is it important to know that aspect of the study?

Were there any sources missing from the list?
Were there any sources you feel could be deleted from the list?

What about the length of the study, in terms of how many weeks, months, or years it was conducted? Were you considering that when you answered?

Q25_contrary data_ contrary study aspects

PROGRAMMING: Randomize ordering of items a-h, leaving “other specify” (i) last

24. How important is it to you to know about the following aspects of studies contradict or are inconclusive about an off-label use of a drug?

	Not at all important	A little important	Somewhat important	Very important	Extremely important
Study population					
Study design, e.g., randomized controlled study, observational study					
Sample size					
Findings related to safety of the off-label use					
Findings related to side effects of the off-label use					
Study sponsor					
Number of studies that contradict or are inconclusive about an off-label use of a drug					
Findings related to effectiveness of the off-label use					
Other, please specify:					

RESEARCH QUESTIONS

• Do respondents have difficulty answering about contradictory and inconclusive findings together?

• Would respondents answer differently if asked separately about findings that contradict and findings that are inconclusive?

• Is the list complete?

How easy or difficult was it to answer this question? What made it {easy/difficult}?

If we split this question into two questions, where one asked about studies that contradict an off-label use and the other asked about studies that are inconclusive, would your answers be the same or different? Why would they be the {same/different}?

IF OTHER SPECIFY, Tell me about your answer to the “Other” category.
IF NEEDED, Why is it important to know that aspect of the study?

IF NOT selected AT PREVIOUS ITEM, Can you tell me why that’s important to know for studies that do not support an off-label use, but it’s not important for studies that do support it?

Q26_contrary data_circumstances

NOTE: This question would be used in cognitive testing and/or pre-testing to develop a closed-ended survey question.

25. Under what circumstances would it be most important for you to know about studies that contradict or are inconclusive about an off-label use of a drug?

RESEARCH QUESTIONS

• Would respondents answer differently if asked separately about findings that contradict and findings that are inconclusive?

• Elicit sufficient information to support development of a close-ended list of response options.

Tell me more about your answer.

If we split this question into two questions, where one asked about studies that contradict an off-label use and the other asked about studies that are inconclusive, would your answers be the same or different?
Why would they be the {same/different}?

Q27_contrary data_contrary sources

PROGRAMMING: Randomize ordering of each response option, leaving the “other specify” option last

26. Of the following sources, please rank the top three which you would most likely use to learn about studies that contradict or are inconclusive about an off-label use of a prescription drug.

Please write 1, 2, or 3 in the boxes below to rank your top three choices.

• Colleagues

• Medical journals

• Google or other online search engines

• Medical reference websites such as UpToDate or Epocrates

• Professional medical associations (e.g., conferences, emails, website)

• FDA

• Pharmaceutical companies

• Other, please specify:
  

RESEARCH QUESTIONS

• Do respondents understand the task?

• Is the list complete?

How easy or difficult was it to answer this question? What made it {easy/difficult}?

IF OTHER SPECIFY, Tell me about your answer to the “Other” category.
IF NEEDED, Is that a source commonly used by other {PCPs/oncologists}?

Were there any sources missing from the list?
Were there any sources you feel could be deleted from the list?

Debrief

Thank you for taking part in this survey. The information provided about {DRUG} and its off-label use for {CONDITION} is a combination of several different studies. While the information presented is accurate, the Brief Report was created specifically for this survey and is not a real publication.

RESEARCH QUESTIONS

• Does the debrief accurately convey the information?

• Do respondents interpret the debrief as applying to both the main report and the contrary data?

In your own words, what is this paragraph saying?

FOR CONDITION 1: To you, does this paragraph apply to both of the pages of material you read, or just to part of it?

FOR CONDITIONS 2-5: To you, does this paragraph apply to the entire page of material you read, or just to part of it?

AS NEEDED, PROBE TO CONFIRM UNDERSTANDING.

Closing

Double-check E-MAIL for any questions from the observers.

That is all the questions I have for you. Is there anything we haven't discussed that you would like to mention?

Discuss any respondent comments.

{SERMO/Reckner} should be sending your $(200/350} to you soon. Please reach out to them with any questions. If you have other questions, our contact information is provided on the consent form we sent you.

Stop audio recorder and Zoom recording.

OMB Control No. 0190-0900 Expires 8/31/2024