2-2 EP attachment 5 - cognitive interview evaluation protocols.

OMB No. 0915-0212

Exp. Date 7/31/2021

A 90-Minute One-On-One Cognitive Interview

Adults Ages 45 and Older

EVALUATION PROTOCOL

November 23, 2020

SESSION OVERVIEW: TOTAL TIME 90 MINUTES	STIMULI
Background (<5 minutes)	--
Adult Literacy in Dentistry (10 minutes)	REALD-30 (PowerPoint Slides)
General Attitudes Toward Oral Health and Experiences with Oral Health Care (Including Behavioral Pre-Test) (10–15 minutes)	Two Poll Questions
Message and Materials Testing (60 minutes)	(1) Brushing and Flossing (revised brochure) messages (PowerPoint Slides) Review of material (2) Dry Mouth (revised brochure) messages (PowerPoint Slides) Review of material (3) Oral Cancer messages (PowerPoint Slides) Review of material
Close (Including Behavioral Post-Test) (<5 minutes)	One Poll Question

BACKGROUND [5 Minutes]

When the participant joins, the moderator will ask them to be sure they have a pen or pencil with them, as well as the unopened mailed envelope/packet. The moderator will also remind the participant to have their reading glasses and hearing aids available, if they wear them, and ensure they can see the shared computer screen.

• Welcome and thank you for your participation

• My name / independent researcher / not an employee of the Health Resources and Services Administration (HRSA) or the federal government (the sponsor of this project)

• My role (facilitate discussion and end on time at x:xx a.m./p.m.) 

• The purpose of today’s discussion is to:

  • Discuss your thoughts and feelings about health

  • Get your feedback on information that HRSA developed

  • Hear your suggestions for how HRSA could best reach you with important health information

• Some guidance for our conversation:  

  • Please talk in a voice at least as loud as mine.

  • There are no right or wrong answers.

  • This discussion is confidential—first names only / no names will be used in the summary report.

  • Please do not share any information which could be used to identify someone, such as someone’s full name.

  • Your participation is voluntary. You are free to decline to answer a question / free to leave at any time without any penalty.

  • I did not have anything to do with the development of the materials we are going to review—if you have constructive criticism, I will not be offended, and I will not be penalized.

    • Please be honest.

  • Our time together is limited—please do not be offended if I have to interrupt you to turn to the next question; there is simply a lot to cover, and we have to finish on time.

  • Please turn your cell phone off or turn the ringtone off.

  • Project staff are listening to the discussion, but we will be the only ones talking during our discussion.

Consent to Audio-Record

• We would like to audio (sound)-record this conversation so that when it comes time to write the summary report, we will have your actual words to ensure accuracy, if needed. However, you will not be identified in the report.

• Although we have our webcams (cameras) on for our live conversation, there will be no video-recording (video-archive) made of our conversation. 

• Do I have your permission to audio (sound)-record this discussion? [Dismiss participant if they do not verbally consent.]

ADULT LITERACY IN DENTISTRY [10 Minutes]

I am now going to show you a list of health-related words on the shared computer screen. As you see each word on the screen, I would like you to read the word out loud. If you do not know how to pronounce that word, please say, "I don’t know." Please do not guess at the pronunciation of a word.

Please note that this activity is not a test of knowledge like in school. YOU are testing words to help make health information easier to understand. This will help us to determine which words to include and not include in health information, so they are understandable to more people. This activity will take just a few minutes.

Note: The moderator and note-taker will time how long it takes each interviewee to complete this activity. The moderator will display one word at a time on the computer screen.

Rapid Estimation of Adult Literacy in Dentistry: 30 Word Version

1. Sugar _______ 11. Abscess _______ 21. Periodontal _______

2. Smoking _______ 12. Extraction _______ 22. Sealant _______

3. Floss _______ 13. Denture _______ 23. Hypoplasia _______

4. Brush _______ 14. Enamel _______ 24. Halitosis _______

5. Pulp _______ 15. Dentition _______ 25. Analgesia _______

6. Fluoride _______ 16. Plaque _______ 26. Cellulitis _______

7. Braces _______ 17. Gingiva _______ 27. Fistula _______

8. Genetics _______ 18. Malocclusion _____ 28. Temporomandibular _______

9. Restoration _______ 19. Incipient _______ 29. Hyperemia _______

10. Bruxism _______ 20. Caries _______ 30. Apicoectomy ___

GENERAL ATTITUDES TOWARD ORAL HEALTH AND EXPERIENCES WITH ORAL HEALTH CARE [10–15 Minutes]

Now, I’d like you to answer the question that appears on your computer screen. We’ll come back to this question later in the conversation.

1. POLL 1 (PRE-TEST): How likely are you to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

1a. Why did you rate this question in this manner? Please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

Please open the packet you received from us. Inside is an envelope that says, “DO NOT OPEN UNTIL THE START OF THE COGNITIVE INTERVIEW.” Please pull out the whole envelope and keep the material in the order that it is in. Do not flip through it. During our discussion, we will look at each piece of material together. Again, please don’t flip ahead.

2. Now look at the top page that has the word “TOP” on it. On this page, please write down the words “oral health” [spell out O-R-A-L] and then write down what these words mean to you. [DISCUSS. PROBE EXTENSIVELY.]

Probes might include any of the following:

• I noticed you included “teeth”—what makes you think of teeth?

• Does “oral health” include any of the following: gums, throat, and bones around the mouth? Are any of these surprising? Why?

  • What, if any, other parts of the body should be included in the broad “oral health” term? Why?

• What do you think people think of when they hear the term “oral health”? Do you think they know what is included within the term “oral health?” Is there a better way to talk about oral health that might make more sense to people? [Offer ideas if participants cannot generate examples on their own, such as “mouth and teeth health.”]

Please take a moment to respond to the question that appears on the computer screen.

3. POLL 2: How concerned are you about your oral health? 
   Use a scale from 1 to 7, with 1 being “Not at all concerned” and 7 being “Extremely concerned.” 
   [SHOW RESULTS. DISCUSS. PROBE REASONS.]

1. How has the importance of oral health changed as you have gotten older? Why?

2. What are the consequences of “bad” oral health and oral health care? What can happen?

3. How might oral health affect any other part of your body? How do you know this—did you read or hear this somewhere?

4. How often do you go to the dentist for a cleaning or checkup? [PROBE FOR MEANING (e.g., what does the participant consider “cleaning” / what does a “check-up” encompass?; if the participant says, “Once a year,” ask, “If it’s January 2021 now, what month and year would that be in this scenario?;” if the participant says, “Two times a year,” ask, “Explain to me which months of the year you would visit at this frequency? How many months are in between each visit?”)]

MESSAGE AND MATERIALS TESTING [60 Minutes Total]

Note: The moderator will test messages contained within three materials. The moderator will spend approximately 20 minutes on each material.

Material 1: Brushing and Flossing (20 Minutes Subtotal)

Note: The moderator will display the below instructions on the shared screen for the interviewee to refer to as needed. The moderator and note-taker will closely watch interviewees’ handling of the material—their body language, facial expressions, etc. This may provide clues or insights into ease or difficulty in comprehension. Also, the moderator will display on the shared screen the material that is being referenced at the appropriate time during the cognitive interview discussion.

• Circle anything that:

  • Is important

  • Is new information

• Cross out anything with an “X” that:

  • You think is unnecessary

  • Is something you disagree with

• Put a question mark next to anything that:

  • You have questions about

  • You do not understand

I am now going to read various statements, one-by-one, on the shared screen aloud. In your packet labeled X, you will also find these messages on paper. As we move along each statement, please use your pen or pencil to put a question mark next to any words or information that you find confusing, circle anything that you like, and cross out anything you dislike. Let’s start with the page that reads, “Oral health is…” As we talk about these sentences, again, please let me know if there were any words or phrases you circled, crossed-out, or placed a question mark next to. This is important for me to know. Also, if you do not know the answer to a question I ask, it is totally okay—please let me know and I will go on to the next question. Again, this is not to test you, this activity is for you to test the way health information is currently written. Your honesty will help health writers develop clearer health information.

5. MESSAGE: Oral health is important for people of all ages. A healthy mouth helps people enjoy their food, chew better, eat well, and avoid pain and tooth loss. With good oral hygiene and regular visits to the dentist, you can maintain your oral health for years.

1. What does “hygiene” mean to you? What does “oral hygiene” mean to you?

   • Can you think of an alternate term for “oral hygiene”?

2. What does “regular” mean in this sentence?

3. Take a look at the first sentence, “Oral health is important for people of all ages.” How believable is that? Why?

4. Take a look at the last few words, “…you can maintain your oral health for years.” What does “maintain” mean to you?

Let’s go to the next page.

6. MESSAGE: Flossing removes dental plaque between teeth where a toothbrush can’t reach. If not removed, dental plaque can build up and cause tooth decay and gum disease.

   1. What does “dental plaque,” “tooth decay,” and “gum disease” mean to you? [ASK ABOUT EACH WORD SEPARATELY.]

   2. What does the sentence mean by “build up”?

   3. What areas in the mouth do you think the writers were referring to when they say, “where a toothbrush can’t reach”?

Let’s go to the next page.

7. MESSAGE: Flossing Tools

NOTE: The moderator will show four images of flossing tools on the shared screen.

1. What is the meaning of “floss holder,” “floss threader,” “oral irrigator,” “interdental brush”? [ASK ABOUT EACH WORD SEPARATELY.]

2. How well did the photographs of each flossing tool help you identify each tool?

3. With which tool(s) are you familiar? Are there any tools you have never seen before? Which ones?

Let’s go to the next page.

8. MESSAGE: Brushing Your Teeth. Gum Disease—If plaque is not removed, some of it can harden below the gum line and irritate the gums. The gums become red, swollen, and may bleed easily. These are signs of gingivitis. Gingivitis is a mild form of gum disease, and you can usually reverse it with daily brushing and flossing. If there are areas in your mouth where your gums have pulled away from the teen (called gum recession), the exposed tooth roots can decay as well.

1. What is the main take-home message of this information?

2. Using your own words, please define “gum disease” / “gingivitis” / “gum recession” / “exposed tooth roots.”

3. What does “daily” mean in, “Gingivitis is a mild form of gum disease, and you can usually reverse it with daily brushing and flossing”?

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Brushing and Flossing.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

9. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

10. What do you think of the images? Would you prefer to see photographs or line drawings in this brochure?

11. Is there anything that is hard for you to read or see? Anything else you would change?

Material 2: Dry Mouth (20 Minutes Subtotal)

We will now review other statements again, one-by-one.

12. MESSAGE: Dry mouth is the feeling that there is not enough saliva in the mouth. Common causes of dry mouth include side effects of certain medications and dehydration, when you lose more fluid than you take in.

    1. Explain to me what this sentence is trying to say.

    2. In your own words, what is “saliva”? “Dehydration”? “Fluid”? [ASK ABOUT EACH WORD SEPARATELY.]

    3. What does “side effects” mean to you in this sentence? What is an example of a side effect from a medication?

Let’s go to the next page.

13. MESSAGE: Dry mouth can make it hard to chew, swallow, or even talk. Having less saliva also increases the risk of developing tooth decay or fungal infections in the mouth since saliva helps keep harmful germs in check.

    1. Explain to me what this sentence is trying to say.

    2. In your own words, explain what “increases the risk” means?

    3. What does “tooth decay” mean? “Fungal infections”? “Germs”? [ASK ABOUT EACH WORD SEPARATELY.]

Let’s go to the next page.

14. MESSAGE: Hundreds of medicines can cause the salivary glands to make less saliva. ​

1. Tell me what “hundreds” means in this sentence? Can you give a specific number example to illustrate what the brochure writers meant by “hundreds”?

2. What does “saliva,” “salivary glands” mean? [ASK ABOUT EACH WORD SEPARATELY.]

Let’s go to the next page.

15. MESSAGE: Diabetes, Sjogren’s syndrome, and HIV/AIDS can all cause dry mouth.

    1. Did you know that diabetes and HIV/AIDS can cause dry mouth?

    2. Have you ever heard of “Sjogren’s syndrome”? If so, where and when did you hear about it and what does it mean? How helpful would it be to include a definition of “Sjogren’s syndrome”?

    3. After reading this sentence, what questions do you have?

Let’s go to the next page.

16. MESSAGE: For dry mouth, you may do the following:​

• Sip water or sugarless drinks often especially during meals. This will make chewing and swallowing easier. It may also improve the taste of food. ​

• Avoid drinks with caffeine, such as coffee, tea, and some sodas. Caffeine can dry out the mouth.​

• Chew sugarless gum or suck on sugarless hard candy to stimulate saliva flow—citrus, cinnamon, or mint-flavored candies are good choices. Some sugarless chewing gums and candies contain xylitol and may help prevent cavities. ​

• Don’t use tobacco or alcohol. They dry out the mouth.​

• Be aware that spicy or salty foods may cause pain in a dry mouth. ​

• Use a humidifier at night.​

1. What words did you circle, cross out, or put a question mark next to on this list? [IF NEEDED, PROBE FOR MEANING OF “XYLITOL,” “CAVITIES,” AND “HUMIDIFIER.”]

2. Based on your own experience, which of these are you most likely to do? What would you add to this list?

Let’s go to the next page.

17. MESSAGE: To maintain good oral health, you should also:​

• Gently brush at least twice a day with fluoride toothpaste.​

• Floss regularly. ​

• Avoid sticky, sugary foods, or brush immediately after eating them.​

• Rinse with water after using an inhaler or taking any syrup-based medicines.​

• Select sugarless cough drops, vitamins, and antacid tablets. ​

• Use a fluoride mouth rinse that does not contain alcohol.​

1. What do you think the bullet with “floss regularly” means? What does “regularly” mean?

2. What does “fluoride,” “antacid tablets” mean? [ASK ABOUT EACH WORD SEPARATELY.]

3. What does “at least twice a day” mean to you? [PROBE FOR SPECIFIC TIMES IN THE DAY, HOURLY INTERVAL BETWEEN BRUSHING TIMES.] Would it be clearer to say, “Brush once in the morning, before breakfast, and a second time in the evening, before dinner?” or “Brush when you wake up and then again about 12–15 hours later”?

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Dry Mouth.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

18. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

19. What do you think of the images? Would you prefer to see photographs or line drawings in this brochure?

20. Is there anything that is hard for you to read or see? Anything else you would change?

Material 3: Oral Cancer (20 minutes subtotal)

We will now review other statements again, one-by-one.

21. MESSAGE: Oral cancer accounts for roughly three percent of all new cancers diagnosed annually in the United States. Approximately 53,000 people will be diagnosed with oral cancer each year and about 10,800 will die from the disease.

    1. What is the main take-home message of this information?

    2. What does “three percent” mean? Can you give me another example of “three percent” using numbers or a proportion? Does “three percent” in this context seem large or small? Why?

    3. What does “annually” mean?

    4. How large or small do the numbers “53,000” and “10,800” in the second sentence seem to you? How surprising is this information?

Let’s go to the next page.

22. MESSAGE: On average, 66 percent of those with the disease will survive 5 years of more.

    1. What does “66 percent” mean? Can you give me another example of “66 percent” using numbers or a proportion?

    2. What does “survive 5 years or more” mean, in your own words?

Let’s go to the next page.

23. MESSAGE: Oral cancer most often occurs in people aged 55-64 and affects more than twice as many men as women.

    1. What is the main take-home message of this information?

    2. What does “more than twice” mean?

    3. What is this information saying about women?

Let’s go to the next page.

24. MESSAGE: HPV. Infection with the sexually transmitted human papillomavirus (specifically the HPV 16 type) has been linked to a subset of oral cancers.

    1. What is the main take-home message of this information?

    2. What “HPV” stand for? What do you think “the HPV 16 type” means?

    3. What does “subset” mean, in your own words?

Let’s go to the next page.

25. MESSAGE: Age. Risk increases with age. Oral cancer most often occurs in people over the age of 40.

    1. What is the main take-home message of this information?

    2. What does “risk” mean, in your own words?

Let’s go to the next page.

26. MESSAGE: See a dentist or physician if any of the following symptoms lasts for more than 2 weeks.

    1. What does “symptoms” mean, in your own words?

    2. What does “lasts for more than 2 weeks” mean, in your own words? Can you give me an example using days of the week?

Let’s go to the next page.

27. MESSAGE: Some parts of the pharynx are not visible during an oral cancer exam.

    1. What does “not visible” mean, in your own words?

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Oral Cancer.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

28. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

29. Would you prefer to see images in this brochure? If so, what types of images?

30. Is there anything that is hard for you to read or see? Anything else you would change?

OBSERVERS’ QUESTIONS AND CLOSE [<5 Minutes]

Now I’d like to ask the observers if they have any additional questions or need clarification on any of the issues we’ve discussed. [Moderator will review any questions from observers via a private communication channel and pose those to the participants.]

Please take a moment to respond to this last question that appears on the computer screen.

31. POLL 3 (POST-TEST): How likely are you to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

1. Why did you rate this question in this manner? Again, please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

NOTE: After the interviewee responds to this question, the moderator will compare the response to the PRE-question with the response to the POST-question and ask: Why do you think your response changed from X to Y to the same question just about 90 minutes ago?

[MODERATOR THANKS INTERVIEWEE AND ENDS THE INTERVIEW.]

OMB No. 0915-0212

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0915-0212 and expires 07/31/2021. This Information Collections Request is voluntary. Public reporting burden for this collection of information is estimated to average 1.5 hours per response, including the time for reviewing instructions and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 18N136B, Rockville, Maryland, 20857.

Exp. Date 7/31/2021

A 90-Minute One-On-One Cognitive Interview

Families and Children

EVALUATION PROTOCOL

November 24, 2020

SESSION OVERVIEW: TOTAL TIME 90 MINUTES	STIMULI
Background (5 minutes)	--
Adult Literacy in Dentistry (10 minutes)	REALD-30 (PowerPoint Slides)
General Attitudes Toward Oral Health and Experiences with Oral Health Care (Including Behavioral Pre-Test) (15 minutes)	Two Poll Questions
Message and Materials Testing (55 minutes)	Two Poll Questions Two videos: (1) Brushing Toddlers’ Teeth (2) Lift the Lip to Prevent Decay (3) Taking Care of Your Baby’s Oral Health brochure (4) Brushing Your Child’s Teeth brochure
Close (Including Behavioral Post-Test) (5 minutes)	One Poll Question

BACKGROUND [5 Minutes]

When the participant joins, the moderator will ask them to be sure they have a pen or pencil with them, as well as the unopened mailed envelope/packet. The moderator will also remind the participant to have their reading glasses and hearing aids available, if they wear them, and ensure they can see the shared computer screen.

• Welcome and thank you for your participation

• My name / independent researcher / not an employee of the Health Resources and Services Administration (HRSA) or the federal government (the sponsor of this project)

• My role (facilitate discussion and end on time at x:xx a.m./p.m.) 

• The purpose of today’s discussion is to:

  • Discuss your thoughts and feelings about health

  • Get your feedback on information that HRSA developed

  • Hear your suggestions for how HRSA could best reach you with important health information

• Some guidance for our conversation:  

  • Please talk in a voice at least as loud as mine.

  • There are no right or wrong answers.

  • This discussion is confidential—first names only / no names will be used in the summary report.

  • Please do not share any information which could be used to identify someone, such as someone’s full name.

  • Your participation is voluntary. You are free to decline to answer a question / free to leave at any time without any penalty.

  • I did not have anything to do with the development of the material we are going to review—if you have constructive criticism, I will not be offended, and I will not be penalized.

    • Please be honest.

  • Our time together is limited—please do not be offended if I have to interrupt you to turn to the next question; there is simply a lot to cover, and we have to finish on time.

  • Please turn your cell phone off or turn the ringtone off.

  • Project staff are listening to the discussion, but we will be the only ones talking during our discussion.

Consent to Audio-Record

• We would like to audio (sound)-record this conversation so that when it comes time to write the summary report, we will have your actual words to ensure accuracy, if needed. However, you will not be identified in the report.

• Although we have our webcams (cameras) on for our live conversation, there will be no video-recording (video-archive) made of our conversation. 

• Do I have your permission to audio (sound)-record this discussion? [Dismiss participant if they do not verbally consent.]

ADULT LITERACY IN DENTISTRY [10 Minutes]

I am now going to show you a list of health-related words on the shared computer screen. As you see each word on the screen, I would like you to read the word out loud. If you do not know how to pronounce that word, please say, "I don’t know." Please do not guess at the pronunciation of a word.

Please note that this activity is not a test of knowledge like in school. YOU are testing words to help make health information easier to understand. This will help us to determine which words to include and not include in health information, so they are understandable to more people. This activity will take just a few minutes.

Note: The moderator and note-taker will time how long it takes each interviewee to complete this activity. The moderator will display one word at a time on the computer screen.

Rapid Estimation of Adult Literacy in Dentistry 30 Word Version

1. Sugar _______ 11. Abscess _______ 21. Periodontal _______

2. Smoking _______ 12. Extraction _______ 22. Sealant _______

3. Floss _______ 13. Denture _______ 23. Hypoplasia _______

4. Brush _______ 14. Enamel _______ 24. Halitosis _______

5. Pulp _______ 15. Dentition _______ 25. Analgesia _______

6. Fluoride _______ 16. Plaque _______ 26. Cellulitis _______

7. Braces _______ 17. Gingiva _______ 27. Fistula _______

8. Genetics _______ 18. Malocclusion _____ 28. Temporomandibular _______

9. Restoration _______ 19. Incipient _______ 29. Hyperemia _______

10. Bruxism _______ 20. Caries _______ 30. Apicoectomy ___

GENERAL ATTITUDES TOWARD ORAL HEALTH AND EXPERIENCES WITH ORAL HEALTH CARE [15 Minutes]

Your perspectives as a person with a child(ren) between 6 months and 3½ years are critically important to what we would like to explore during our discussion today.

Please open the packet you received from us. Inside is an envelope that says, “DO NOT OPEN UNTIL THE START OF THE INTERVIEW.” Please pull out the whole envelope and keep the material in the order that it is in. Do not flip through it. During our discussion, we will look at each piece of material together. Again, please don’t flip ahead.

32. Now look at the top page that has the word “TOP” on it. On this page, please write down the words “oral health” [spell out O-R-A-L] and then write down what these words mean to you. [DISCUSS. PROBE EXTENSIVELY.]

Probes might include any of the following:

• I noticed you included “teeth”—what makes you think of teeth?

• Does “oral health” include any of the following: gums, throat, and bones around the mouth? Are any of these surprising? Why?

  • What, if any, other parts of the body should be included in the broad “oral health” term? Why?

• What do you think people think of when they hear the term “oral health”? Do you think they know what is included within the term “oral health?” Is there a better way to talk about oral health that might make more sense to people? [Offer ideas if participants cannot generate examples on their own, such as “mouth and teeth health.”]

Please take a moment to respond to the question that appears on the computer screen.

33. POLL 1 (PRE-TEST): How likely are you to take your child of 6 months to 3½ years to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

2a. Why did you rate this question in this manner? Please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

34. How often do you take your child(ren) to the dentist for a checkup (that might include a cleaning)? [PROBE EXACT TIME INTERVALS AND IF LESS THAN ONCE A YEAR (EVERY 12 MONTHS).]

    1. What has been your child’s health care provider’s recommendation for how often to take your child to the dentist for a check-up?

    2. Do you follow this recommendation?

       1. [IF NO] What are some of the reasons that you do not follow this recommended “schedule”?

       2. [IF YES] What makes it easy for you to follow this recommended “schedule”?

    3. How often do you think very young children should go to a dentist?

Please take a moment to respond to the question that appears on the computer screen.

35. POLL 2 (PRE-TEST): How important is it to you to brush your child’s teeth every day?

(Reminder: Focus your response to this question on your children 6 months to 3½ years.)

Use a scale from 1 to 7, with 1 being “Not at all important” and 7 being “Extremely important.” [SHOW RESULTS.] Explain your rating response.

MESSAGE AND MATERIALS TESTING [55 Minutes Total]

Note: The moderator will test messages contained within two videos and two materials and will spend about 10–15 minutes on each material. The moderator will play each video via the shared computer screen for participants to see and hear.

Video 1: Brushing Toddlers' Teeth (~15 Minutes Subtotal)

We are going to watch a short 2-minute video called Brushing Toddlers’ Teeth.

While we are watching the video together, please use a stuffed animal, doll, or pillow to act out what you are seeing in the video, while it is happening. [Note: participants will have been invited to consider bringing one of these items to the interview during recruitment, prior to the interview.]

1. [After the video ends] What did you think about how fast or how slow the video is? (Did it go too fast or too slow or was the speed just about right?)

2. What do you think of the length of the video? (Is the video too long, too short, or just about right?) What would be an ideal length for the video?

3. If you had watched this video on your own, how likely would you be to watch it again? Why?

There are several, different positions you can take to brush your toddler's teeth.

4. Please show me one position that the video recommended for how to brush your child's teeth. Again, please feel free to act out one of the recommended positions using the stuffed animal / doll / pillow for this role-playing purpose. Please let me know if you would like to watch the video a second time. [Note: If any participant requests to watch the video again, show the video a maximum of two times.] [The moderator will primarily observe participants' body language and also, but to a lesser degree, verbal explanations. The moderator will watch participants "act out" possible positions (by moving their bodies, arms, hands, or fingers) for brushing a toddler's teeth including [note: there are multiple possible correct answers]: (1) sitting down on a chair and cradling the child in your arm; (2) standing behind child; (3) standing in front of child, looking at the child; and (4) sitting on the floor and having child lay on your lap. The moderator will also observe whether participants add that brushing must occur in a circular motion, and left-to-right or right-to-left.]

[FLEXIBLE PROBES

To encourage the participant to say more: "Tell me more about that", "What makes you say that?", "Is there anything else?", and "Would you mind showing me that again?"

To clarify what the participant just said: "You said the video is confusing—would you tell me which parts are confusing?”

To follow-up on a participant's behavior or body movement: "I noticed that you did—. Can you tell me about that?"]

5. What is the age of the child that this video is intended for? [LISTEN FOR: 2 YEARS OF AGE.]

6. Using your pen or pencil as a “pretend” toothbrush, please show me how you should brush your child(ren)’s teeth. (Remember, this would be a child who is a toddler—or, around 2 years of age.)

7. When should you first start brushing a child’s teeth? [LISTEN FOR: WHEN TEETH FIRST ERUPT.]

   1. Was this information new to you?

   2. How many times should you brush your child(ren)’s teeth? [LISTEN FOR: EACH DAY.]

8. The video talked about using toothpaste with fluoride. What is fluoride?

   1. Why is fluoride important?

   2. The video said to use a rice kernel size for children under age 3. What do they mean by a rice kernel size or a smear? What suggestions, if any, do you have for better explaining this?

   3. The video said to use a pea size amount of toothpaste with fluoride for children beginning at age 3. How would you describe this amount to someone?

      1. How could this instruction be made clearer?

9. What questions do you have after watching the video? What else would you like included in the video? Why?

Please take a moment to respond to the question that appears on the computer screen.

10. POLL 3 (POST-TEST): How important is it to you to brush your child’s teeth every day?

[Moderator to remind participants: Please remember to focus your response to this question on your children 6 months to 3½ years.]

Use a scale from 1 to 7, with 1 being “Not at all important” and 7 being “Extremely important.” [SHOW RESULTS.] Explain your rating response.

Video 2: Lift the Lip to Prevent Decay (~15 Minutes Subtotal)

Now we are going to watch another short 2-minute video called Lift the Lip to Prevent Decay.

While we are watching the video together, again, please use a stuffed animal, doll, or pillow to act out what you are seeing in the video, while it is happening.

11. [After the video ends] What did you think about how fast or how slow this video is? (Did it go too fast or too slow or was the speed just about right?)

12. What do you think of the length of this video? (Is the video too long, too short, or just about right?) What would be an ideal length for this video?

13. If you had watched this video on your own, how likely would you be to watch it again? Why?

14. Please show me what the video said you can do to make sure you are checking your child 6 months to 3½ years for possible change in tooth color or tooth decay. Again, please feel free to act this out using the stuffed animal / doll / pillow. Please let me know if you would like to watch the video a second time. [Note: If any participant requests to watch the video again, show the video a maximum of two times.] [LISTEN/WATCH FOR: LIFT CHILD(REN)’S LIP.]

[The moderator will primarily observe participants' body language and also but to a lesser degree, verbal explanations. The moderator will watch participants "act out" the lifting the lip movement, top and bottom lips, high enough to see the gum line (by moving their bodies, arms, hands, and fingers).]

[FLEXIBLE PROBES

To encourage the participant to say more: "Tell me more about that," "What makes you say that?," "Is there anything else?," and "Would you mind showing me that again?"

To clarify what the participant just said: "You said the video is confusing—would you tell me which parts are confusing?”

To follow-up on a participant's behavior or body movement: "I noticed that you did—. Can you tell me about that?"]

After participants act out the lift-the-lip movements:

15. What is the age of the child that this video is intended for? [LISTEN FOR: 2 YEARS OF AGE.]

16. How often does the video recommend to lift your child's lip in this manner? [LISTEN FOR: ONCE A MONTH.] (Remember, this would be a child who is a toddler—or, around 2 years of age.)

17. What did you learn about where tooth decay begins? [LISTEN FOR: GUM LINE.]

    1. In your own words, please tell me what tooth decay is?

    2. Are there any words you would use instead of “tooth decay” that are clearer? [LISTEN FOR: CAVITIES.]

    3. [IF NEEDED] What is a cavity?

18. What types of things are you looking for when you lift the lip? [LISTEN FOR: DISCOLORATION, DECAY.] What does “discoloration” mean?

19. What would a health care professional recommend that you do if you notice discoloration or decay in your child’s teeth? [LISTEN FOR: SEE ORAL HEALTH CARE PROVIDER.]

20. What suggestions do you have for how this video could be made easier to understand?

Please take a moment to respond to the question that appears on the computer screen.

21. POLL 4: How likely are you to try any of the advice in the Lift the Lip to Prevent Decay video?

Use a scale from 1 to 7, with 1 being “Very likely” and 7 being “Not at all likely.”

[SHOW RESULTS. DISCUSS.] What specifically, of the tips the video shared, are you going to try in the next month?

Print Material 1: Healthy Habits for Happy Smiles: Taking Care of Your Baby’s Oral Health (~15 Minutes Subtotal)

Note: The moderator will display the below instructions on the shared screen for the interviewee to refer to as needed. The moderator and note-taker will closely watch interviewees’ handling of the material—their body language, facial expressions, etc. This may provide clues or insights into ease or difficulty in comprehension. Also, the moderator will display on the shared screen the material that is being referenced at the appropriate time during the cognitive interview discussion.

Please flip to the next item in your package. It is a brochure with the title, Healthy Habits for Happy Smiles: Taking Care of Your Baby’s Oral Health.

Please read this brochure during the next 3 minutes. As you read through it, please circle anything that is important or new information. Cross out anything with an “X” that does not need to be included in the material, or that you may disagree with. If there are things you have questions about or do not understand, put a question mark next to it.

I am now going to ask you about several sentences within the brochure. As we talk about these sentences, please let me know if there were any words or phrases you circled, crossed out, or placed a question mark next to. This is important for me to know. If you do not know the answer to a question I ask, it is totally okay—please let me know and I will go on to the next question. Again, this is not to test you, this activity is for you to test the way health information is currently written. Your honesty will help health writers develop clearer health information.

Note: The moderator will let the participant know which section of the brochure they are referring to and read each sentence (message) aloud. Each section of the brochure will be displayed on the computer screen for the participant to see. The moderator will also probe for words or phrases circled, crossed out, or questioned after EACH sentence (message), as needed.

22. MESSAGE: Healthy primary (baby) teeth help children chew food and speak clearly.

    1. Explain to me what this information means to you.

    2. What does “primary teeth” mean to you?

    3. How would you define “healthy” in the context of this sentence?

    4. What could be changed in this information to make it easier to understand?

23. MESSAGE: Clean your baby’s gums even before you can see the first tooth. Use a clean, damp washcloth to wipe the gums. Do this twice a day, especially after night feeding.

    1. Explain to me what this information means to you.

    2. What does “twice a day” mean to you? Please give me an example of the actual times in any one day that you would do this with your baby?

    3. What does “night feeding” mean to you?

    4. What surprises you about this information?

    5. What questions do you have?

    6. What could be changed in this information to make it easier to understand?

24. MESSAGE: Do not put your baby to bed with a bottle or sippy cup filled with breast milk, infant formula, fruit juice, or other liquids besides water. Liquids that contain sugar can cause tooth decay.

    1. Explain to me what this information means to you.

    2. What is an example of a “liquid that contains sugar”? Do each of the liquids mentioned here (e.g., breast milk, infant formula, fruit juice) contain sugar?

    3. What surprises you about this information?

    4. What questions do you have?

    5. What could be changed in this information to make it easier to understand?

25. MESSAGE: After your baby’s first tooth comes in, use a baby toothbrush with soft bristles and a small head. Put a smear (size of a grain of rice) of fluoride toothpaste on the toothbrush, and brush all sides of your baby’s teeth.

    1. Explain to me what this information means to you.

    2. What does “first tooth” mean?

    3. What is “fluoride toothpaste”?

    4. How well is “smear” defined?

    5. What does “all sides of your baby’s teeth” mean?

    6. What could be changed in this information to make it easier to understand?

“Look and Feel”

26. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

27. Would you prefer to see photographs or line drawings in this brochure?

Print Material 2: Healthy Habits for Happy Smiles: Brushing Your Child’s Teeth (~15 Minutes Subtotal)

Please flip to the next item in your package. It is a brochure with the title, Healthy Habits for Happy Smiles—Brushing Your Child’s Teeth.

Please read this brochure during the next 3 minutes. As you read through it, please circle anything that is important or new information. Cross out anything with an “X” that does not need to be included in the material, or that you may disagree with. If there are things you have questions about or do not understand, put a question mark next to it. [Moderator will display the O, X, ? screen instructions.]

I am now going to ask you about several sentences within the brochure, like last time. As we talk about these sentences, please let me know if there were any words or phrases you circled, crossed out, or placed a question mark next to.

28. MESSAGE: Brush your child’s teeth after breakfast and before bed.

    1. Explain to me what this information means to you

    2. How would you describe “after breakfast” / “before bed”—how would this work in your household?

    3. What could be changed in this information to make it easier to understand?

29. MESSAGE: After brushing, have your child spit out the remaining toothpaste but not rinse. The small amount of toothpaste that stays in the child’s mouth is good for the teeth.

    1. Explain to me what this information means to you.

    2. How much would you say a “small amount of toothpaste” is?

    3. What does “…but not rinse” mean to you?

    4. What surprises you about this information?

    5. What questions do you have after reading this information?

    6. What could be changed in this information to make it easier to understand?

30. MESSAGE: Young children like to do things by themselves. It’s good to let children brush their teeth while an adult watches. But children under age 7 or 8 cannot brush their teeth well yet. An adult needs to brush the child’s teeth too.

    1. Explain to me what this information means to you.

    2. What questions do you have after reading this information?

    3. How realistic is what this information is asking a parent/caregiver to do?

    4. What could be changed in this information to make it easier to understand?

31. MESSAGE: Find a position where your child is comfortable and you can see your child’s teeth while you brush. For example, sit on the floor with your baby’s or young child’s head in your lap. Or stand behind your child in front of a mirror.

    1. Explain to me what this information means to you.

    2. We already saw a video on brushing (the first video). Is there anything new in this information that was not in the video [LISTEN FOR: RECOMMENDATION TO STAND BEHIND THE CHILD IN FRONT OF A MIRROR]?

    3. How realistic is what this information is asking a parent/caregiver to do?

    4. What could be changed in this information to make it easier to understand?

“Look and Feel”

32. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

33. Would you prefer to see photographs or line drawings in this brochure?

OBSERVERS’ QUESTIONS AND CLOSE [<5 Minutes]

Now I’d like to ask the observers if they have any additional questions or need clarification on any of the issues we’ve discussed. [Moderator will review any questions from observers via a private communication channel and pose those to the participants.]

34. POLL 5 (POST-TEST): How likely are you to take your child of 6 months to 3½ years to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

1. Why did you rate this question in this manner? Again, please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

NOTE: After the interviewee responds to this question, the moderator will compare the response to the PRE-question with the response to the POST-question and ask: Why do you think your response changed from X to Y to the same question just about 90 minutes ago?

[MODERATOR THANKS INTERVIEWEE AND ENDS THE INTERVIEW.]

OMB No. 0915-0212

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0915-0212 and expires 07/31/2021. This Information Collections Request is voluntary. Public reporting burden for this collection of information is estimated to average 1.5 hours per response, including the time for reviewing instructions and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 18N136B, Rockville, Maryland, 20857.

Exp. Date 7/31/2021

A 90-Minute One-On-One Cognitive Interview

People with HIV

EVALUATION PROTOCOL

November 23, 2020

SESSION OVERVIEW: TOTAL TIME 90 MINUTES	STIMULI
Background (<5 minutes)	--
Adult Literacy in Dentistry (10 minutes)	REALD-30 (PowerPoint Slides)
General Attitudes Toward Oral Health and Experiences with Oral Health Care (Including Behavioral Pre-Test) (10-15 minutes)	Two Poll Questions
Message and Materials Testing (60 minutes)	(1) Oral Health: Tips for People Living with HIV Trifold Brochure (PowerPoint Slides) Review of material (2) Dry Mouth Messages (PowerPoint Slides) Review of material (3) Brushing and Flossing Messages (PowerPoint Slides) Review of material
Close (Including Behavioral Post-Test) (<5 minutes)	One Poll Question

BACKGROUND [5 Minutes]

When the participant joins, the moderator will ask them to be sure they have a pen or pencil with them, as well as the unopened mailed envelope/packet. The moderator will also remind the participant to have their reading glasses and hearing aids available, if they wear them, and ensure they can see the shared computer screen.

• Welcome and thank you for your participation

• My name / independent researcher / not an employee of the Health Resources and Services Administration (HRSA) or the federal government (the sponsor of this project)

• My role (facilitate discussion and end on time at x:xx a.m./p.m.) 

• The purpose of today’s discussion is to:

  • Discuss your thoughts and feelings about health

  • Get your feedback on information that HRSA developed

  • Hear your suggestions for how HRSA could best reach you with important health information

• Some guidance for our conversation:  

  • Please talk in a voice at least as loud as mine.

  • There are no right or wrong answers.

  • This discussion is confidential—first names only / no names will be used in the summary report.

  • Please do not share any information which could be used to identify someone, such as someone’s full name.

  • Your participation is voluntary. You are free to decline to answer a question / free to leave at any time without any penalty.

  • I did not have anything to do with the development of the materials we are going to review—if you have constructive criticism, I will not be offended, and I will not be penalized.

    • Please be honest.

  • Our time together is limited—please do not be offended if I have to interrupt you to turn to the next question; there is simply a lot to cover, and we have to finish on time.

  • Please turn your cell phone off or turn the ringtone off.

  • Project staff are listening to the discussion, but we will be the only ones talking during our discussion.

Consent to Audio-Record

• We would like to audio (sound)-record this conversation so that when it comes time to write the summary report, we will have your actual words to ensure accuracy, if needed. However, you will not be identified in the report.

• Although we have our webcams (cameras) on for our live conversation, there will be no video-recording (video-archive) made of our conversation. 

• Do I have your permission to audio (sound)-record this discussion? [Dismiss participant if they do not verbally consent.]

ADULT LITERACY IN DENTISTRY [10 Minutes]

I am now going to show you a list of health-related words on the shared computer screen. As you see each word on the screen, I would like you to read the word out loud. If you do not know how to pronounce that word, please say, "I don’t know." Please do not guess at the pronunciation of a word.

Please note that this activity is not a test of knowledge like in school. YOU are testing words to help make health information easier to understand. This will help us to determine which words to include and not include in health information, so they are understandable to more people. This activity will take just a few minutes.

Note: The moderator and note-taker will time how long it takes each interviewee to complete this activity. The moderator will display one word at a time on the computer screen.

Rapid Estimation of Adult Literacy in Dentistry: 30 Word Version

1. Sugar _______ 11. Abscess _______ 21. Periodontal _______

2. Smoking _______ 12. Extraction _______ 22. Sealant _______

3. Floss _______ 13. Denture _______ 23. Hypoplasia _______

4. Brush _______ 14. Enamel _______ 24. Halitosis _______

5. Pulp _______ 15. Dentition _______ 25. Analgesia _______

6. Fluoride _______ 16. Plaque _______ 26. Cellulitis _______

7. Braces _______ 17. Gingiva _______ 27. Fistula _______

8. Genetics _______ 18. Malocclusion _____ 28. Temporomandibular _______

9. Restoration _______ 19. Incipient _______ 29. Hyperemia _______

10. Bruxism _______ 20. Caries _______ 30. Apicoectomy ___

GENERAL ATTITUDES TOWARD ORAL HEALTH AND EXPERIENCES WITH ORAL HEALTH CARE [15 Minutes]

Now, I’d like you to answer the question that appears on your computer screen. We’ll come back to this question later in the conversation.

36. POLL 1 (PRE-TEST): How likely are you to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

1a. Why did you rate this question in this manner? Please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

Please open the packet you received from us. Inside is an envelope that says, “DO NOT OPEN UNTIL THE START OF THE INTERVIEW.” Please pull out the whole envelope and keep the material in the order that it is in. Do not flip through it. During our discussion, we will look at each piece of material together. Again, please don’t flip ahead.

37. Now look at the top page that has the word “TOP” on it. On this page, please write down the words “oral health” [spell out O-R-A-L] and then write down what these words mean to you. [DISCUSS. PROBE EXTENSIVELY.]

Probes might include any of the following:

• I noticed you included “teeth”—what makes you think of teeth?

• Does “oral health” include any of the following: gums, throat, and bones around the mouth? Are any of these surprising? Why?

  • What, if any, other parts of the body should be included in the broad “oral health” term? Why?

• What do you think people think of when they hear the term “oral health”? Do you think they know what is included within the term “oral health?” Is there a better way to talk about oral health that might make more sense to people? [Offer ideas if participants cannot generate examples on their own, such as “mouth and teeth health.”]

Please take a moment to respond to the question that appears on the computer screen.

38. POLL 2: How concerned are you about your oral health? 

Use a scale from 1 to 7, with 1 being “Not at all concerned” and 7 being “Extremely concerned.” 

[SHOW RESULTS. DISCUSS. PROBE REASONS.]

1. How does your rating response apply specifically to your HIV diagnosis? 

2. Has oral health become more important to you since you were diagnosed with HIV? Why or why not? In what ways? For example, what do you differently?

3. Do you believe “good” or “bad” oral health can affect your ENTIRE (THE REST OF YOUR) body as well as your mouth?

   1. [IF NO] Why not?

   2. [IF YES] How does “good” or “bad” oral health affect your ENTIRE (THE REST OF YOUR) body as well as your mouth?

39. How often do you go to the dentist for a cleaning or checkup? [PROBE FOR MEANING (e.g., what does the participant consider “cleaning”/ what does a “check-up” encompass?; if the participant says, “Once a year,” ask, “If it’s January 2021 now, what month and year would that be in this scenario?”; if the respondent says, “Two times a year,” ask, “Explain to me which months of the year you would visit at this frequency? How many months are in between each visit?”]

MESSAGE AND MATERIALS TESTING [60 Minutes Total]

Note: The moderator will test messages contained within three materials and will spend approximately 20 minutes on each material.

Material 1: Oral Health: Tips for People Living with HIV (20 Minutes)

Note: The moderator will display the below instructions on the shared screen for the interviewee to refer to as needed. The moderator and note-taker will closely watch the participant’s handling of the material—their body language, facial expressions, etc. This may provide clues or insights into ease or difficulty in comprehension. Also, the moderator will display on the shared screen the material that is being referenced at the appropriate time during the cognitive interview discussion.

I am now going to read various statements, one-by-one, on the shared screen aloud. In your packet labeled X, you will also find these messages on paper. As we move along each statement, please use your pen or pencil to put a question mark next to any words or information that you find confusing, circle anything that you like, and cross out anything you dislike. Let’s start with the page that reads, “Oral health is…” As we talk about these sentences, again, please let me know if there were any words or phrases you circled, crossed-out, or placed a question mark next to. This is important for me to know. Also, if you do not know the answer to a question I ask, it is totally okay—please let me know and I will go on to the next question. Again, this is not to test you, this activity is for you to test the way health information is currently written. Your honesty will help health writers develop clearer health information.

40. MESSAGE: Oral health is the health of your mouth, including your teeth, gums, throat, and the bones around the mouth.

    1. What does this sentence mean to you? [PROBE FOR MEANING OF “ORAL HEALTH IS THE HEALTH OF YOUR MOUTH.”]

    2. [IF NEEDED] What could be changed in this statement to make it easier to understand? [PROBE FOR ALTERNATIVES TO “ORAL HEALTH.”]

Let’s go to the next page.

41. MESSAGE: See your dentist regularly for check-ups and cleanings.

    1. Tell me what this sentence means to you in your own words.

    2. How would you define “regularly” in this sentence? What do you think the writers of this sentence meant by “regularly” in this sentence?

    3. What does “check-ups” mean to you? What does “cleanings” mean to you?

    4. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

42. MESSAGE: Brushing is necessary for removing dental plaque, a sticky film of bacteria. Plaque buildup can cause tooth decay, gum disease, and bad breath.

    1. Explain to me what this sentence means to you.

    2. How well is “dental plaque” defined in this sentence?

       1. What does “bacteria” mean to you?

       2. What words would you use to better describe “dental plaque”?

       3. What does “tooth decay” mean? What does “gum disease” mean to you? What does “bad breath” mean to you?

    3. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

43. MESSAGE: Flossing cleans plaque from the parts of your teeth that your toothbrush can’t reach.

    1. What does this sentence mean to you?

    2. What does “flossing” mean to you?

    3. What areas in the mouth do you think the writers were referring to when they say where “your toothbrush can’t reach”?

    4. What surprises you about this sentence?

    5. What questions do you still have?

    6. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

44. MESSAGE: Take all your HIV medications on schedule to protect your immune system and help prevent oral infections.

    1. What does this sentence mean in your own words?

    2. What does “HIV medications” mean to you?

    3. What does “on schedule” mean to you?

    4. What does “your immune system” mean to you”?

    5. What does “oral infections” mean to you?

    6. What surprises you about this sentence?

    7. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

45. MESSAGE: Don’t smoke. Your health care provider or dentist can help you quit.

    1. What does “health care provider” mean to you? [PROBE FOR WHO THEY CONSIDER TO BE “HEALTH CARE PROVIDERS.”]

    2. When you see “smoke” in this sentence, what does that make you think of?

    3. What does “quit” mean in this sentence?

    4. What surprises you about this sentence?

    5. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

46. MESSAGE: Your mouth reflects your overall health.

    1. What does this sentence mean?

    2. What does “overall” in “overall health” mean in your own words?

    3. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

47. MESSAGE: Most oral problems are preventable and easily treated if found early.

    1. Tell me what this sentence means.

    2. What is an “oral problem” in your own words?

    3. What does “preventable” mean in this sentence?

    4. What does “treated” mean in this sentence?

    5. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

48. MESSAGE: People with human immunodeficiency virus (HIV) are at special risk for oral health problems.

    1. Tell me what this sentence means to you.

    2. Define “special risk” for me.

    3. What are your reactions to the word “special” in this sentence? [LISTEN FOR: OFFENSIVE OR APPROPRIATE.]

    4. What words or term would you suggest using instead of “special risk”? [PROBE FOR “AT RISK,” “AT HIGHER RISK,” “GREATER CHANCE FOR,” “GREATER PROBABILITY FOR,” AND/OR OTHER ALTERNATIVES FOR THE WORD “SPECIAL.”]

    5. What are some examples, if you know of any, of these “special risks”? What might these look like?

    6. [IF NEEDED] What could be changed in this statement to make it easier to understand?

Let’s go to the next page.

People with human immunodeficiency virus (HIV) are at special risk for oral health problems…

• Chronic dry mouth

• Bleeding gums

• Loose teeth

• Bone loss around the teeth

• Cancer (Kaposi’s sarcoma)

• Oral warts

• Canker sore/fever blisters

• Thrush (Candida/yeast)

• Dental Cavities

49. What does each of these terms mean, in your own words?

50. For each of these terms, would you use a different term or have a different term that you would use instead that is easier to understand? [PROBE FOR ALTERNATIVE WORDS/PHRASES (e.g., “blood from gums” instead of “bleeding gums”).]

51. From the perspective of a person with HIV, what oral health problems are missing from this list?

52. What questions come to mind after reading this list? [LISTEN FOR: WANT MORE INFORMATION ABOUT A PROBLEM LISTED—SUCH AS HOW DOES THE PROBLEM MANIFEST ITSELF, IS IT LIKELY A RESULT OF HIV OR FROM TREATMENTS FOR HIV, WHAT ARE TYPICAL TYPES OF TREATMENTS FOR EACH, DEFINITIONS/GLOSSARY FOR EACH TERM, LINKS TO MORE INFORMATION ON EACH.]

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Oral Health: Tips for People Living with HIV.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

53. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

54. What do you think of the images? Would you prefer to see photographs or line drawings in this brochure?

55. Is there anything that is hard for you to read or see? Anything else you would change?

Material 2: Dry Mouth (20 Minutes Subtotal)

We will now review other statements again, one-by-one.

56. MESSAGE: Dry mouth is the feeling that there is not enough saliva in the mouth. Common causes of dry mouth include side effects of certain medications and dehydration, when you lose more fluid than you take in.

    1. Explain to me what this sentence is trying to say.

    2. In your own words, what is “saliva”? “Dehydration”? “Fluid”? [ASK ABOUT EACH WORD SEPARATELY.]

    3. What does “side effects” mean to you in this sentence? What is an example of a side effect from a medication?

Let’s go to the next page.

57. MESSAGE: Dry mouth can make it hard to chew, swallow, or even talk. Having less saliva also increases the risk of developing tooth decay or fungal infections in the mouth since saliva helps keep harmful germs in check.

    1. Explain to me what this sentence is trying to say.

    2. In your own words, explain what “increases the risk” means?

    3. What does “tooth decay” mean? “Fungal infections”? “Germs”? [ASK ABOUT EACH WORD SEPARATELY.]

Let’s go to the next page.

58. MESSAGE: Hundreds of medicines can cause the salivary glands to make less saliva. ​

3. Tell me what “hundreds” means in this sentence? Please give a specific number example to explain what the brochure writers meant by “hundreds”?

4. What does “saliva” / “salivary glands” mean? [ASK ABOUT EACH WORD SEPARATELY.]

Let’s go to the next page.

59. MESSAGE: Diabetes, Sjogren’s syndrome, and HIV/AIDS can all cause dry mouth.

    1. Did you know that diabetes and HIV/AIDS can cause dry mouth?

    2. Have you ever heard of “Sjogren’s syndrome”? If so, where and when did you hear about it and what does it mean? How helpful would it be to include a definition of “Sjogren’s syndrome”?

    3. After reading this sentence, what questions do you have?

Let’s go to the next page.

60. MESSAGE: For dry mouth, you may do the following:​

• Sip water or sugarless drinks often especially during meals. This will make chewing and swallowing easier. It may also improve the taste of food. ​

• Avoid drinks with caffeine, such as coffee, tea, and some sodas. Caffeine can dry out the mouth.​

• Chew sugarless gum or suck on sugarless hard candy to stimulate saliva flow—citrus, cinnamon, or mint-flavored candies are good choices. Some sugarless chewing gums and candies contain xylitol and may help prevent cavities. ​

• Don’t use tobacco or alcohol. They dry out the mouth.​

• Be aware that spicy or salty foods may cause pain in a dry mouth. ​

• Use a humidifier at night.​

3. What words did you circle, cross out, or put a question mark next to on this list? [IF NEEDED, PROBE FOR MEANING OF “XYLITOL,” “CAVITIES,” AND “HUMIDIFIER.”]

4. Based on your own experience, which of these are you most likely to do? What would you add to this list?

Let’s go to the next page.

61. MESSAGE: To maintain good oral health, you should also:​

• Gently brush at least twice a day with fluoride toothpaste.​

• Floss regularly. ​

• Avoid sticky, sugary foods, or brush immediately after eating them.​

• Rinse with water after using an inhaler or taking any syrup-based medicines.​

• Select sugarless cough drops, vitamins, and antacid tablets. ​

• Use a fluoride mouth rinse that does not contain alcohol.​

4. What do you think the bullet with “floss regularly” means? What does “regularly” mean?

5. What does “fluoride“ /” antacid tablets” mean? [ASK ABOUT EACH WORD SEPARATELY.]

6. What does “at least twice a day” mean to you? [PROBE FOR SPECIFIC TIMES IN THE DAY, HOURLY INTERVAL BETWEEN BRUSHING TIMES.] Would it be clearer to say, “Brush once in the morning, before breakfast, and a second time in the evening, before dinner?” or “Brush when you wake up and then again about 12–15 hours later”?

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Dry Mouth.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

62. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

63. What do you think of the images? Would you prefer to see photographs or line drawings in this brochure?

64. Is there anything that is hard for you to read or see? Anything else you would change?

Material 3: Brushing and Flossing (20 Minutes Subtotal)

We will now review other statements again, one-by-one.

65. MESSAGE: Oral health is important for people of all ages. A healthy mouth helps people enjoy their food, chew better, eat well, and avoid pain and tooth loss. With good oral hygiene and regular visits to the dentist, you can maintain your oral health for years.

5. What does “hygiene” mean to you? What does “oral hygiene” mean to you?

   • Can you think of an alternate term for “oral hygiene”?

6. What does “regular” mean in this sentence?

7. Take a look at the first sentence, “Oral health is important for people of all ages.” How believable is that? Why?

8. Take a look at the last few words, “…you can maintain your oral health for years.” What does “maintain” mean to you?

Let’s go to the next page.

66. MESSAGE: Flossing removes dental plaque between teeth where a toothbrush can’t reach. If not removed, dental plaque can build up and cause tooth decay and gum disease.

    1. What does “dental plaque,” “tooth decay,” and “gum disease” mean to you? [ASK ABOUT EACH WORD SEPARATELY.]

    2. What does the sentence mean by “build up”?

    3. What areas in the mouth do you think the writers were referring to when they say, “where a toothbrush can’t reach”?

Let’s go to the next page.

67. MESSAGE: Flossing Tools

NOTE: The moderator will show four images of flossing tools on the shared screen.

1. What is the meaning of “floss holder” / “floss threader” / “oral irrigator” / “interdental brush”? [ASK ABOUT EACH WORD SEPARATELY.]

2. How well did the photographs of each flossing tool help you identify each tool?

3. Which tool(s) have you seen before? Are there any tools you have never seen before? Which ones?

Let’s go to the next page.

68. MESSAGE: Brushing Your Teeth. Gum Disease—If plaque is not removed, some of it can harden below the gum line and irritate the gums. The gums become red, swollen, and may bleed easily. These are signs of gingivitis. Gingivitis is a mild form of gum disease, and you can usually reverse it with daily brushing and flossing. If there are areas in your mouth where your gums have pulled away from the teen (called gum recession), the exposed tooth roots can decay as well.

4. What is the main take-home message of this information?

5. Using your own words, please define “gum disease” / “gingivitis” / “gum recession” / “exposed tooth roots.”

6. What does “daily” mean in, “Gingivitis is a mild form of gum disease, and you can usually reverse it with daily brushing and flossing”?

Hard-copy Material

Now, please open envelope X and take out the brochure titled, “Brushing and Flossing.” All of the information we just reviewed is in this brochure. Now that you have the physical brochure in your hands:

69. What do you think of the design of the brochure? What's one word you'd use to describe the design or its “look and feel”? What do you like? What do you not like? Why? [Probe: colors, images, layout, font type, font size, quality/texture of paper, amount of words compared to white space, length, section headers.]

70. What do you think of the images? Would you prefer to see photographs or line drawings in this brochure?

71. Is there anything that is hard for you to read or see? Anything else you would change?

OBSERVERS’ QUESTIONS AND CLOSE [<5 Minutes]

Now I’d like to ask the observers if they have any additional questions or need clarification on any of the issues we’ve discussed. [Moderator will review any questions from observers via a private communication channel and pose those to the participants.]

72. POLL 3 (POST-TEST): How likely are you to visit the dentist in the next 6 months?

Use a scale from 1 to 7, with 1 being “Not at all likely” and 7 being “Extremely likely.”

1. Why did you rate this question in this manner? Again, please talk about the extent to which your rating relates to the COVID-19/coronavirus pandemic. [Listen for/probe: If a number is low, is the number low because of the pandemic or primarily other reasons?]

NOTE: After the interviewee responds to this question, the moderator will compare the response to the PRE-question with the response to the POST-question and ask: Why do you think your response changed from X to Y to the same question just about 90 minutes ago?

[MODERATOR THANKS INTERVIEWEE AND ENDS THE INTERVIEW.]

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0915-0212 and expires 07/31/2021. This Information Collections Request is voluntary. Public reporting burden for this collection of information is estimated to average 1.5 hours per response, including the time for reviewing instructions and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 14N136B, Rockville, Maryland, 20857.

38