Vendor Cognitive Testing Report Example

Attachment 5: Example of Vendor Cognitive Testing Report

COGNITIVE TESTING of the BRFSS Hepatitis Questions

Top Line Summary Report

Background/Purpose

The purpose of this study is to conduct cognitive interviews to test questions about hepatitis for BRFSS. These cognitive interviews were conducted to learn about how adults interpret, understand and respond to a set of questions and response options proposed for inclusion in the BRFSS survey used by CDC to collect data about the health of adults throughout the United States.

Cognitive testing is a technique used during survey design to gain a sharper understanding of underlying comprehension or retrieval problems among the respondent population. The results of cognitive interviews may lead to revisions of survey questions and changes in question order so as to enhance response rates and data quality. In a cognitive interview, an interviewer administers draft survey questions to a respondent while collecting additional verbal information about the survey responses with the goal of evaluating the quality of the responses or to determine if the wording is generating the intended information.¹

This topline summary report presents the initial findings from nine cognitive interviews. All interviews were conducted by a trained interviewer using a CDC-approved, semi-structured interview guide. They averaged 30 minutes in length and were audio-recorded. In person interviewees were required to review and sign an informed consent statement. Phone interviewees were required to verbally consent to the interview after being read an informed consent statement.

Each participant was invited to respond to all the items in the guide, in the order the items appeared. Throughout the interview, ICF’s trained qualitative research specialist noted respondent comprehension issues, think-aloud processes, and additional feedback. This report summarizes feedback for each item.

Recruitment

Adults who have been diagnosed with hepatitis B or C and adults who have been treated for hepatitis C were recruited to participate in cognitive interviews. As the incidence of hepatitis in the general population of Vermont (site of in-person interviews) is relatively low, and treatment for hepatitis can be a potentially sensitive subject matter, a variety of recruitment methods were utilized, including Craigslist advertisements, flyers posted at a dialysis clinics, and outreach to an online hepatitis support group. ICF staff members were not eligible to participate, but were asked to inform friends and family who met these criteria about the possibility of participating in the study, resulting in one participant recruited through personal networks. Potential respondents who are unable to participate in an in-person interview in Burlington, VT were able to participate in a telephone interview. All respondents received an incentive of $50 for participating in the in-person or telephone interviews.

Respondents

Cognitive testing was conducted with nine individuals, including five females and four males. Four participants had been diagnosed with hepatitis B, and 6 participants had been diagnosed with hepatitis C. Of the selected females, two women were between the ages of 18-40 years old, one was 41-64 years old, and two were 65 or older. One female self-identified as white non-Hispanic, one identified as white Hispanic, two self-identified as Black or African American, and one identified as American Indian or Alaska Native. Two men were between the ages of 18 and 40 years old, and one was 41-64 years old, and the other two were aged 65 or older. One self-identified as white non-Hispanic, one self-identified as Hispanic, one self-identified as Asian, and one self-identified Native Hawaiian or Pacific Islander. As shown in Tables 1 and 2 below, participants represented a wide range of incomes and education levels.

Table 1: Participant Income

Income	Number of Respondents	Percentage
Under $25,000	3	33%
$25,000 – $34,999
$35,000 - $49,999	1	11%
$50,000 - $74,999	4	44%
$75,000 or above	1	11%
Skipped
Total:	9	100%

Table 2: Participant Education

Education Attained	Number of Respondents	Percentage
Some High school or less
High school graduate (or equivalent)	2	22%
Some college or technical school (1-4 years, no degree)	2	22%
College graduate (e.g., BA, BS, etc.)	4	44%
Graduate degree (e.g., MA, MS, PhD, etc.)	1	11%
Medical degree (e.g., RN, NP, MD, DDS)
Skipped
I do not wish to answer
	9	100%

Overall Findings

All questions were asked of all nine participants. Comprehension was very high for all the questions and all the different response options. Respondents found the questions to be clearly worded and easy to follow.

The main challenges respondents faced when answering these questions include:

1. Respondent lack of clarity around diagnosis. Three respondents had questions about their own diagnosis, which created some difficulty with answering questions. Two of these respondents tested positive for hepatitis antibodies, but did not have the hepatitis virus. These respondents were not sure how to answer questions 1, 3, and 6.

Recommendation: We recommend that help language be included in the instrument to clarify whether individuals testing positive for hepatitis antibodies (not the virus) should answer yes to this question. We also recommend including a response option for “not sure whether hepatitis A, B, or C”

2. Respondent recall issues around vaccines. Four out of the nine participants had difficulty recalling whether they had received the hepatitis B or A vaccine. While respondents gave an initial “yes” or “no” response to this question, when probed respondents indicated that they were not sure, or couldn’t remember.

Recommendation: We recommend providing guidance to help respondents identify events that would likely trigger vaccination (born after the year when the vaccination became available; received prior to starting college; provided by employer due to occupational exposure, etc.)

Findings by Question

MODULE 1 – Cognitive Testing of Emerging Core-Hepatitis

1. Has a doctor, nurse, or other health professional ever told you that you had hepatitis C?

READ IF RESPONDENT ASKS A QUESTION: Hepatitis C is an infection of the liver from the hepatitis C virus.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

• Many participants asked the interviewer to repeat whether they had said hepatitis “B” or “C” as it can be difficult to differentiate these letters over the phone, even with the intentional emphasis by the interviewer.

• Some participants had been told that they had hepatitis C, but questions around their diagnosis made them unsure whether it was accurate to answer “yes” to this question. For example, one participant stated that they were told they had hepatitis C “antibodies,” but follow up lab work was “inconclusive.” Another participant stated that at the time of their diagnosis, there was no differentiation between hepatitis A, B, and C. Although they believed they had hepatitis C, they could not be certain.

• All participants found the wording of this question to be clear.

Clarity/Comprehension: High

Recommendations:

• In the intro, have the interviewer state “C as in Charlie.”

• Two of the 9 participants (one with hepatitis C, one with hepatitis B [see Question 3]) cited instances where testing indicated that presence of hepatitis antibodies, but not the presence of the hepatitis virus. As the presence of antibodies may indicate that the individual may have contracted hepatitis at one point, participants were not sure whether they should answer “yes” to this question. Both participants raised this question without probing from the interviewer. As such, we recommend that help language be included in the instrument to clarify whether individuals testing positive for hepatitis antibodies (not the virus) should answer yes to this question.

• Include a response option for “not sure whether hepatitis A, B, or C”

2. Were you ever prescribed medicine to treat hepatitis C?

READ IF RESPONDENT IS NOT SURE: Many different types of medicine have been used to treat hepatitis C including injections and oral medicine.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

There were no issues encountered with this question. Participants found the wording to be clear and were very sure of their responses.

Clarity/Comprehension: High

Recommendation: Keep the question worded as is.

Thank you for answering those questions. Now I would like to ask you some questions about hepatitis B.

3. Has a doctor, nurse, or other health professional ever told you that you had hepatitis B?

READ IF RESPONDENT ASKS A QUESTION: Hepatitis B is an infection of the liver from the hepatitis B virus.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

• Many participants asked the interviewer to repeat whether they had said hepatitis “B” or “C” as it can be difficult to differentiate these letters over the phone, even with the intentional emphasis by the interviewer.

• One participant stated that at the time of their diagnosis, there was no differentiation between hepatitis A, B, and C. Although they believed they had hepatitis C, they could not be certain.

• One participant stated that tests indicated the presence of hepatitis B antibodies. They expressed uncertainty as to whether they should answer yes to this question (as they may have been exposed to hepatitis B in the past) or no (because of the presence of antibodies indicated that they are immune).

• There were no issues with question wording or participant comprehension.

Clarity/Comprehension: High

Recommendation:

• In the intro, have the interviewer state “B as in boy.”

• Two of the 9 participants (one with hepatitis C [see Question 1], one with hepatitis B) cited instances where testing indicated that presence of hepatitis antibodies, but not the presence of the hepatitis virus. As the presence of antibodies may indicate that the individual may have contracted hepatitis at one point, participants were not sure whether they should answer “yes” to this question. Both participants raised this question without probing from the interviewer. As such, we recommend that help language be included in the instrument to clarify whether individuals testing positive for hepatitis antibodies (not the virus) should answer yes to this question.

• Include a response option for “not sure whether hepatitis A, B, or C”

4. Are you currently taking medicine to treat hepatitis B?

READ IF RESPONDENT IS NOT SURE: Many different types of medicine can be used to treat hepatitis B including injections and oral medicine.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

There were no issues encountered with this question. Participants found the wording to be clear, and were very sure of their responses.

Clarity/Comprehension: High

Recommendation: Keep the question worded as is.

MODULE 2 – Cognitive Testing of Hepatitis Treatment and Vaccination

5. Were you treated for hepatitis C in 2015 or after?

READ IF RESPONDENT IS NOT SURE: Most hepatitis C treatments offered in 2015 or after were oral medicines or pills, including Harvoni, Viekira, Zepatier, Epclusa, and others.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

There were no issues encountered with this question. Participants found the wording to be clear and were very sure of their responses. When asked to repeat the question in their own words, all participants correctly referenced the “2015 or after” timeline.

Clarity/Comprehension: High

Recommendation: Keep the question worded as is.

6. Were you treated for hepatitis C prior to 2015?

READ IF RESPONDENT IS NOT SURE: Most hepatitis C treatments offered prior to 2015 were injections or shots and pills given weekly or more often over many months.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

There were no issues encountered with this question. Participants found the wording to be clear, and were very sure of their responses. When asked to repeat the question in their own words, all participants correctly referenced the “prior to 2015” timeline.

Clarity/Comprehension: High

Recommendation: Keep the question worded as is.

7. Do you still have hepatitis C?

READ IF RESPONDENT IS NOT SURE: You may still have hepatitis C and feel healthy. Your blood must be tested again to tell if you still have hepatitis C.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

• One participant who had questions about their original diagnosis was also not sure whether they still have hepatitis C, as antibodies are present and follow up testing was “inconclusive.”

• All participants who received treatment after being diagnosed with hepatitis C were very sure of their response, as they received regular follow-up blood tests and discussed the results with their physicians.

• There were no issues with question wording or participant comprehension.

Clarity/Comprehension: High

Recommendation: Keep the question worded as is. Clarification on questions related to diagnosis and the existence of antibodies should resolve the questions raised by this participant.

Now I would like to ask you some questions about hepatitis vaccines. Vaccines are for hepatitis B and hepatitis A.

8. Have you ever received the hepatitis B vaccine? The hepatitis B vaccine is completed after the third shot is given.

INTERVIEWER NOTE: Response is “Yes” only if respondent has received the entire series of three shots.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

• Four out of the 9 respondents were not sure of their answers to this question:

  • One respondent answered “pretty sure,” but with some hesitation. This respondent had campaigned for access to this vaccination due to work hazards. Although the details surrounding the vaccination were very clear to her, she stated that she “kind of” remembered receiving the vaccination, but that it shows up on her vaccination card.

  • Three respondents said “no,” but after probing seemed less sure of their answers, indicating that they may have received it as a child or as an adult. While their initial answer was “no,” they were not sure of their answer.

• Another respondent said “yes” before the interviewer could finish the question. When the second half of the question was stated, the respondent then clarified that they had received 2 out of the 3 shots.

• Respondents noted that the information about the number of shots given to complete the vaccine was informative and helpful.

Clarity/Comprehension: High

Recommendation:

• Since many respondents answered this question before it could be finished, we recommend reordering this question as follows: The hepatitis B vaccine is completed after the third shot is given. Have you completed the hepatitis B vaccine?

• 4 out of the 9 respondents had difficulty remembering whether they had received this vaccine. These respondents answered “yes” or “no” rather than “don’t know” to this question, but then indicated that they were unsure of their answer after probing. We recommend providing guidance to help respondents identify events that would likely trigger vaccination (born after year when vaccination became available; received prior to starting college; provided by employer due to occupational exposure, etc.)

9. Have you ever received the hepatitis A vaccine? The hepatitis A vaccine is completed after the second shot is given.

INTERVIEWER NOTE: Response is “Yes” only if respondent has received two shots.

1. Yes

2. No

3. Don’t know / Not sure

4. Refused

Respondents experienced the same issues with recall for this question as with question 8.

Clarity/Comprehension: High

Recommendation:

• Since many respondents answered this question before it could be finished, we recommend reordering this question as follows: The hepatitis A vaccine is completed after the second shot is given. Have you completed the hepatitis A vaccine?

• We recommend providing guidance to help respondents identify events that would likely trigger vaccination (born after year when vaccination became available; received prior to starting college; provided by employer due to occupational exposure, etc.)

WRAP UP

1. How comfortable were you sharing this information about yourself?

All respondents felt comfortable sharing this information about themselves. Several mentioned that they felt reassured by promises of confidentiality.

2. Would you be comfortable sharing this information about someone in your household?

All but one respondent indicated that they felt comfortable sharing this information about people in their household.

3. Is there anything else about the survey questions and responses that you would like share with me?

The only additional feedback came from respondents who indicated that they wanted to see more questions to be able to share more information about their experience (including treatment).

1 Beatty PC (2003). “Answerable Questions: Advances in the Methodology for Identifying and Resolving Questionnaire Problems in Survey Research.” Doctoral Dissertation, University of Michigan in Dissertation Abstracts International 64(09): 3504A.

8