HCPS Round 2 Cognitive Interview Report

August 2019

2019 Health Center Patient Survey

Cognitive Interview Findings – Round 2

Draft: August 30, 2019

Prepared for

Alissa Harvey

Health Resources and Services Administration

Bureau of Primary Health Care

Parklawn Building, 5600 Fishers Lane

Rockville, MD 20857

Prepared by

RTI International

3040 E. Cornwallis Road

Research Triangle Park, NC 27709

RTI Project Number 0214097.001.100.001.007

1. Methods

Cognitive interviews are one-on-one interviews used to assess the questionnaire in terms of general understanding, question and response wording, skip logic, and visual aids. The goal is to gain an understanding of how well the questions work when administered to a sample of the survey’s target population. During the first round of interviews, the interview follows a cognitive protocol with pre-scripted probes. Cognitive interviewers also use spontaneous probes to gain a better understanding of questions. During the second round of cognitive interviews, we conducted the entire instrument from beginning to end in order to collect timing data. Interviewers noted if the participant had any challenges with questions but were instructed to move forward with the interview. If time allowed, interviewers could revisit these items at the conclusion of the interview.

1a. Recruited and Screened Participants

A total of 23 cognitive interviews were conducted during the second round of interviews. We conducted an additional 8 interviews using the round one protocol. These interviews were not completed prior to delivery of the round one report. All attempts were made to diversify the sample in terms of age, gender, and race (see table 1a-1e). Completed round two interviews included 5 in English, 6 in Spanish, 4 in Tagalog, 4 in Chinese, and 4 in Vietnamese. The table includes the timing of each round two interview.

Table 1a: Distribution of Interview Participants – English

R	R Age	Type of Interview	IF PROXY- Gender and age of child	R Gender	R Race	Interview Timing
1	23	Adult		Female	White	70 minutes
2	22	Adult		Male	White	55 minutes
3	13	Adolescent		Female	White	45 minutes
4	41	Adult		Female	AA	72 minutes
5	44	Adult		Female	AA	56 minutes

Table 1b: Distribution of Interview Participants – Spanish

R	R Age	Type of Interview	IF PROXY- Gender and age of child	R Gender	R Race	Interview Timing
1	40	Adult		Female	Mexican	100 minutes
2	55	Adult		Female	El Salvadoran	70 minutes
3	14	Adolescent		Male	Mexican	45 minutes
4	49	Adult		Female	Puerto Rican	65 minutes
5	47	Adult		Female	Mexican	60 minutes
6	15	Adolescent		Female	Columbian	40 minutes

Table 1c: Distribution of Interview Participants – Tagalog

R	R Age	Type of Interview	IF PROXY- Gender and age of child	R Gender	R Race	Interview Timing
1*	45	Adult		Female	Filipino	-
2*	65	Adult		Female	Filipino	-
3*	64	Adult		Male	Filipino	-
4	28	Proxy	1	Female	Filipino	90 minutes
5	72	Adult		Male	Filipino	125 minutes
6	22	Adult		Male	Filipino	115 minutes
7	50	Adult		Female	Filipino	120 minutes

*Interview conducted using R1 protocol – Interview completed following delivery of R1 report

Table 1d: Distribution of Interview Participants – Vietnamese

R	R Age	Type of Interview	IF PROXY- Gender and age of child	R Gender	R Race	Interview Timing
1*	35	Adult		Male	Vietnamese	-
2*	33	Adult		Female	Vietnamese	-
3	17	Adolescent		Female	Vietnamese	70 minutes
4	46	Proxy	12	Female	Vietnamese	75 minutes
5	13	Adolescent		Male	Vietnamese	80 minutes
6	91	Adult		Female	Vietnamese	65 minutes

*Interview conducted using R1 protocol – Interview completed following delivery of R1 report

Table 1e: Distribution of Interview Participants – Chinese

R	R Age	Type of Interview	IF PROXY- Gender and age of child	R Gender	R Race	Interview Timing
1*	67	Adult		Male	Chinese	-
2*	40	Adult		Female	Chinese	-
3*	31	Adult		Female	Chinese	-
4	86	Adult		Male	Chinese	78 minutes
5	41	Adult		Female	Chinese	42 minutes
6	35	Adult		Female	Chinese	49 minutes
7	33	Adult		Female	Chinese	49 minutes

*Interview conducted using R1 protocol – Interview completed following delivery of R1 report

The remaining round 1 and round 2 interviews were conducted between August 4^th and August 29^th, 2019. Interviews were conducted by several in-house methodologists and contracted language specialists. This insured that we had good variance in reporting and timings. Adult English interviews ranged from 55 minutes to 1 hour and 12 minutes. A more detailed discussion of timing is in Section 2 of this report. This ensured that we would have accurate timing data as we administered each module without interruption. No probing was done at this time – only debriefing probes were administered following the interview. At the conclusion of each interview, participants were provided $25 as a token of appreciation for completing the interview.

2. Timing Estimates and Areas to Focus Elimination of Items

The adult timings, with the exception of Tagalog, indicate that the instrument will require few edits to ensure an average administration time of 60 minutes (English). While the timings were somewhat consistent across languages, the Tagalog interviews were very long. After debriefing with the interviewer, it was clear that this was due to several factors.

1. Tagalog is similar to Spanish and generally takes about 30% longer to administer than English.

2. The Tagalog translation includes formal language (“old Tagalog”) that is not clear to many respondents. This caused some confusion, required the interviewer to repeat and simplify questions during the interview. RTI will review the entire Tagalog translation to include simplified and more commonly used terminology to assist the interviewers with administration. This will help to ensure administration times are similar to Spanish.

3. The round two Tagalog respondents stopped and discussed questions with the interviewer, and in a few cases, took breaks. It was difficult to determine the actual timings as the interviewer did not adjust the timings of the interviews as a result of these disruptions. RTI will conduct several internal timings following the simplification of the Tagalog translation.

The total administration time for the adult, adolescent, and proxy interviews are displayed in Tables 2a-2c below. These timings begin at the first question of the main instrument and end with the last item. These timings do not include screener administration, which in 2014 averaged 3 minutes. Tables 2b and 2c are based on a few interviews so these may not be representative what is expected.

Table 2a: Average Timing Estimates – Adult Interviews (n=16) – In Hours and Minutes

LANGUAGE	TOTAL AVG INT TIME
ENGLISH (n=4)	1h 3m
SPANISH (n=4)	1h 14m
TAGALOG (n=3)	2h 0m
VIETNAMESE (n=1)	1h 5m
CHINESE (n=4)	55m

Table 2b: Timing Estimates – Adolescent Interviews (n=5) – In Hours and Minutes

LANGUAGE	TOTAL AVG INT TIME
ENGLISH (n=1)	45m
SPANISH (n=2)	43m
TAGALOG (n=0)	NA
VIETNAMESE (n=2)	1h 15m
CHINESE (n=0)	NA

Table 2c: Timing Estimates – Proxy Child Interviews (n=2) – In Minutes

LANGUAGE	TOTAL AVG INT TIME
ENGLISH (n=0)	NA
SPANISH (n=0)	NA
TAGALOG (n=1)	1h 30m
VIETNAMESE (n=1)	1h 15m
CHINESE (n=0)	NA

Our focus has been on ensuring questionnaire administration in English averages of 60 minutes for an adult, 30 minutes for the child proxy, and 50 minutes for adolescent (which includes 10 minutes of questions administered to the parent or guardian). The English adult and adolescent interview are close to our expected administration time. The proxy timings seem to be outliers. Child-proxy respondents receive a smaller subset of questions then adolescent or adult respondents. These interviews should be shorter than reflected. We will conduct some internal timings on the child proxy interviews to determine an estimated average.

Our early instrumentation activities focused on preparing the instrument for the inclusion of additional questions important to HRSA. This included tightening up several questions and removing several open-ended responses. As a result, several questions were dropped or modified, and some questions were combined. Table 2d lists the number of questions included in each module in 2014 and the current 2019 instrument. The last column displays the net different in the current instrument after deletions and additions. These counts do not include questions that ask for the respondent to specify details if “Other” was selected. There was generally no difference from 2014 to 2019 in the number of “Other” questions and would have little effect on overall respondent burden.

Table 2d: Comparison of question counts, by module, from 2014 instrument to 2019 instrument

MODULE	2014 question count	2019 question count	Net difference between 2014 - 2019
A – Introduction	11	13	+2
B – Access to Care	7	7	0
C – Routine Care	37	26	-11
D - Conditions	80	83	+3
E – Conditions Follow-up	69	59	-10
F – Cancer Screening	55	45	-10
G – Health Center Services	54	56	+2
H – Health Insurance	35	24	-11
I – Prescription Medication	12	27	+15
J - Dental	38	38	0
K – Mental Health	50	55	+5
L – Substance Use	109	114	+5
M – Prenatal Care / Family Planning	30	36	+6
N – HIV Testing	14	14	0
O – Living Arrangements	13	19	+6
P – Neighborhood Characteristics	-	8	+8
Q – Income and Assets	13	15	+2
R – Demographics	33	35	+2
TOTAL	660	674	+14

* Added module in 2019

We initially had concerns that the questionnaire may be too long. However, after reviewing the timings and the question counts, the instrument is close to the 2014 instrument in terms of the overall number of questions and time of administration. However, there is a margin or error that we cannot account for as these timings are based on a small number of cognitive participants, and there are many variables that could affect the timing of administration during the national study, when interviews are conducted at health centers.

Furthermore, we are most concerned with English and Spanish interviews due to the large number of expected respondents. We know that the other languages generally take longer to administer, as does Spanish, but we want to ensure that our estimates reflect what we originally proposed.

Based on this analysis and the necessity to account for unknown variability in the estimates, we suggest a goal of reducing the current instrument by about 6 minutes or approximately 20 survey items. This can include deletions along with modifications to existing questions to streamline administration. RTI suggests the following deletions/edits to reduce administration time:

1. Deletion of Module P (Neighborhood characteristics). We do not have comparable data in 2014 and these 8 items may reduce the administration time by about 2 minutes. Everyone currently receives these items so it will impact administration time for adult, adolescent, and proxy respondents.

2. Ask ADL/IADL questions only of respondents 50 years of age and older (CON27a – CON33). We recognize that some respondents under the age of 50 are likely to have some problems with ADL/IADL activities, however, this is largely an issue affecting older respondents. Since adults over the age of 50 are likely to receive fewer questions than younger adults, this would have an impact on overall timing of adult interviews.

3. There were some areas where question stems and/or responses are repeated. We can format these items so that the stem of the question is repeated twice and then placed in parentheses as optional to the interviewer. Furthermore, job aids may reduce administration time for repeated response sets. This will help to reduce overall administration time. Examples include:

   1. “In the last 12 months…” is currently repeated for HEA13-HEA25 and again from HEA28-HEA36. We could just repeat this twice and add the others in parentheses.

   2. Response scale (Never, Sometimes, Usually, Always) is repeated on HEA15, HEA18, HEA23, HEA25, HEA27, HEA28, HEA29, HEA30, HEA31, HEA32, HEA33, HEA34, HEA36, HEA51, HEA52. We should consider using a showcard to display the responses and instruct the interviewer to use the showcard for these items. This would avoid having the interviewer repeat the response categories each time.

4. Consider possible reduction of questions from the Health Services module. Many of these are CAHPS items and work well together, but this section feels extraordinarily long to administer. There may be some questions that are less useful from an analytic perspective.

5. Consider the analytic value of the series of questions about Autism, ADD/ADHD and developmental delay (MEN2_AUT-MEN2_HAVAUT) – These are asked of all respondents and would save administration time if removed.

3. Question-Specific Findings

During round two, we completed 8 remaining round one interviews using the round one protocol. These were interviews conducted in Chinese, Tagalog, and Vietnamese. The issues we found were either issues documented in the round one report or specific translation changes there are needed. This was specifically true of the Tagalog translation. The interviewer noted challenges as the translation uses formal wording and needs to be simplified for this population. RTI will have language methodologists for each respective language review the item-level comments to adjust the translation. For Tagalog, a more thorough review of the instrument will occur to simplify the translation.

10