GenIC Submitted Under Cognitive Testing and Pilot Testing

GenIC Submitted Under

Cognitive Testing and Pilot Testing for the National Center for Chronic Disease Prevention and Health Promotion

OMB Control Number: 0920-1291 Expiration 5/31/2026

1. Date: June 22, 2023

2. Name, CIO/Program: Marquisette Glass Lewis, NCCDPHP/DPH/PHSB

3. Title of Study: Cognitive testing and RDD push-to-web pilot testing for BRFSS

4. Study Type: Cognitive testing and RDD push-to-web pilot testing

5. Purpose of Study: The purpose of this request is to seek OMB approval to undertake cognitive and pilot testing of survey questions for the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) of the Centers for Disease Control and Prevention (CDC). Approval is being sought as a GenIC under an existing OMB Generic Information Collection clearance (Control Number: 0920-1291 Expiration 5/31/2026).

The questions to be tested may be adopted for future use on the Behavioral Risk Factor Surveillance System (BRFSS). Cognitive and pilot testing are used frequently to test survey questions and survey methodologies under consideration by NCCDPHP divisions and programs. Testing of these questions does not establish definitively whether these questions will be adopted for use on the surveys.

In the table below each question undergoing cognitive testing is provided, including existing versions of questions (if applicable) and a justification for testing. A total of 25 questions from the BRFSS will be included in this set of tests: three from the Demographics core; three from the Long Term COVID-19 Effects core; five from the Cognitive Decline optional module, one from the Social Determinants and Health Equity optional module; three from the COVID-19 Vaccination module; one from the Arthritis/Healthy Aging optional module; and nine from the Caregiver optional module.

Table 1. Questions to be included in cognitive testing
New Question/ introduction Text	Old Question/Introduction Text	Justification for change or addition	Comments
Health Care Access
What is the current primary source of your health care coverage?	What is the current primary source of your health insurance?	Wording revised: To provide a term that allows for a broader range of responses, including those how have health care coverage, but not health insurance.
Demographics
Not including cell phones or numbers used for computers, fax machines or security systems, do you have more than one landline telephone number in your household?
How many of these landline telephone numbers are residential numbers
How many cell phones do you have for personal use?
Long-term COVID Effects
Have you ever tested positive for COVID-19 (using a rapid point-of-care test, self-test, or laboratory test) or been told by a doctor or other health care provider that you have or had COVID-19?	Has a doctor, nurse, or other health professional ever told you that you tested positive for COVID 19?	Wording revised: With the increased use of home tests over the past year, a health care provider might not have been involved in delivering positive test result.
Do you currently have symptoms lasting 3 months or longer that you did not have prior to having coronavirus or COVID-19?	Did you have any symptoms lasting 3 months or longer that you did not have prior to having coronavirus or COVID-19?	Wording revised: The 2022 question assessed period prevalence (from start of pandemic to survey date). Point prevalence will be more useful in 2023 for assessing health care needs because it will more closely reflect ongoing the burden of long-term symptoms as transmission wanes.
Do these long-term symptoms reduce your ability to carry out day-to-day activities compared with the time before you COVID-19?		New question: Assessment of functional impairment is necessary to describe the impact of long-term COVID effects and inform and inform the public health response. In 2023, assessing the impact of symptoms on daily activity is now a higher priority (has more information value), as frequencies of various symptoms following COVID will have been well-studied by then.
Cognitive Decline
During the past 12 months, have you experienced difficulties with thinking or memory that are happening more often or are getting worse?	During the past 12 months, have you experienced confusion or memory loss that is happening more often or is getting worse?	Wording revised: 1) Removed “confusion.” Current research on subjective cognitive decline (SCD) does not suggest confusion is a major component of SCD. “Difficulties with thinking or memory” was a specific suggestion for phrasing by the individuals living with early-stage dementia and reflected how they would have first described their subjective symptoms with cognition.
Are you worried about these difficulties with thinking or memory?		This is a new question. Current research on subjective cognitive decline (SCD) suggests a strong correlation between those who express worry about their difficulties with thinking or memory and future risk of developing dementia. This data will further identify population burden of cognitive impairment.
Have you or anyone else discussed your difficulties with thinking or memory with a health care provider?	Have you or anyone else discussed your confusion or memory loss with a health care professional?	Wording revised: The change to “provider” is to align with other questions on the BRFSS. The proposed change of order — to move the question to third rather than last — is to improve the flow of questions and place similar/cascading questions next to one another.
During the past 12 months, have your difficulties with thinking or memory interfered with day-to-day activities, such as managing medications, paying bills, or keeping track of appointments?	During the past 12 months, as a result of confusion or memory loss, how often have you given up day-to-day household activities or chores you used to do, such as cooking, cleaning, taking medications, driving, or paying bills? Would you say it is…	Wording revised: Based on current research on subjective cognitive decline (SCD), the proposed activities listed align well with difficulties first noted by those experiencing SCD. Clinical researchers on the advisory group noted that the cognitive effort required for “paying bills” was different than the effort required to “clean.” Further, the input from those living with early-stage dementia cited “managing medications” and “paying bills” as two of the activities when they first noticed cognitive issues in themselves. “keeping track of appointments” was added as another example that required similar cognitive load. The decision to change “given up” to “interfered with” was to resolve the ambiguity around what “given up” meant. The advisory group noted that “interfered with” would be easier for respondents to answer.
During the past 12 months, have your difficulties with thinking or memory interfered with your ability to work or volunteer?	During the past 12 months, how often has confusion or memory loss interfered with your ability to work, volunteer, or engage in social activities outside the home? Would you say it is…	Wording revised: Question was simplified to ascertain additional burden among those experiencing subjective cognitive decline (SCD). “engage in social activities” was removed due to mild confusion over what the phrase meant. “outside the home” was removed since respondents may work or volunteer from home.
Social Determinants and Health Equity
How often do you feel lonely? Is it…	How often do you feel socially isolated from others?	After discussions with NCHS about the use of the term “socially isolated” to address loneliness the question has been change to “lonely”
COVID Vaccination
How many COVID-19 vaccinations have you received? 1 One 2 Two 3 Three 4 Four 5 Five 6 Six or more 7 Don’t know / Not sure 9 Refused	How many COVID-19 vaccinations have you received? 1 One 2 Two 3 Three 4 Four or more 7 Don’t know / Not sure 9 Refused	With the latest booster dose recommendation, some respondents could have received as many as 6 total doses by 2024
During what month and year did you receive your most recent COVID-19 vaccine?		Since there are likely to be updated recommendations for COVID booster vaccination by 2024, knowing how many doses of COVID vaccine received (already included in the COVID Vaccination module), the brand of first dose, and the date of most recent dose will allow for determination of receipt of the primary series and most recently recommended booster dose, whatever the recommendation might be
Which brand of COVID-19 vaccine did you receive for your first dose? 1 Pfizer-Biontech/Comirnaty 2 Moderna/Spikevax 3 Johnson&Johnson/Janssen 4 Novavax 5 One of the other brands that require 2 shots but unsure of name 6 Other 77 Don’t know 99 Refused		Since there are likely to be updated recommendations for COVID booster vaccination by 2024, knowing how many doses of COVID vaccine received, the brand of first dose, and the date of most recent dose will allow for determination of receipt of the primary series and most recently recommended booster dose, whatever the recommendation might be. Respondents reporting at least 3 doses of vaccine will have completed a primary series and received at least 1 additional dose, regardless of the brand of first dose, and based on the date of most recent vaccination we can determine if the most recent dose was the most recently recommended booster. For those reporting receipt of only 1 or 2 doses, we need to know if the first dose was Johnson & Johnson (which is a 1-dose primary series) or one of the vaccines requiring 2 doses for the primary series to know if the respondent has completed a primary series and if the most recent dose would be considered a booster or the second dose in the primary series.
Arthritis/Healthy Aging
SKIP INFO: The question below is only asked it the respondent answered yes to the question: (Ever told) (you had) some form of arthritis, rheumatoid arthritis, gout, lupus, or fibromyalgia? Has a doctor or other health professional ever suggested physical activity or exercise to help your arthritis or joint symptoms? 1 Yes 2 No 7 Don’t know / Not sure 9 Refused		Capturing this data will enable the Arthritis Program to have baseline, mid-point and end-point data for the awardees to be funded under the DP-23-0001 NOFO, “State Public Health Approaches to Addressing Arthritis.”
Caregiver
During the past 30 days, did you provide regular care or assistance to a friend or family member who has a health problem or disability? Interviewer note: If caregiving recipient has died in the past 30 days, code 8 and say: I’m so sorry to hear of your loss. 1 Yes 2 No 7 Don’t know / Not sure 8 Caregiving recipient died in past 30 days 9 Refused		No changes to question, this is part of a complete module revision. The Interviewer Instructions were slightly modified to sound more empathetic.
What is their relationship to you?	What is his or her relationship to you? For example is he or she your (mother or daughter or father or son)?	Several changes are proposed to this question: 1) Simplified wording in question. 2) Wording change in both the question and response options to be gender neutral. 3) Removal of Interviewer Instructions to enhance clarity. 4) Consolidation of response options. Based on discussion of the Alzheimer’s Disease and Health Aging work group (also known as the Aging and Health work group, the proposed, consolidated response options reflect: 1) Frequent aggregation of these relationships in existing Caregiver Module data analysis (e.g., “Mother,” “Father,” “Father-in-law” and “Mother-in-law” are frequently collapsed into a single measure in existing analysis). 2) Utility to state programmatic activities (e.g., services supporting caregivers of children are distinct from services supporting caregivers of adults and thus are distinct) response options).
What is the main health problem or disability that the person you care for has?	What is the main health problem, long‐term illness, or disability that the person you care for has?	This question was moved to improve the flow of questions by placing similar/cascading questions next to each other. Q2-Q4 all ask questions about the care recipient. “Long-term illness” was deleted to mirror wording from the first question. Two overarching changes are proposed: a) changes to response options and b) reordering the response options. For a) changes to response options we deleted response options due to consistently low response rates (>.5% annually). For: 1) “Human Immunodeficiency Virus Infection (HIV)” 2) “Substance Abuse or Addiction Disorders” New response options – both added to reflect conditions identified by the work group’s caregiving experts as high-burden conditions not otherwise assessed: 1) “Hearing or vision loss” 2)“Movement disorders such as Parkinson’s, spinal cord injury, or multiple sclerosis” Altered response options: Due to consistently low response rate, “Asthma” was consolidated with “Chronic respiratory conditions such as Emphysema or COPD.” Further, the work group determined this combination remains applicable to state programmatic activities focused on these conditions. The work group’s caregiving and state programmatic experts identified additional conditions to be included alongside existing options. These new conditions provide a more accurate representation of response options. 2)“Stroke” was added to “Heart disease, hypertension, or stroke” 3)“Traumatic brain injury” was added to “Injuries including broken bones or traumatic brain injury” For b) reordering the response options, first, the response options carried forward into the proposed revision were reordered so the more frequent responses appear first, based on existing response rates. This is to shorten the average time to administer since respondents are more likely to encounter their response earlier in the list. Secondly, since the two new proposed response options listed above do not have response rate data, they appear near the end of the list. Finally, the work group agreed to keep “Old age, infirmity, or frailty” and “Other” as the final two response options. This aligns with the current module.
Does the person you care for also have Alzheimer’s disease, dementia or other cognitive impairment?		No change to wording. This question is fourth in the proposed revision to ensure it follows Q3. Q2-Q4 all ask questions about the care recipient.
In the past 30 days, did you provide regular care for this person by helping with nursing or medical tasks such as injections, wound care, or tube feedings?		This is a new question. The work group’s caregiving experts noted a growing trend of caregivers providing medical or nursing care to their care recipients. Given the difficulty and unique differences of these vs. other tasks, the work group recommends adding this question. This question appears first among the three “tasks” questions. They descend in order of intensity. The medical/nursing tasks are most intense while household tasks are the least intense
In the past 30 days, did you provide regular care for this person by managing personal care such as bathing, getting to the bathroom, or helping to eat?	In the past 30 days, did you provide care for this person by… Managing personal care such as giving medications, feeding, dressing, or bathing?	The examples cited in the proposed revision align with other caregiving survey methodology and activities of daily living (ADLs), and represent the three common personal care tasks. “Feeding” was changed to “helping to eat” to better account for the full spectrum of what this care task entails. The set of questions Q5-Q8 all ask about the burden caregivers face. In case of drop-off, Q5-Q9 are ordered by importance to understand caregiving burden.
In the past 30 days, did you provide regular care for this person by managing household tasks such as help with transportation, shopping, or managing money?	In the past 30 days, did you provide care for this person by… Managing household tasks such as cleaning, managing money, or preparing meals?	The examples cited in the proposed revision align with other caregiving survey methodology and instrumental activities of daily living (IADLs), and represent the three common household care tasks. Further, input from those living with early-stage dementia cited “help with transportation” as a household task they frequently needed assistance with. The set of questions Q5-Q8 all ask about the burden caregivers face. In case of drop-off, Q5-Q9 are ordered by importance to understand caregiving burden.
In an average week, how many hours do you provide regular care or assistance? Would you say…	In an average week, how many hours do you provide care or assistance? Would you say…	The word “regular” was added to the question to mirror wording used in the first question. Based on discussion of the work group, the proposed, consolidated response options reflect: 1) Frequent aggregation of these relationships in existing Caregiver Module data analysis (e.g. “Up to 8 hours per week” and “9 to 19 hours per week” are frequently collapsed into a single measure in current research). 2) Utility to state programmatic activities (e.g. few state programs and services distinguish between 0-8 hours of care vs 9-19 hours of care per week). 3) Wording aligns with similar response options on other BRFSS questions. The set of questions Q5-Q9 all ask about the burden caregivers face. In case of drop-off, Q5-Q9 are ordered by importance to understand caregiving burden and based on the natural flow.
For how long have you provided regular care to this person?	For how long have you provided care for that person? Would you say…	The word “regular” was added to the question to mirror wording used in the first question. Based on discussion of the work group, the proposed, consolidated response options reflect: 1) Frequent aggregation of these relationships in existing Caregiver Module data analysis (e.g., “1 month to less than 6 months” and “6 months to less than 2 years” are frequently collapsed into a single measure in current research). 2) Wording aligns with similar response options on other BRFSS questions. The set of questions Q5-Q9 all ask about the burden caregivers face. In case of drop-off, Q5-Q9 are ordered by importance to understand caregiving burden and based on the natural flow.

The pilot will test the use of text messaging to invite people to a web survey. Pilot tests will be used to inform future survey construction.

No data from cognitive and pilot testing will be used for population prevalence estimation or rigorous analysis of health data. Data will be used to determine effectiveness of approaches of data collection methodologies, comparisons of software and other technologies, research on optimal questionnaire formats (question formats, wording, response sets), subject recruitment and refusal conversion, and new sample methods.

Overall results of the cognitive testing processes are linked to improved data quality and efficiency in the data collection process. Without pretesting of questions and processes provided in cognitive testing, respondent burden would be increased, and data quality would suffer.

6. Respondent Characteristics:

Cognitive testing

Respondents that are U.S. residents, 18 years of age and older and living in a private residence, will be recruited to participate. A total of 25 respondents will be recruited for each set of questions with the following criteria

• Twelve are ages 18-44 and 13 ages 45+

• At least 10 male and 10 female

• At least 13 have a landline telephone

• 13 have received at least 1 COVID-19 vaccination and 12 have not received at least 1 COVID-19 vaccination

• At least 13 who have tested positive for COVID-19

• At least 5 Hispanic participants

• Mix of races with 9 White, 9 Black or African American, 5 Asian/Pacific Islander

• A mix of education levels with no more than 9 with a post graduate degree

• Must be able to speak in English

Push-to-web pilot testing

A total of 2,400 respondents states will be recruited for RDD push-to-web. Respondents must be U.S. residents, 18 years of age and older and living in a private residence, within one of the four states included the pilot study. States included in the project will be selected by the diversity of population characteristics have diverse characteristics in terms of urban/rural regions, population/ race/ ethnicity heterogeneity, and population size (large and small states) drawn for pilot testing.

7. Study Methods:

Cognitive Testing:

The cognitive testing data will be collected either by telephone or other virtual methods, ZOOM, SKYPE, etc. Refer to Attachment 1 for cognitive testing schedule. Data will be compiled into a detailed report for all participating programs.

Pilot Testing:

The data will be collected via web-based survey and CATI. Data will then be compared to ongoing surveillance using identical questions. Refer to Attachment 2 for pilot testing schedule

8. Recruitment and Incentives: Westat will be the contractor working with the NCCDPHP on the cognitive and pilot tests.

Cognitive Testing:

Westat will recruit from online sources and from general advertisements available to the public for some cognitive testing. During the set of cognitive tests, participants may also be draw from RDD samples and recruited by phone (see Attachment 3). Participants will be provided at $50 incentive. This incentive amount is considered appropriate because of the following participation needs of our important populations:

• Online participants must have a functioning device (e.g., computer, tablet, or phone) with broadband Internet, which may incur costs from renting equipment and/or data usage on their Internet plans for the duration of the interviews.

• Online participants are required to join the interview from a quiet location where there are no distractions, which may require hourly childcare/daycare costs or special accommodations during that time.

• Time and effort spent toward the online and phone screening, check-in and check-out procedures, and setting up the ZoomGov online platform on the device chosen.

Cognitive testing will be completed virtually for this project. The incentive is an effective method of drawing attention to the study and gaining cooperation for completing it. It is not intended as a payment for their time but rather a means for increasing response rates. The practical consequences of volunteers perceiving an insufficient incentive may result in:

• Increased time and cost of recruitment due to lower response and enrollment levels, and/or the need to schedule additional interviews to achieve the targeted number of participants

• Increased likelihood of “no-shows” with some replacements needed beyond those over-recruited

Pilot Testing:

Modified protocols of the BRFSS RDD sample for cell phones will be used to identify and recruit potential participants for the RDD push-to-web Participants will be provided at $10 incentive will be given upon completion of the web survey. The incentive will be in the form of a $10 Amazon.com gift card.

9. Personally Identifiable Information (PII): General demographic characteristics of respondents will be collected and associated with paradata on questions posed to participants and question responses. The cognitive test sample files may include phone numbers or addresses. Persons who are recruited for cognitive testing may report their addresses during the recruitment process. CDC will not retain any individually identifiable information and will not maintain sample files of phone numbers or addresses. Respondents’ phone numbers and other identifiable information will be kept in files separate from response files and will not be connected to responses. If recordings are made of cognitive interviews, no personally identifying information will be connected to the recordings and they will be retained only for the time needed for research purposes. They will be stored in secure locations. After completion of the cognitive testing, all files will

be destroyed. Westat and CDC will be the only entities with access to the dataset(s).

A summary report of the RDD Push-to-web collection of BRFSS Supplemental Data Using Web-Based Methods will be provided to state health departments in the states included in the test. Information in summary form may be used for presentations on methodology, but combined datasets from participating states will not be provided. Results may also be used to prepare and present methodological research papers at professional conferences or for peer reviewed journals. No data from the pilot test will be used to produce prevalence estimates or analyze public health status.

10. Informed Consent/Voluntary Participation: Individuals participating in the cognitive tests will be voluntarily recruited but will be reminded that they may refuse to answer specific questions. In all cognitive tests where a telephone interview is conducted, protocols for voluntary screening will be used that match procedures for the surveys that are being tested. This screener informs respondents that they do not have to participate and that they may refuse to answer any question. Individuals participating in the cognitive testing by phone are also informed of the voluntary nature of their participation in an introduction (Attachment 4).

Verbal consent is obtained from participants during the initial contact and/or screening process (see Attachment 4 for cognitive testing and Attachment 5 for pilot testing example) or by application to the vendor for participation by responding to an advertisement. In all cognitive tests conducted by phone/virtual meeting software an introductory script, including the voluntary nature of the survey, precedes the survey questions.

During the initial screening for RDD push-to-web, an interviewer will obtain informed consent. The potential participants will be informed that their telephone number was randomly selected and that participation in the study is completely voluntary. The interviewer will explain the nature of the study and approximately how long the survey will take. The potential participant will be told that they do not have to answer any question that they do not want to, and they can stop the survey at any time. They will be informed that their response on the survey will not be connected to any personal information, and it they have any questions they can call the survey point of contact.

No PII will be collected. All telephone numbers in the RDD push-to-web method will be retained by a contractor for the use of the pilot and then files will be destroyed. The PII section details how information will be secured.

11. Analysis of Data: This cognitive testing for BRFSS will be conducted by Westat. Westat will be responsible for production of reports outlining their findings including question specific suggestions for improvement, problems with interpretation of question wording, suggestions for ordering of questions. An example of a cognitive testing report is provided in Attachment 5. Cognitive testing results will be shared with the CDC programs which requested cognitive testing of the propose questions and /or with state or local partners who participate in data collection systems. No data from the cognitive tests will be used to produce prevalence estimates or analyze public health status.

Analyses from the RDD push-to-web pilot will include calculating cooperation rates, completion rates, overall response rates, and the distribution of demographic characteristics for persons completing the web-based survey for the RDD push-to-web. Data for each contact mode will be evaluated on differences in responses (including item refusal, demographic comparisons and health outcomes) by types of questions. We will also compare state and mode differences in cost effectiveness, response rates, and data quality (such as item nonresponse) among respondents who complete the web-based survey via traditional RDD and RDD push-to-web.

12. Collection Timeline:

Cognitive testing:

Cognitive testing recruitment will take approximately 3 weeks. The interview process will take 3 weeks, and it will take place simultaneously with recruitment. Transcribing of responses, data set creation and report writing will take 4 weeks

Pilot testing:

Pilot testing will take approximately 7 weeks followed by data set creation and report writing, taking 10 weeks.

13. Burden Table: The table below illustrates burden for each set of questions including in this request.

Estimates of Annualized Hour and Cost Burden

Table 2A
Type of Respondents	Stage of Survey Administration	Number of respondents	Number of responses per respondent	Average burden per response (in hours)	Total burden in hours
General US Adult Population	Cognitive testing	25	1	1.0	25
	Screening for respondents RDD push-to-web	6,000	1	0.1	600
	Respondents via RDD push-to-web	2,400	1	0.28	672
Total					1,297

Table 2B Estimated Respondent Burden for Cognitive Testing
Type of Information Collection	Total Burden Hours	Average Hourly Wage Rate*	Total Cost Burden
Cognitive testing	25	$29.76	$744
Surveying respondents by RDD push-to-web	1.272	$29.76	$37,850

*Based upon the average hourly earnings for all occupations from the Bureau of Labor Statistics May 2022 National Occupational Employment and Wage Estimates (available at https://www.bls.gov/oes/current/oes_nat.htm).