Download:
pdf |
pdfMEDICARE CURRENT BENEFICIARY SURVEY
(MCBS)
Task 9.30 PaRIS Cognitive Testing Report
HHSM-500-2014-00035I, Task Order # HHSM-500-T0002
APRIL 29, 2021
PRESENTED TO:
William Long, Contracting Officer’s
Representative
Centers for Medicare and Medicaid Services
7111 Security Boulevard
Baltimore, MD 21244
PRESENTED BY:
Susan Schechter, Project Director
NORC at the University of Chicago
55 East Monroe Street
30th Floor
Chicago, IL 60603
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Table of Contents
Executive Summary .......................................................................................................................4
Introduction ....................................................................................................................................5
Methods ...........................................................................................................................................6
Sampling and Recruitment ......................................................................................................6
Data Collection ........................................................................................................................7
Data Analysis Methods............................................................................................................8
Findings...........................................................................................................................................8
General Findings on Administration of the PaRIS Survey .....................................................8
Terminology: “On the Level” (Q8) .......................................................................................10
Results ..........................................................................................................................10
Terminology: “Health Professional(s)” (Q36, Q37, Q39, Q41, Q42) ...................................11
Results ..........................................................................................................................12
Terminology: “Single Professional” (Q51, Q52) ..................................................................12
Results ..........................................................................................................................13
Terminology: “Care Plan” (Q60, Q61, Q62, Q63) ................................................................13
Results ..........................................................................................................................14
Terminology: “Care Records” (Q69) ....................................................................................15
Results ..........................................................................................................................15
Terminology: “Primary Care Centre” (Q59, Q69, Q70, Q71, Q72, Q74, Q75, Q76, Q82,
Q93) 16
Results ..........................................................................................................................17
Demographics Probes (Q99, Q100, Q107, Q108, Q109) ......................................................17
Results on Gender (Q99) .............................................................................................18
Results on Sexual Orientation (Q100) .........................................................................18
Results on Location of Birth (Q107) ............................................................................19
Results on Citizenship (Q108) .....................................................................................19
Results on “Hamlet” (Q109) .......................................................................................19
Q50. Health Problems ...........................................................................................................19
APRIL 29, 2021 | I
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Discussion and Recommendations .............................................................................................21
General Findings on Administration of the PaRIS Survey ...................................................21
Discussion of Probed Terminology .......................................................................................21
Considerations for Retaining MCBS Demographic Items ....................................................23
Notes on Other Items Identified during Testing ....................................................................26
Conclusion .............................................................................................................................27
Appendix A. PaRIS Cognitive Interview Screener ...................................................................28
Appendix B. PaRIS Cognitive Interview Questionnaire ..........................................................31
Appendix C. PaRIS Cognitive Interview Probes ......................................................................50
APRIL 29, 2021 | II
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
List of Exhibits
Exhibit 1.
Distribution of Sex, Ethnicity, Race, and Educational Attainment among Cognitive
Interview Respondents ..................................................................................................... 7
Exhibit 2.
Questionnaire Specification Errors .................................................................................. 9
Exhibit 3.
Q8 Question Text and Response Options ...................................................................... 10
Exhibit 4.
“On the Level” Cognition .............................................................................................. 10
Exhibit 5.
Questions Containing “Health professionals”................................................................ 11
Exhibit 6.
“Health Professionals” Cognition .................................................................................. 12
Exhibit 7.
“Single Professional” Question Text ............................................................................. 12
Exhibit 8.
“Single Professional” Cognition .................................................................................... 13
Exhibit 9.
“Single Professional” Provider Type Frequency ........................................................... 13
Exhibit 10. “Care Plan” Question Text ............................................................................................. 14
Exhibit 11. “Care Plan” Cognition ................................................................................................... 14
Exhibit 12. Cross Tabulation of Response to Q60 and Cognition of “Care Plan” ........................... 15
Exhibit 13. “Care Records” Question Text ....................................................................................... 15
Exhibit 14. “Care Records” Cognition.............................................................................................. 16
Exhibit 15. “Primary Care Centre” Question Text ........................................................................... 16
Exhibit 16. “Primary Care Centre” Cognition .................................................................................. 17
Exhibit 17. Demographics Items for Probing ................................................................................... 18
Exhibit 18. Understanding of “Hamlet” (Q109) ............................................................................... 19
Exhibit 19. Q50 Question Text and Response Options .................................................................... 20
Exhibit 20. Demographic Items from PaRIS Compared to MCBS .................................................. 23
Exhibit 21. Question Comprehension Issues .................................................................................... 26
Exhibit 22. PaRIS Cognitive Interview Probes................................................................................. 50
APRIL 29, 2021 | III
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Executive Summary
The Medicare Current Beneficiary Survey (MCBS) is a longitudinal panel, multi-purpose survey of a
nationally representative sample of the Medicare population, conducted by the Centers for Medicare
& Medicaid Services (CMS) through a contract with NORC at the University of Chicago.
In 2017, the Organization for Economic Cooperation and Development (OECD) launched the
Patient-Reported Indicator Surveys (PaRIS) initiative, a collaborative, innovative effort where
countries work together to develop, standardize, and implement indicators that measure outcomes
and experiences of health care that matter most to people 1. The Department of Health and Human
Services (DHHS) is considering the Medicare Current Beneficiary Survey (MCBS) to potentially be
the primary means of implementing the PaRIS Patient Questionnaire (subsequently referred to as the
“PaRIS survey”) in the U.S., in the coming years.
NORC conducted nine cognitive interviews on the draft PaRIS instrument to test the ability of
Medicare respondents to understand the PaRIS questionnaire in the context of the American health
care system. The survey was developed from existing, validated sources. However, some of these
sources, such as the European Health Interview Survey and Commonwealth Fund International
Health Policy Survey of Older Adults, may include terminology unfamiliar to U.S. respondents.
This report summarizes recommendations on revisions to some PaRIS items when they are
administered in the U.S.; in particular, these recommendations would be used to modify PaRIS items
if it is administered to MCBS respondents. It also provides recommendations on general revisions to
some PaRIS items that should be considered by the OECD Working Group as part of the Field Trial
Phase.
1
For more information, please visit: https://www.oecd.org/health/paris/
APRIL 29, 2021 | 4
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Introduction
In 2017, the Organization for Economic Cooperation and Development (OECD) launched the
Patient-Reported Indicator Surveys (PaRIS) initiative, a collaborative, innovative effort where
countries work together to develop, standardize, and implement indicators that measure outcomes
and experiences of health care that matter most to people. One of the ways to accomplish this goal is
to close critical data gaps that will inform health systems. The PaRIS International Survey of People
Living with Chronic Conditions focuses on closing gaps in two areas: Patient Reported Experience
Measures (PREMS) which measure how patients experience health care and Patient Reported
Outcome Measures (PROMS) which measure how patients assess the results of the care they receive.
The PaRIS Patient Questionnaire, administered to patients, includes both PREMS and PROMS items
and aims to collect vital information about primary health care, by asking about topics such as the
respondent’s health, health behaviors, patient activation and confidence in managing their health
care, experiences with health care and health providers including access to health care, quality of life,
physical functioning, and psychological well-being 2.
The PaRIS Patient Questionnaire will be conducted internationally with respondents ages 45 and
older who receive care in primary care or ambulatory settings. The Department of Health and Human
Services (DHHS) is considering the Medicare Current Beneficiary Survey (MCBS) to potentially be
the primary means of implementing the PaRIS Patient Questionnaire in the U.S. in the coming years.
The MCBS is a continuous, multi-purpose survey of a representative national sample of the Medicare
population, including the population of beneficiaries aged 65 and over and beneficiaries aged 64 and
below with certain disabling conditions, residing in the U.S. The MCBS is conducted by the Centers
for Medicare & Medicaid Services (CMS) through a contract with NORC at the University of
Chicago. Given the age and health characteristics of Medicare beneficiaries and the survey’s existing
operational infrastructure, the MCBS is a natural choice for a one-time implementation of the PaRIS
Patient Questionnaire. This report will subsequently refer to the PaRIS Patient Questionnaire as the
“PaRIS survey”.
The PaRIS survey was drafted by an international consortium of methodological and health care
experts who followed a rigorous approach including developing a conceptual framework, conducting
an extensive literature review, identifying existing survey items, and mapping existing items to the
PaRIS conceptual framework. Examples of existing validated items in the draft PaRIS survey include
the Porter-Novelli Scale for measuring patient activation and the World Health Organization Well
Being Index (WHO-5) to measure mental health.
2
For more information, please visit: https://www.oecd.org/health/paris/
APRIL 29, 2021 | 5
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
■
The PaRIS survey is being developed and implemented over the course of three phases:
■
Development Phase (Phase 1, Feb 2020-Jan 2021): A consortium of survey experts from
participating countries developed the draft PaRIS survey via a comprehensive review of existing,
validated sources.
■
Field Trial Phase (Phase 2, anticipated February 2021-July 2022): Cognitive testing of the draft
PaRIS survey; Pilot test of draft survey in European Union countries.
■
Main Survey Phase (Phase 3, anticipated August 2022-December 2023): Full implementation of
finalized PaRIS survey in all participating countries, including collection and dissemination of
data.
As part of Phase 2, NORC conducted nine cognitive interviews of the draft PaRIS survey instrument
to test the ability of Medicare respondents to understand the PaRIS questionnaire in the context of the
American health care system. While the survey was developed from existing, validated sources,
some of these sources, such as the European Health Interview Survey and Commonwealth Fund
International Health Policy Survey of Older Adults, may include terminology unfamiliar to U.S
respondents. Accordingly, the goal of this testing effort was to assess the understanding of some of
the healthcare related concepts among Medicare beneficiaries in the U.S. This report summarizes
recommendations on revisions to some PaRIS items when they are administered in the U.S.; in
particular, these recommendations would be used to modify PaRIS items if it is administered to
MCBS respondents. It also provides recommendations on general revisions to some PaRIS items that
should be considered by the OECD Working Group as part of the Field Trial Phase.
Methods
The methodological design of the PaRIS cognitive testing was consistent with the design of typical
cognitive testing research. This section discusses the methodologies used to conduct the study.
Sampling and Recruitment
NORC recruited potential respondents through personal networking and community outreach. NORC
staff made outreach to their networks to identify second- or third-degree contacts who are Medicare
eligible and had at least one chronic condition. NORC also engaged with community organizations in
the Chicago area serving people who generally qualify as Medicare beneficiaries. These
organizations assisted with publicizing the study by disseminating flyers and word-of-mouth. Flyers
were also posted on social media groups geared towards people over 65. Community organizations
and social media groups were selected specifically with an aim to recruit a diverse group of
respondents, with priority placed on respondents who were non-white, Hispanic, or had an education
level lower than a college degree. Community organizations were recruited via email and phone
outreach to organizational leadership such as directors or other contacts identified on organizational
websites.
APRIL 29, 2021 | 6
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
All interviews were conducted in English. Due to the COVID-19 pandemic, all interviews were
conducted remotely. Potential respondents were given the option to do the interview either via phone
or Zoom video conference.
All respondents were provided a $40 incentive for participating. Respondents ranged in age from 66
to 79. Exhibit 1 shows distributions by sex, ethnicity, race, and educational attainment as reported
during screening.
Exhibit 1. Distribution of Sex, Ethnicity, Race, and Educational Attainment among Cognitive
Interview Respondents
Characteristic
Count
Sex
Female
5
Male
4
Ethnicity
Non-Hispanic
9
Hispanic
0
Race
Black or African American
2
Asian
1
White
6
Education
High school graduate
2
Some college, but no degree
2
Associate degree
1
Bachelor's degree
1
Master's, professional or doctorate degree
3
Data Collection
Potential respondents were screened over the phone to determine their eligibility for participation.
The screener is provided in Appendix A. The screener consisted of items to measure eligibility, such
as age, Medicare enrollment status, chronic condition diagnosis, and basic demographics. To be
eligible to participate, respondents had to self-report eligibility for Medicare and have at least one
chronic condition. Once respondents were determined to be eligible, interviewers scheduled the
remote interview. Respondents were asked during screening if they preferred a Zoom or phone
interview.
APRIL 29, 2021 | 7
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Before starting the interview, respondents were read a series of informed consent questions and were
asked for verbal consent. Interviews were recorded with the respondent’s permission so that analysts
could reference the recording during analysis as needed. Interviewers administered the questionnaire
remotely as a paper-and-pencil interview (PAPI). The questionnaire instrument is provided in
Appendix B.
The questionnaire was administered in sections and interviewers debriefed with respondents on each
section before moving on to the next section. The debriefings consisted of reviewing terminology
used by certain pre-selected questionnaire items and probing for information about how the
respondent interpreted these terms and arrived at their answers. This cognitive testing effort primarily
used retrospective probing techniques, meaning respondents were administered the full questionnaire
section and then retrospectively asked questions about the items being tested. The questions were
pre-selected for probing based on a comprehensive review of the questionnaire; the corresponding
probes are shown in Appendix C. Respondents were also asked for general feedback on the clarity of
items in each questionnaire section. After each interview, interviewers wrote notes summarizing the
themes and responses.
Data Analysis Methods
Analysts reviewed the interview notes and listened to recordings for additional clarity. Analysts
identified themes in the responses and created frequency counts of themes identified. This approach
provides insight into the confusion respondents may have experienced in responding to specific
survey items. It also serves to identify items that respondents did not answer as intended and provides
insight into whether the concepts were understood by U.S. respondents.
Findings
The discussion below outlines general findings on the administration of the PaRIS survey, findings
based on the specific terminology used in items that were probed, and findings on other
comprehension issues that arose during testing.
General Findings on Administration of the PaRIS Survey
In general, the PaRIS survey flowed well and made sense to respondents. The average cognitive
interview duration with probing was 1 hour and 20 minutes. Interviewers had some difficulties
administering the PAPI over the phone or via video conference, with some questions harder to
administer by an interviewer than would have been as a self-administered survey.
In the course of cognitive testing, interviewers identified a number of issues in the PaRIS
specifications relating to skip logic and response option errors. Some of these errors were identified
APRIL 29, 2021 | 8
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
prior to cognitive interviewing during the pre-interview questionnaire review and are listed here to
provide a comprehensive list of all recommended changes. These errors are summarized in Exhibit 2.
Exhibit 2.
Question
Number
9
Questionnaire Specification Errors
Error Type
Mismatched
Point of View
Question Text
9. In the past 7 days, I had a
problem with my sleep.
Description of Error
Q9 is written in the first-person,
whereas the subsequent items (1013) are written in the second-person.
10. In the past 7 days, how would
you rate your fatigue on average?
73
Question Text 73. How often did you have a
health problem but did not seek
care, or did take a prescription
medicine because of the cost?
The word “not” is missing between
“did” and “take.”
75
Routing Error
If “None of these” is selected at
question 75, respondent should skip
question 76.
78
Question Text 78. Have you been in a hospital
emergency department for your
own medical care?
Item has an unspecified reference
period. Respondents were unsure if
this question referred to “ever” or a
specific period of time.
79
Question Text 79. Have you been in the hospital
for one night or longer?
Item has an unspecified reference
period. Respondents were unsure if
this question referred to “ever” or a
specific period of time.
80
Response
Options
80. What was the purpose of this
visit?
Response option “First contact for a
new problem” is listed in the code
list twice (the first two response
options).
101
Response
Options
101. Which of these best describes
your current work situation?
The eighth response option in the
code list says “Answer” which
appears to be a placeholder.
107
Response
Options
107. Were you born in the United
States of America?
- Yes
- No (if you answered ‘No”, please
state what country you are a citizen
of: __________________________
If you answer no, the text says to
enter the country you are a “citizen
of” but it should read “what country
you were born”. Citizenship and
place of birth are not the same and
Q108 asks about citizenship.
75. As far as you know, which of
the following online services does
your primary care centre offer?
APRIL 29, 2021 | 9
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Terminology: “On the Level” (Q8)
This question was selected for probing based on concerns that the term “on the level” is likely not
understood by respondents from the U.S.
Exhibit 3.
Q8 Question Text and Response Options
Question
Number
Question Text
Response Options*
- I only get breathless with strenuous exercise
- I get short of breath when hurrying on the
level or up a slight hill
- I walk slower than people of the same age
on the level because of breathlessness or have
to stop for breath when walking at my own
pace on the level
- I stop for breath after walking 100 yards or
after a few minutes on the level
- I am too breathless to leave the house
* Underlining added to emphasize the terms being tested; underlining was not part of the actual
question.
8
Please select the response that best
describes your current situation in
relation to your breathing:
Results
During probing, it was found that five respondents understood the term while four either did not
understand or had questions about it, as shown in Exhibit 4.
Exhibit 4.
“On the Level” Cognition
Cognition
Count
Understood
5
Knew two meanings and asked for clarification
1
Did not understand, asked what it meant
2
Interpreted Incorrectly
1
One respondent interpreted the term incorrectly, saying they thought it meant "...as something
compared to people my age". Another asked for clarification, noting that there are two definitions of
“on the level”, either (1) flat pavement or (2) honest and straight to the point. Cognitive interviewers
noted that even respondents who were able to properly describe the term as “on a flat surface”, most
gleaned the definition from contextual clues and were not actually familiar with the term.
APRIL 29, 2021 | 10
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Terminology: “Health Professional(s)” (Q36, Q37, Q39, Q41, Q42)
Eleven items use the term “health professionals” or “health professional.” Probing was conducted
specifically on Q36 to test this term. The term “health professionals” or “health professional” was
selected for probing to test whether respondents would include the full scope of health professionals
in their understanding of the item as included in the definition provided in the introduction to the
section “Managing your health and healthcare”, occurring between Q18 and Q19.
Exhibit 5.
Questions Containing “Health professionals”
Question
Number
Question Text*
Introduction
to “Managing
your health
and health
care” section,
between Q18
and Q19
The next few questions ask about your recent physical activity, diet and use of
tobacco and alcohol and the support you may have received from health
professionals (doctors, nurses, and other health professionals).
20
In the past 12 months, has any health professional talked with you about your
physical activity?
23
In the past 12 months, has any health professional talked with you about healthy
eating?
26
In the past 12 months, has any health professional talked with you about the health
risks of smoking or using tobacco and ways to quit?
28
In the past 12 months, has any health professional talked with you about alcohol
use?
29
I rely on health professionals to tell me everything I need to know to manage my
health
36
My health professionals and I work together to manage my health
37
When I read or hear something that is relevant to my health care, I bring it up with
my health professionals
39
When I think something is wrong with my health care, I raise my concerns with
my health professionals
41
How confident are you that you can follow instructions from health professionals
about how you should care for yourself at home?
42
How confident are you that you can follow instructions from health professionals
about how to change your habits or lifestyle?
* Underlining added to emphasize the terms being tested; underlining was not part of the actual
question.
APRIL 29, 2021 | 11
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Results
During probing, it was found that three respondents understood the term “health professional” to
mean doctors (MDs), three understood it as doctors and physical therapists, and three understood as
strictly their primary care doctor.
Exhibit 6.
“Health Professionals” Cognition
Cognition
Count
Doctors (MDs)
3
Doctors & Physical Therapists
3
Primary Care Doctor
3
Three respondents interpreted the term in relation to the doctor they see, with one stating, “My
primary doctor at the VA; that's the only health provider I see.” and another stating, “My primary is
generally the only person that I see right now.” Cognitive interviews showed that only three
respondents interpreted “health professionals” more broadly to include physical therapists and none
included other health professionals like nurses or nurse practitioners included in the definition
between Q18 and Q19.
Terminology: “Single Professional” (Q51, Q52)
Two items in the PaRIS survey use the term “single professional” (Q51 and Q52). Probing was
conducted specifically on Q51, though Q52 shares similar terms. Probing was conducted to test
whether the term “single professional” would have meaning to a respondent from the U.S. and
whether a respondent from the U.S. understood the concept of care coordination.
Exhibit 7.
“Single Professional” Question Text
Question
Number
51
Question Text*
Do you have a single professional who takes responsibility for coordinating
your care across the services that you use?
52
Is this the same professional that you see for most of your health problems?
* Underlining added to emphasize the terms being tested; underlining was not part of the actual
question.
APRIL 29, 2021 | 12
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Results
In general, most respondents understood the terminology used in Q51, as shown in Exhibit 8. The
respondent who did not understand said “Not sure about the question; everything has to go through
our PCP….I guess I don’t really understand the question.”
Exhibit 8.
“Single Professional” Cognition
Cognition
Count*
Understood
7
Didn't Understand
*One respondent was not probed on this item.
1
While most respondents generally understood the question, some had a hard time answering because
the concept of care coordination is not as established in the U.S. as compared to other countries.
Exhibit 9 shows the frequency of the type of providers respondents were thinking of as a “single
professional” when responding to Q51.
Exhibit 9.
“Single Professional” Provider Type Frequency
Responses to Q51
Count*
Primary care doctor
4
VA
1
Doesn't have one but primary is closest
2
Doesn't have one
*One respondent was not probed on this item.
1
Two respondents said their primary care doctor provides most of their referrals. However, neither
respondent considered this as “coordinating” their care because the act of scheduling appointments
falls to the patient, and the patient is free to see someone other than the referred provider. One
respondent said “When you say ‘coordinating’, she [the doctor] looks at things and she makes a
suggestion but it's up to me which doctor I would like to go to…" The respondent who said they
didn’t have a single professional coordinating their care understood the question to mean one person
who they can call for all of their care, which the respondent does not have. The respondent felt they
do all the coordination themselves so answered the question with a response of “No”.
Terminology: “Care Plan” (Q60, Q61, Q62, Q63)
The term “care plan” appears in the introduction to the section titled “Your experiences of health
care” (before Q46) and four subsequent questions (Q60, Q61, Q62, and Q63). Probing was
conducted specifically on Q60 though the others share similar terms. This item was selected for
APRIL 29, 2021 | 13
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
probing to test whether the term “care plan” would have meaning to a respondent from the U.S. and
whether MCBS respondents would be able to accurately respond to the question.
Exhibit 10. “Care Plan” Question Text
Question
Number
Introduction to
“Your
experience of
health care”
section, between
Q45 and Q46
Question Text*
The following questions concern the care you are receiving from primary care
services for your health and wellbeing. Primary care services are usually the
first point of contact with the health service for a wide range of new and longterm health problems. They are frequently delivered by teams of health
professionals, usually coordinated by a family doctor or a general practitioner
working in a primary care centre. In your case this may be also another
member of the primary care team, such as a nurse, or another specialist doctor.
We also want to ask you in relation to a care plan, which is an agreement
between you and your health professionals detailing the type of health care you
need and how this care will be given. Not everybody has a care plan.
60
Do you have a care plan that takes into account all your health and wellbeing
needs?
61
Is this care plan (or plan of care) available to you?
62
To what extent have you found your care plan (or plan of care) useful for you
to manage your health and wellbeing?
63
To what extent do all the professionals involved in your care appear to be
following the same care plan?
* Underlining added to Q60-63 emphasize the terms being tested; underlining was not part of the
actual question. Underlining for the introduction to “Your experience of health care” section,
between Q45 and Q46, is part of the original instrument.
Results
Five of the nine respondents did not understand the meaning of “care plan,” as shown in Exhibit 11.
Respondents who did not understand the term said things such as "I don't know if it's a plan, she [the
doctor] looks at everything so I am not sure if that qualifies as a plan…” and “Does doing what my
doctor says count as a care plan?” Three respondents referred to their health insurance or Medicare as
their “care plan,” which is not the correct interpretation.
Exhibit 11. “Care Plan” Cognition
Cognition
Count
Understood
4
Did not Understand
5
APRIL 29, 2021 | 14
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Not only did five respondents not understand the meaning of “care plan” when probed, four of those
five responded “Yes” to Q60, saying that they did in fact have a care plan. This suggests
measurement error in the question as respondents are saying they have a “care plan” when they don’t
know what it is and some are erroneously equating “care plan” with health insurance.
Exhibit 12. Cross Tabulation of Response to Q60 and Cognition of “Care Plan”
Response to Q60
Understood
Did not Understand
Total
Yes
1
4
5
No
2
0
2
Not Sure
1
1
2
Total
4
5
9
Terminology: “Care Records” (Q69)
This item was selected for probing because of the use of the term, “care records,” which is not
commonly used in the U.S.
Exhibit 13. “Care Records” Question Text
Question
Number
69
Question Text*
The next few questions are about your experience with your primary care centre in the
last 12 months.
Were there times when you had to repeat information that should have been in your
care records?
* Underlining added to emphasize the terms being tested; underlining was not part of the actual
question.
Results
The majority of respondents were able to define “care records” when probed, but there was still some
confusion. Most respondents defined “care records” as medical records maintained by their primary
care doctor. Two respondents defined “care records” to be their medication list, which is a narrower
definition than the question likely intended.
One respondent thought the question was asking if records are shared across physicians in different
practices. This is likely a result of not understanding the term “primary care centre,” not “care
records.”
APRIL 29, 2021 | 15
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Exhibit 14. “Care Records” Cognition
Cognition
Count
Understood
7
Did not Understand
2
Terminology: “Primary Care Centre” (Q59, Q69, Q70, Q71, Q72, Q74, Q75, Q76, Q82,
Q93)
The term “primary care centre” appears in ten questions without ever being defined. Probing on this
term was conducted specifically on Q70 and Q74 to test whether a respondent from the U.S. would
be able to accurately interpret “primary care centre.”
Exhibit 15. “Primary Care Centre” Question Text
Question
Number
Question Text*
59
How often does someone at your primary care centre talk to you about specific goals for
your health?
69
The next few questions are about your experience with your primary care centre in the
last 12 months.
Were there times when you had to repeat information that should have been in your care
records?
70
How often do you believe you have had any safety problems in your primary care centre
(such as not getting an appointment when needed; wrong or delayed diagnosis or
treatment; or problems with communications between healthcare professionals)?
71
How often did you feel that the health professionals at your primary care centre
encouraged you to talk about any concerns about your healthcare?
72
How often did you have a health problem but did not seek care because of difficulties in
travelling to your primary care centre?
74
The next questions are about the online services that your primary care centre may offer
to patients like you, such as a website, smartphone app or video consultations:
How easy is it for you to use your primary care centre’s website to look for information
or access services?
75
As far as you know, which of the following online services does your primary care
centre offer?
76
Which of the following online services of your primary care centre have you used?
APRIL 29, 2021 | 16
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Question
Number
Question Text*
82
Where was this professional working?
- A primary care centre
- An outpatient department of a hospital
- Other
- Not sure
93
In the last 12 months, have you avoided making an appointment at your primary care
centre for any reason?
* Underlining added to emphasize the terms being tested; underlining was not part of the actual
question.
Results
The term “primary care centre” was understood by most respondents and frequently equated with
their primary care doctor’s office. Upon initial probing at Q70 one respondent said she was thinking
about a recent appointment with a specialist, suggesting she didn’t really understand the term.
However, when asked what “primary care centre” meant to them, she said it meant “taking care of
the chronic stuff that is not done by a specialist.”
Exhibit 16. “Primary Care Centre” Cognition
Cognition
Count
Understood
8
Did not Understand
1
In addition to comprehending “primary care centre,” respondents had some trouble understanding
Q74, which asked about difficulties accessing the primary care centre’s website. One respondent
asked if “website” was referring to the primary care center’s actual website or if it included their
patient portal for online records. Many practices in the U.S. have separate websites for the patient
portal than for the website with practice information.
Demographics Probes (Q99, Q100, Q107, Q108, Q109)
Five questions in the “About Yourself” section were selected for additional probing. Q99, Q100,
Q107, and Q108 were selected for probing because of general sensitivity of these questions. Q109
was selected to test the cognition of the term “hamlet” by respondents in the U.S.
APRIL 29, 2021 | 17
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Exhibit 17. Demographics Items for Probing
Question
Number
99
100
107
Question Text
Are you…
- Female
- Male
- Other
- Prefer not to say
This question is about your sexual orientation. Do you identify as:
- Heterosexual or straight (that is, attracted to the opposite sex)
- Homosexual (gay or lesbian, that is, attracted to the same sex)
- Bisexual (attracted to both sexes)
- Other
- Prefer not to say
Were you born in the United States of America?
- Yes
- No (if you answered ‘No”, please state what country you are a citizen of:
____________________________
108
Are you a citizen of the United States of America?
- Yes
- No (if you answered ‘No”, please state what country you are a citizen of:
____________________________
109
Which of these options best describes the place you live in?
- A village, hamlet or rural area (fewer than 3,000 people)
- A small town (3,000 to about 15,000 people)
- A town (15,000 to about 100,0000 people)
- A city (100,000 to about 1,000,000 people)
- A large city (with over 1,000,000 people)
- Don’t know
Results on Gender (Q99)
All respondents understood the question and did not have any concerns with answering it. However,
all respondents reported they were either female or male, so this sample did not test the sensitivity of
the question with people who do not identify as male or female.
Results on Sexual Orientation (Q100)
All respondents understood the question and did not have any concerns with answering it. However,
all respondents responded that they were heterosexual so this sample did not test the sensitivity of the
question with respondents of other sexual orientations.
APRIL 29, 2021 | 18
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Results on Location of Birth (Q107)
All respondents understood the question and did not have any concerns with answering it. However,
all respondents reported they were born in the U.S. so this sample did not test the sensitivity of the
question with respondents born in other countries and now living in the U.S.
Results on Citizenship (Q108)
All respondents understood the question and did not have any concerns with answering it. However,
all respondents reported they were U.S. citizens so this sample did not test the sensitivity of the
question with respondents who are not U.S. citizens.
Results on “Hamlet” (Q109)
All respondents understood Q109 and were able to provide a fairly accurate response. One
respondent googled their town’s size in order to accurately respond to the question. Interviewers
observed that most respondents were responding based on the population estimates provided in the
response options, and not the designation of town, city, etc. For example, one respondent lives in a
municipality of 10,000 people that is officially designated as a village. They selected the response
option of “A small town (3,000 to about 15,000 people)”, not “A village, hamlet or rural area (fewer
than 3,000 people)” even though either could have been accurate.
When probed on the meaning of “hamlet”, answers ranged from describing it as a term used in
Europe or another country, a literary word, rural area, or not knowing. Respondents answered this
question based on other context clues in the question and most did not really know what a hamlet is.
Exhibit 18. Understanding of “Hamlet” (Q109)
Definition of Hamlet
Count
Term used in Europe
3
Literary term
1
Rural area
2
Does not know
2
Q50. Health Problems
This question was not selected for probing, but the cognitive interview analysis indicate that this item
was unclear to respondents and warrants further consideration.
APRIL 29, 2021 | 19
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Exhibit 19. Q50 Question Text and Response Options
Question
Number
50
Question Text
Have you ever been told by a doctor
that you have any of the following
health problems?
Please specify which health problem(s):
Response Options
- High blood pressure
- Cardiovascular or heart condition
- Diabetes (type 1 or 2)
- Arthritis or ongoing problem with back or
joints
- Ongoing depression, anxiety or other
mental health conditions (e.g. bipolar
disorder or schizophrenia)
- Breathing conditions (e.g. asthma or
COPD)
- Chronic kidney disease/Chronic liver
disease
- Cancer (diagnosis or treatment in the last 5
years)
- Alzheimer’s disease or other cause of
dementia
Neurological conditions (e.g., epilepsy or
migraine)
- Other long-term problems: ________
________________________________
- I have never been told by a doctor that I
have any of these problems…..
Results
While respondents generally understood what was being asked at this item, many were unsure of
what conditions should be included for each response option. Two respondents asked if borderline
diabetes should be included in the diabetes response category. Some conditions that were reported by
respondents in the “Other long term problems” option would not be considered chronic conditions by
most public health agencies. Responses included taking medication for their thyroid, gallbladder
issues, obesity, a motion disorder, and esophageal issues. None of these would be considered chronic
conditions under CMS definitions 3.
“Chronic Conditions”, Centers for Medicare & Medicaid Services. January 15, 2021. https://www.cms.gov/Research-StatisticsData-and-Systems/Statistics-Trends-and-Reports/Chronic-Conditions/CC_Main
3
APRIL 29, 2021 | 20
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Discussion and Recommendations
This section will discuss the findings above and make recommendations for potential modifications
to the PaRIS Survey administration to the MCBS audience.
General Findings on Administration of the PaRIS Survey
The PaRIS survey was designed to be a self-administered paper-and-pencil (PAPI) survey. However,
the cognitive interviews were administered as a telephone interview to mirror the modes most likely
to be used when implemented on the MCBS. Interviewers observed a number of challenges in
administering the PaRIS survey verbally. Once a data collection mode is selected the implementation
of the PaRIS survey on the MCBS, it is recommended that the survey be reviewed for edits to
accommodate the selected mode(s).
Discussion of Probed Terminology
Based on the findings discussed in the previous section, below are recommendations on
modifications to make when administering to the MCBS population.
■
“On the Level” (Q8) - When these items are administered on the MCBS, it would be beneficial
to replace “on the level” with a synonym better understood by respondents in the U.S., such as
“on a flat surface”.
■
“Health Professional(s)” (Q36, Q37, Q39, Q41, Q42) - Respondents would benefit from
repeating the term’s definition in the question text, as help text, or as an on-screen definition.
■
“Other Long Term Problems” (Q50) – If the analytic objectives of the PaRIS survey are to
measure the experiences of people living with chronic conditions, then it is recommended that the
wording for the “Other long term problems” category should be revised to only include chronic
conditions and exclude other general health problems.
■
“Single Professional” (Q51, Q52) - It would be helpful to move or repeat the explanation of
coordinated care from the introduction of this series (which occurs before item Q46), making it
available as help text or as an on-screen definition so that it is top of mind for respondents when
answering Q51 and Q52. If the analytic objective of these items is to measure a respondent’s
perception of whether their care is coordinated by a single professional, then migrating and/or
repeating the definition currently included before Q46 when the PaRIS survey is administered on
the MCBS is likely to be a sufficient revision. Alternatively, if the primary goal is to measure the
respondent’s experience of care coordination in the U.S. health system as compared to other
countries, then it would be helpful to include a uniform definition and examples of care
coordination at these items to assist respondents in the U.S. in answering. In lieu of additional
examples at how care is coordinated, respondents living in the U.S. are likely to interpret the
APRIL 29, 2021 | 21
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
concept of “care coordination” at this item differently than respondents in other countries,
particularly those respondents in countries that have national health systems.
■
“Care Plan” (Q60, Q61, Q62, Q63) - To aid in response, the definition of “care plan” in the
introduction to this series (which occurs between items Q45 and Q46) should be moved to
immediately precede Q60, which is when the term is first used in a question. If the analytic
objective of these items is to measure respondent’s perception of whether they have a care plan,
then migrating the definition currently included before Q46 is likely to be a sufficient revision.
Alternatively, if the primary goal is to measure the respondent’s experience of care plans in the
U.S. health system as compared to other countries then it would be helpful to include a uniform
definition and examples of care plans at these items to assist respondents in the U.S. in
answering. In lieu of additional examples or parameters around what a care plan is, respondents
living in the U.S. are likely to interpret the concept differently than respondents in other
countries, particularly those respondents in countries that have more established and standardized
care plans in use.
■
“Care Records” (Q69) - It would be beneficial to replace the term “care records” with “medical
records” to be more readily understood by respondents in the U.S. Help text or an on-screen
definition defining “care records” or “medical records” to include all types of care, not just
medication, may also be useful.
■
“Primary Care Centre” (Q59, Q69, Q70, Q71, Q72, Q74, Q75, Q76, Q82, Q93) - Given the
lack of colloquial use of the term “primary care centre” in the U.S., it would be beneficial to
substitute with an alternative term for fielding on the MCBS. Alternatives to be considered
include “primary care office,” “primary care practice,” or “primary care doctor’s office”.
Additionally, help text or on-screen definitions may be useful at Q74 to explain whether
“website” is inclusive of patient portals that may be hosted on other websites but used by the
practice or accessed through the practice website.
■
Gender (Q99) – This item performed satisfactorily. However, the MCBS already collects gender
as part of the main MCBS interview so it is recommended that the PaRIS survey use gender as
already collected by the MCBS instead of re-asking it during the PaRIS survey.
■
Sexual Orientation (Q100), Place of Birth (Q107), and Citizenship (Q108) - The items on
sexual orientation, place of birth, and citizenship performed satisfactorily and do not require
additional changes for inclusion on the MCBS.
■
Type of Municipality (Q109) - The MCBS has address information for all respondents so it
would be less burdensome and more accurate to create this variable during data processing
instead of asking it to be self-reported.
APRIL 29, 2021 | 22
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Considerations for Retaining MCBS Demographic Items
In addition to the demographic items that were probed on, there are nine other demographic items on
the PaRIS survey that obtain information already collected by the MCBS, as shown in Exhibit 20.
These nine MCBS items are slightly different when compared to the PaRIS survey due to differences
in response options or question framing. Items such as education, weight, and height on the PaRIS
survey lack cultural considerations for respondents living in the U.S., as they collect weight in kilos,
height in centimeters, and ask about educational attainment in terms not used in the U.S. Items on
income, employment status, household size, and activities of daily living, although framed differently
on the PaRIS survey, reflect the same analytic objective as their MCBS equivalents. Therefore, it is
recommended that the existing MCBS items be used when fielding the PaRIS survey on the MCBS
to reduce respondent burden.
Exhibit 20. Demographic Items from PaRIS Compared to MCBS
PaRIS Question Text
99. How old are you?
98. What is the highest
educational level that
you have attained?
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
PaRIS Response
Options
44 years old or
younger
45-49 years old
50-54 years old
55-59 years old
60-64 years old
65-69 years old
70-74 years old
75-79 years old
80-84 years old
85 years or older
Early childhood
education
Primary education
Lower secondary
education
Upper secondary
education
Post-secondary
non-tertirary
education
Short-cycle tertiary
education
Bachelor's degree
or equivalent level
Master's degree or
equivalent level
MCBS
Question Text
That makes [you/(SP)]
(AGE) today. Is that
correct?
MCBS Response
Options
(01) YES
(02) NO
What is the highest
degree or level of
school [you have/(SP)
has] completed?
(01) NO SCHOOLING
COMPLETED
(02) NURSERY
SCHOOL TO 8TH
GRADE
[IF THE SAMPLE
(03) 9TH-12TH
PERSON ATTENDED GRADE, NO
SCHOOL IN A
DIPLOMA
FOREIGN COUNTRY, (04) HIGH SCHOOL
IN AN UNGRADED
GRADUATE (HIGH
SCHOOL, HOME
SCHOOL DIPLOMA
SCHOOLING, OR
OR THE
UNDER OTHER
EQUIVALENT)
(05)
UNIQUE
VOCATIONAL/TECH
CIRCUMSTANCES,
NICAL/BUSINESS/TR
REFER THE
ADE SCHOOL
RESPONDENT TO
APRIL 29, 2021 | 23
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
PaRIS Question Text
97. How tall are you?
96. How much do you
weigh?
102. Which of these
categories does your
household income
usually fall into?
PaRIS Response
Options
• Doctoral or
equivalent level
•
•
•
•
•
•
•
•
•
103. In the past 12
months, did you have
problems paying or
•
•
•
•
___centimetres
Prefer not to answer
Don't know
_____kilos
Prefer not to answer
Don't know
MCBS
Question Text
THE SHOWCARD
AND ASK FOR THE
NEAREST
EQUIVALENT.]
How tall [are you/is
(SP)]?
How much [do
you/does (SP)] weigh?
[WEIGHT SHOULD
BE RECORDED IN
POUNDS]
Now thinking about all
Up to $X a month
of last year, that is
[LOW INCOME
calendar year
(X=$75% OF
[CURRENT YEAR –
MEDIAN
1], how much income
HOUSEHOLD
did [you/(SP)] and [(SP
INCOME)]
Between $X and $Y FIRSTNAME
a month [MEDIAN LASTNAME)/(SPOUS
E FIRSTNAME
INCOME]
$Y or more a month LASTNAME)/(PARTN
ER FIRSTNAME
[HIGH INCOME
LASTNAME)] receive
(Y=$125% OF
from these businesses
MEDIAN
or property before any
HOUSEHOLD
federal or state taxes
INCOME)]
were taken out?
Don't know
Since (LAST HF
Yes
MONTH YEAR) [have
No
you/has (SP)] had
Not applicable
MCBS Response
Options
CERTIFICATE OR
DIPLOMA (BEYOND
THE HIGH SCHOOL
LEVEL)
(06) SOME
COLLEGE, BUT NO
DEGREE
(07) ASSOCIATE
DEGREE
(08) BACHELOR'S
DEGREE
(09) MASTER'S,
PROFESSIONAL OR
DOCTORATE
DEGREE
(-8) Don't Know
(-9) Refused
(01) continuous answer
(-8) DON'T KNOW
(-9) REFUSED
(01) continuous answer
(-8) DON'T KNOW
(-9) REFUSED
(1) [continuous
response]
(-8) DON’T KNOW
(-9) REFUSED
(01) YES
(02) NO
(-8) Don't Know
APRIL 29, 2021 | 24
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
PaRIS Question Text
were unable to pay any
medical bills?
110. How many
children under the age
of 18 live with you, in
your household?
PaRIS Response
Options
_____children under
the age of 18
MCBS
Question Text
problems paying or
were unable to pay any
medical bills?
[At the time of the last
interview, [you were
living by yourself/(SP)
was living by
[himself/herself]]].
MCBS Response
Options
(-9) Refused
(01) YES
(02) NO
(-8) Don't Know
(-9) Refused
[Besides [you/(SP)],
[is/was] there anyone
else living or staying in
the household [as of
(DATE OF
DEATH/DATE OF
INSTITUTIONALIZA
TION)]? Remember to
include people who
[are/were] temporarily
absent and any children
who [may live/may
have lived] in the
household.
111. How many people
aged 18 and over live
with you, in your
household? Please do
not count yourself.
_____people aged 18
and over
[At the time of the last
interview, [you were
living by yourself/(SP)
was living by
[himself/herself]]].
(01) YES
(02) NO
(-8) Don't Know
(-9) Refused
[Besides [you/(SP)],
[is/was] there anyone
else living or staying in
the household [as of
(DATE OF
DEATH/DATE OF
INSTITUTIONALIZA
TION)]? Remember to
include people who
[are/were] temporarily
absent and any children
who [may live/may
have lived] in the
household.
APRIL 29, 2021 | 25
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
PaRIS Question Text
114. Do you need
physical care or
support, such as help
with eating, dressing,
bathing, moving around
the house or assistance
outside the house such
as for using
transportation?
PaRIS Response
Options
• Yes
• No
MCBS
Question Text
Because of a physical,
mental, emotional, or
memory problem, [do
you/does (SP)] have
any difficulty...
MCBS Response
Options
(01) YES
(02) NO
(-8) Don't Know
(-9) Refused
…bathing or
showering?
…dressing?
… eating?
… getting in or out of
bed or chairs?
… walking?
… using the toilet,
including getting up
and down?
Notes on Other Items Identified during Testing
In addition to the findings above on the items that were probed on, interviewers noted a number of
comprehension issues on other questions, outlined in Exhibit 21.
These issues should be shared with the OECD Working Group to help improve the survey for all
respondents.
Exhibit 21. Question Comprehension Issues
Question
Number
19
35
Question Text
Interviewer Observations
In the past week, on how many days did
you do at least 30 mins of either vigorous
or moderate activity (such as running,
swimming, riding a bike, brisk walking,
or dancing)?
Respondent interpreted “vigorous” as
“rigorous” leading to great confusion. For
example, they did not consider “walking”
to be rigorous for them personally.
I have difficulty understanding a lot of
the health information that I read.
The term “health information” is incredibly
broad. One respondent interpreted it as
anything from reading and understanding
lab results (which they do not understand)
to findings from nutrition research (which
they do).
APRIL 29, 2021 | 26
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Question
Number
45
Question Text
Interviewer Observations
How confident are you in using
information from the Internet to make
health decisions?
Respondent was unclear if the question
was asking about a) their ability to decide
whether a site/article was a legitimate
source of information, or b) if overall what
is on the internet is reliable. Respondent
did note that if the source is legitimate,
they may use it.
56
Are you offered regular follow-up for
any of your health problems?
Some respondents were unsure if this
question referred to a medical provider or
another person, such as a spouse.
58
Are you considered as a ‘whole person’
rather than just a disease/condition in
relation to your care?
Some respondents were unclear what was
being asked at this item- specifically they
were unfamiliar with the phrase "whole
being.”
80
The next questions are about the last time
that you had a consultation with a
primary care doctor, nurse or other health
professional to get care for yourself (this
may have been with your main health
professional or another health
professional and includes consultations
over the phone or in a doctor’s office, a
clinic, or the outpatient department of a
hospital. Do not include care you got
when you stayed overnight in a hospital,
dental care visits or accident and
emergency care.
Respondent was very specifically focused
on the word “consultation” from the
section introduction preceding this
question. Question had to be repeated.
What was the purpose of this visit?
81
Who did you see?
Respondent was unsure on how to answer
this question if the primary care doctor had
referred them to a specialist.
102
Which of these categories does your
household income usually fall into?
Multiple respondents were unsure of their
monthly income, but were aware of their
annual income.
Conclusion
In general, the PaRIS survey performed well during cognitive testing. Clearly some wording changes
should be made if the survey is administered to MCBS respondents. In addition, there are other
findings that should be shared with the OECD Working Group as part of the Field Trial Phase
(Phase 2).
APRIL 29, 2021 | 27
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Appendix A. PaRIS Cognitive Interview Screener
PARTICIPANT NUMERIC IDENTIFIER: ___________________________
1. BENEFICIARY SEX [CODE WITHOUT ASKING IF POSSIBLE.]
a) MALE
b) FEMALE
2. How old are you?
_______ years
3. Medicare is the federal health insurance program for people who are 65 or older, certain younger
people with disabilities, and people with End-Stage Renal Disease. I need to confirm, do you receive
health insurance through Medicare?
a) YES
b) NO I am sorry, but only people who receive insurance through Medicare are eligible
for this study.
READ IF NECESSARY: Do you have a Medicare card? Medicare Part A includes coverage for hospital
stays and Part B includes coverage for doctor’s services. Part C, Medicare Advantage Plans, is offered
through private insurance companies under contract with Medicare. Some people opt to add on
Part D, which is prescription drug coverage.
4. I am going to read you a list of medical conditions. Have you ever been told by a doctor that you
have any of the following health problems? For each condition, please respond yes or no.
INTERVIEWER: READ EACH CONDITION SLOWLY AND CIRCLE YES OR NO FOR EACH CONDITION.
High blood pressure
YES / NO
Cardiovascular or heart condition
YES / NO
Diabetes (type 1 or 2)
YES / NO
Arthritis or ongoing problems with back or joints
YES / NO
Ongoing depression, anxiety, or other mental health condition (e.g.
bipolar disorder or schizophrenia)
YES / NO
Breathing condition (e.g. asthma or COPD)
YES / NO
Chronic kidney disease / Chronic liver disease
YES / NO
Cancer (diagnosis or treatment in the last 5 years)
YES / NO
Alzheimer’s disease or other cause of dementia
YES / NO
Neurological condition (e. g. epilepsy or migraine)
YES / NO
APRIL 29, 2021 | 28
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
IF RESPONDENT HAS NONE OF THE ABOVE CONDITIONS THANK THEM USING THIS SCRIPT:
Thank you for volunteering for this study. Unfortunately, you do not meet the criteria for this study. I
appreciate the time you took to speak with me today.
5.
What is the highest degree or level of school you have completed?
a) NO SCHOOLING COMPLETED
b) NURSERY SCHOOL TO 8TH GRADE
c) 9TH-12TH GRADE, NO DIPLOMA
d) HIGH SCHOOL GRADUATE (HIGH SCHOOL DIPLOMA OR THE EQUIVALENT)
e) VOCATIONAL/TECHNICAL/BUSINESS/TRADE SCHOOL CERTIFICATE OR DIPLOMA (BEYOND
THE HIGH SCHOOL LEVEL)
f) SOME COLLEGE, BUT NO DEGREE
g) ASSOCIATE DEGREE
h) BACHELOR'S DEGREE
i) MASTER'S, PROFESSIONAL OR DOCTORATE DEGREE
6. Are you of Hispanic, Latino, or Spanish origin?
a) YES
b) NO
7. What is your race? [SELECT ALL THAT APPLY. READ RESPONSE OPTIONS IF NEEDED.]
a) American Indian or Alaska Native
b) Asian
c) Black or African American
d) Native Hawaiian or other Pacific Islander
e) White
8. Would you prefer to conduct the interview over the phone or over video, using Zoom?
a) PHONE
b) ZOOM
9. CONFIRM CONTACT INFORMATION AND SCHEDULE APPOINTMENT:
Ok, let’s schedule an appointment to do the interview.
[TAKE INFORMATION]
IF PHONE:
10. Can you confirm that [PHONE NUMBER] is the best number to contact you on [DATE] for the
interview?
[TAKE INFORMATION]
IF ZOOM:
APRIL 29, 2021 | 29
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
11. Could you please provide an email address to send you the Zoom link information to for the
interview?
[TAKE INFORMATION]
Thank you for volunteering to participate. We will speak with you on [REPEAT DATE AND TIME OF
APPOINTMENT] by [REPEAT MODE]. I will reach out the day before the interview to remind you of the
appointment and to confirm the name [and phone number] of the interviewer who will be contacting
you.
AFTER HANGING UP, ENTER THE INFORMATION FROM THIS SCREENER INTO THE RECRUITMENT
SPREADSHEET ALONG WITH THE DATE, TIME, AND MODE OF THE APPOINTMENT.
APRIL 29, 2021 | 30
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
Appendix B. PaRIS Cognitive Interview Questionnaire
MATERIALS NEEDED FOR INTERVIEW
□
□
INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET) AND SHOWCARD
PaRIS QUESTIONNAIRE
STEP 1: PREPARE THE RESPONDENT
LOG ONTO THE ZOOM OR PHONE CALL WITH THE RESPONDENT, INTRODUCE YOURSELF, AND
MAKE SURE THEY’RE READY TO START.
EXAMPLE SCRIPTS:
• “Hello, my name is [NAME] from NORC at the University of Chicago. Is [RESPONDENT
NAME] available?”
• “I’ll explain a little more about what we’re doing and then if you don’t have any
questions we can get started. The interview will take about an hour. After the interview
is done I’ll take your information and we’ll mail you a check for $40 to thank you for
participating.”
STEP 2: COMPLETION OF THE QUESTIONNAIRE
COMPLETE THE FOLLOWING QUESTIONS AND THEN ADMINISTER THE PARIS QUESTIONNAIRE.
INT1. The Medicare Current Beneficiary Survey (MCBS) asks Medicare beneficiaries about their health
status, sources of health care, satisfaction with care, and health care expenditures. In today’s interview I
will be asking you about your experiences with health care providers.
Periodically during the interview, I will ask you some questions about how you decided to answer
some of the survey questions. If at any point, we ask a question or use a word that you do not
understand, please let me know so we can discuss it further. Getting your feedback on the questions
can help make the questions better.
[CONTINUE]
INT2. All survey information will be kept private to the extent permitted by law, as prescribed by the
Privacy Act of 1974. Medicare benefits will not be affected in any way by survey responses or
participation. You may skip questions or end the interview at any time and you will still receive the
$40 payment.
Do you agree to participate in this interview?
a. YES -> GO TO NEXT QUESTION
b. NO -> STOP INTERVIEW AND THANK THEM FOR THEIR TIME.
APRIL 29, 2021 | 31
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
INT3. In order to have a complete record of your comments, with your permission, your interview
session will be recorded. The recording will be stored electronically on NORC’s secure servers. We
plan to use the recording to ensure that we capture all of the feedback you provide us. Only staff
directly involved in this research project will have access to the recording. Any quotes used in
presentations and publications will not include any names or any information that could identify any
participant.
Is it okay for me to proceed with the recording on?
a. YES -> TURN RECORDING ON
b. NO -> PROCEED WITHOUT RECORDING
APRIL 29, 2021 | 32
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
4 DELSA/HEA/PARIS(2021)2
Your health
05
In general, how would you rate your
satisfaction with your social activities
and relationships?
06
In general, please rate how well you
carry out your usual social activities
and roles. (This includes activities at
home, at work and in your community,
and responsibilities as a parent, child,
spouse, employee, friend, etc.)
07
To what extent are you able to carry out
your everyday physical activities such
as walking, climbing stairs, carrying
groceries, or moving a chair?
The first few questions are about your health.
01
In general, would you say your health
is:
02
In general, would you say your quality
of life is:
03
In general, how would you rate your
physical health?
04
In general, how would you rate your
mental health, including your mood and
your ability to think?
For Official Use
APRIL 29, 2021 | 33
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
5
The next few questions are about any
specific symptoms you may have
recently experienced.
08
12
In the past 7 days, how would you rate
your pain on average?
13
In the past 7 days, how often have you
been bothered by emotional problems
such as feeling anxious, depressed or
irritable?
Please select the response that best
describes your current situation in
relation to your breathing:
1.
2.
3.
4.
5.
09
In the past 7 days, I had a problem with
my sleep
10
In the past 7 days, how would you rate
your fatigue on average?
For the next few questions please think about
how you have been feeling over the last two
weeks.
14
11
I have felt cheerful and in good spirits
1.
2.
In the past 7 days, how much did pain
interfere with your day to day activities?
3.
4.
5.
6.
RESPONSE OPTIONS CONTINUE
For Official Use
APRIL 29, 2021 | 34
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
6 DELSA/HEA/PARIS(2021)2
15
I have felt calm and relaxed
1.
The next few questions ask about your
recent physical activity, diet and use of
tobacco and alcohol and the support you
may have received from health
professionals (doctors, nurses, and other
health professionals).
2.
3.
4.
5.
6.
16
Managing your health and
health care
I have felt active and vigorous
19
In the past week, on how many days did
you do at least 30 mins of either
vigorous or moderate activity (such as
running, swimming, riding a bike, brisk
walking, or dancing)?
20
In the past 12 months, has any health
professional talked with you about your
physical activity?
21
How often do you eat fruits (excluding
juice)?
1.
2.
3.
4.
5.
6.
17
My daily life has been filled with things
that interest me
1.
2.
3.
4.
5.
6.
18
I woke up feeling fresh and rested
1.
2.
3.
4.
5.
6.
1.
2.
3.
4.
5.
6.
7.
For Official Use
APRIL 29, 2021 | 35
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
22
How often do you eat vegetables or
salad (excluding juice and potatoes)?
27
1.
2.
7
During the past 12 months, how often
have you had an alcoholic drink of any
kind (that is beer, wine, spirits, liqueurs
or other alcoholic beverages)?
Go to Q29
3.
4.
5.
6.
7.
23
In the past 12 months, has any health
professional talked with you about
healthy eating?
24
Do you smoke at all nowadays?
Go to Q26
Go to Q26
25
28
In the past 12 months, has any health
professional talked with you about
alcohol use?
The next few questions about how you and
the health professionals that you usually see
support your health and health care.
29
I rely on health professionals to tell me
everything I need to know to manage
my health
30
Most health issues are too complex for
me to understand
Have you ever smoked daily, or almost
daily, for at least one year?
Go to Q26
Go to Q27
26
In the past 12 months, has any health
professional talked with you about the
health risks of smoking or using
tobacco and ways to quit?
For Official Use
APRIL 29, 2021 | 36
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
8 DELSA/HEA/PARIS(2021)2
31
32
33
34
35
I actively try to prevent diseases and
illnesses
36
My health professionals and I work
together to manage my health
37
When I read or hear something that is
relevant to my health care, I bring it up
with health professionals
38
I try to understand my personal health
risks
39
When I think something is wrong with
my health care, I raise my concerns
with my health professionals
I leave it to health professionals to
make the right decisions about my
health
It is important to me to be informed
about health issues
I need to know about health issues so I
can keep myself and my family healthy
I have difficulty understanding a lot of
the health information that I read
RESPONSE OPTIONS CONTINUE
For Official Use
APRIL 29, 2021 | 37
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
40
How confident are you that you can
manage your own health and
wellbeing?
41
How confident are you that you can
follow instructions from health
professionals about how you should
care for yourself at home?
42
43
How confident are you that you can
follow instructions from health
professionals about how to change
your habits or lifestyle?
How confident are you that you can
identify when it is necessary for you to
get medical care?
9
44
How confident are you that you can
identify when you are having side
effects from your medications?
45
How confident are you in using
information from the Internet to make
health decisions?
Your experience of health
care
The following questions concern the care
you are receiving from primary care
services for your health and wellbeing.
Primary care services are usually the first
point of contact with the health service for a
wide range of new and long-term health
problems. They are frequently delivered by
teams of health professionals, usually
coordinated by a family doctor or a general
practitioner working in a primary care
centre. In your case this may be also
another member of the primary care team,
such as a nurse, or another specialist
doctor. We also want to ask you in relation
to a care plan, which is an agreement
between you and your health professionals
detailing the type of health care you need
and how this care will be given. Not
everybody has a care plan.
For Official Use
APRIL 29, 2021 | 38
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
10 DELSA/HEA/PARIS(2021)2
46
Is there one doctor you usually go to for
most of your health problems?
50
Have you ever been told by a doctor that you
have any of the following health problems?
Please specify which health problem(s):
1.
Go to Q50
Go to Q50
Go to Q50
47
Who is this doctor?
2.
3.
4.
5.
6.
7.
8.
9.
10.
48
How long have you been seeing this
doctor?
1.
2.
99.
Go to Q69
3.
4.
The next few questions are about the care
you may have received for these health
conditions.
5.
6.
49
How often have you been to this
doctor´s office in the past 12 months?
51
I have not visited in the past 12
months
Do you have a single professional who
takes responsibility for coordinating
your care across the services that you
use?
Go to Q54
Go to Q54
52
Is this the same professional that you
see for most of your health problems?
Go to Q54
RESPONSE OPTIONS CONTINUE
For Official Use
APRIL 29, 2021 | 39
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
53
11
57
Are you involved as much as you
wanted to be in decisions about your
care?
58
Are you considered as a ‘whole person’
rather than just a disease/condition in
relation to your care?
59
How often does someone at your
primary care centre talk to you about
specific goals for your health?
Who is this professional?
1.
2.
3.
4.
5.
6.
7.
8.
54
55
Is your healthcare organized in a way
that works for you?
Are you offered regular follow-up for any
of your health problems?
1.
2.
3.
6.
7.
4.
5.
6.
56
1.
2.
3.
4.
5.
Do you discuss what is most important
for you in managing your own health
and wellbeing?
60
Do you have a care plan that takes into
account all your health and wellbeing
needs?
Go to Q64
Go to Q64
For Official Use
APRIL 29, 2021 | 40
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
12 DELSA/HEA/PARIS(2021)2
61
Is this care plan (or plan of care)
available to you?
3.
4.
5.
6.
62
66
To what extent have you found your care
plan (or plan of care) useful for you to
manage your health and wellbeing?
Are your family, friends and/or carers
involved as much as you wanted them
to be in decisions about your care?
1.
2.
3.
4.
63
To what extent do all the professionals
involved in your care appear to be
following the same care plan?
5.
6.
67
How many different medications as
prescribed by a doctor or a nurse are
you taking on a regular or ongoing
basis?
Go to Q69
64
Do you have enough support from the
health care professionals to help you to
manage your own health and wellbeing?
1.
2.
68
3.
4.
In the past 12 months, has a health
professional reviewed with you the
medications you take?
5.
6.
7.
65
To what extent do you receive useful
information at the time you need it to
help you manage your health and
wellbeing?
1.
2.
RESPONSE OPTIONS CONTINUE
For Official Use
APRIL 29, 2021 | 41
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
The next few questions are about your
experience with your primary care centre in
the last 12 months.
69
Were there times when you had to
repeat information that should have
been in your care records?
72
2.
3.
6.
73
How often do you believe you have had
any safety problems in your primary
care centre (such as not getting an
appointment when needed; wrong or
delayed diagnosis or treatment; or
problems with communications
between healthcare professionals)?
2.
3.
4.
5.
6.
71
How often did you feel that the health
professionals at your primary care
centre encouraged you to talk about
any concerns about your healthcare?
2.
3.
6.
The next questions are about the online
services that your primary care centre may
offer to patients like you, such as a website,
smartphone app or video consultations:
74
1.
2.
3.
4.
5.
6.
How often did you have a health
problem but did not seek care, or did
take a prescription medicine because of
the cost?
1.
4.
5.
1.
How often did you have a health
problem but did not seek care because
of difficulties in travelling to your
primary care centre?
1.
4.
5.
70
13
How easy is it for you to use your
primary care centre’s website to look
for information or access services?
1.
2.
3.
4.
5.
6.
7.
For Official Use
APRIL 29, 2021 | 42
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
14 DELSA/HEA/PARIS(2021)2
75
As far as you know, which of the
following online services does your
primary care centre offer?
1.
Other health services
78
Have you been in a hospital emergency
department for your own medical care?
79
Have you been in the hospital for one
night or longer?
2.
3.
4.
5.
6.
76
Which of the following online services
of your primary care centre have you
used?
1.
2.
3.
4.
5.
6.
When taking all things into consideration in
relation to the care you have received…
77
Overall, how do rate the medical care
that you have received in the past 12
months from your primary care centre?
1.
2.
3.
4.
5.
6.
7.
The next questions are about the last time
that you had a consultation with a primary
care doctor, nurse or other health
professional to get care for yourself (this may
have been with your main health professional
or another health professional and includes
consultations over the phone or in a doctor’s
office, a clinic, or the outpatient department
of a hospital. Do not include care you got
when you stayed overnight in a hospital,
dental care visits or accident and emergency
care.
80
What was the purpose of this visit?
1.
2.
3.
4.
5.
For Official Use
APRIL 29, 2021 | 43
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
81
Who did you see?
15
85
Was the time you waited a problem for
you?
86
Did this health professional spend
enough time with you?
87
Did this health professional explain
things in a way that was easy to
understand?
88
Did this health professional give you an
opportunity to ask questions or raise
concerns about recommended treatment?
1.
2.
3.
4.
5.
6.
82
83
Where was this professional working?
Which of the following best describes
the type of care you received?
1.
2.
3.
4.
5.
6.
84
How quickly did you get an
appointment to see this health care
professional?
For Official Use
APRIL 29, 2021 | 44
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
16 DELSA/HEA/PARIS(2021)2
89
Did you have confidence and trust in
the healthcare professional you saw or
spoke to?
93
In the last 12 months, have you avoided
making an appointment at your primary
care centre for any reasons?
1.
2.
3.
4.
5.
6.
7.
90
Overall, how would you rate the quality
of this consultation?
8.
94
In the last 12 months, how often have
you been concerned that you may not
be receiving the care you needed
because services were focusing on the
response to the COVID-19 pandemic?
The next questions are about your experience
of care in relation to the COVID/coronavirus
pandemic.
91
92
In the last 12 months has a health care
professional confirmed that you have
had COVID-19?
About yourself
Finally, we would like to ask you about some
information about yourself. It is important to
collect this information to understand
whether there are any differences in the
quality of services experienced by different
groups of people.
In the last 12 months have you been
vaccinated for COVID-19?
For Official Use
APRIL 29, 2021 | 45
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
95
How old are you?
99
100
17
Are you…?
This question is about your sexual
orientation. Do you identify as:
1.
2.
96
How much do you weigh?
3.
4.
______ lbs
5.
101
97
How tall are you?
______ feet ____ inches
Which of these best describes your
current work situation?
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
For Official Use
APRIL 29, 2021 | 46
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
18 DELSA/HEA/PARIS(2021)2
102
Which of these categories does your
household income usually fall into?
106
Having enough money to pay for other
monthly bills, like electricity, heat, and
your telephone?
107
Were you born in the United States
of America?
108
Are you a citizen of the United States
of America?
Up to $4,020 a month
Between $4,020 and $6,700 a month
$6,700 or more a month
103
In the past 12 months, did you have
problems paying or were unable to pay
any medical bills?
How often in the past 12 months would you
say you were worried or stressed about the
following things?
104
Having enough money to buy nutritious
meals?
109
Which of these options best describes
the place you live in?
1.
2.
3.
105
Having enough money to pay your rent
or mortgage?
4.
5.
6.
110
How many children under the age of 18
live with you, in your household?
For Official Use
APRIL 29, 2021 | 47
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
DELSA/HEA/PARIS(2021)2
111
How many people aged 18 and over live
with you, in your household? Please do
not count yourself.
112
Should you need help, how easy is it for
you to get help from a close family
member (including your partner)?
113
114
19
116
Do you need support for health care,
such as administering medicines,
changing bandages or arranging for
appointments with primary care
providers?
117
How strongly do you agree or disagree
that the health care system can be
trusted?
118
Who was the main person or people
that filled in this questionnaire?
Should you need help, how easy is it for
you to get help from friends,
neighbours and co-workers?
Do you need physical care or support,
such as help with eating, dressing,
bathing, moving around the house or
assistance outside the house such as
for using transportation?
Thank you for taking the time to
complete this questionnaire.
115
Do you need emotional care or support,
such as comfort, advice or counseling?
For Official Use
APRIL 29, 2021 | 48
NORC | MCBS Task 9.30 PaRIS Cognitive Testing Report
Thank you for participating in the interview today. To thank you for your time, we’d like to send you a
check for $40. Could you please provide me with an address to mail the check to?
NAME: __________________________________
STREET: _________________________________________________________________
CITY: ___________________________________________________________________
STATE: __________________________ ZIP: _________________________________
GENERAL PROBES: Suggested general neutral probing for issues that arise.
•
•
•
•
•
•
•
•
How did you decide on that answer?
Can you tell me more about that?
Can you give me an example of that?
Tell me what you are thinking.
What did you think about when I asked that question?
What did you think about in deciding on your answer?
What doctors did you include when you answered this question?
What does [QUESTION/TERM] mean to you?
SEXUAL ORIENTATION AND GENDER IDENTIFY PROBES: Suggested general neutral probing specifically
for the SOGI items.
•
•
•
•
•
Did you have any trouble deciding on that answer?
o If YES, What were you concerned about?
Were there any words that you were uncertain about?
o Which ones?
o How did this affect your answer?
What does Bisexual mean to you?
[IF R ANSWERED: OTHER IN RESPECT TO GENDER IDENTITY: ]
o What else would best represent how you think about yourself?
[IF R ANSWERED: I don’t know how to answer: ]
o What makes it difficult for you to answer this question?
APRIL 29, 2021 | 49
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Appendix C. PaRIS Cognitive Interview Probes
Exhibit 22. PaRIS Cognitive Interview Probes
Survey Section
Question Text
Probes
Your Health
8. Please select the response that best
describes your current situation in
relation to your breathing:
- I only get breathless with strenuous
exercise
- I get short of breath when hurrying
on the level of up a slight hill
- I walk slower than people of the
same age on the level because of
breathlessness or have to stop for
breath when walking at my own pace
on the level
- I stop for breath after walking 100
yards or after a few minutes on the
level
- I am too breathless to leave the
house
What were you thinking about when you
responded to this question?
What does "on the level" mean to you?
Your Health
n/a
Before I move on to the next section,
were there any questions or concepts in
this section that you found to be
confusing?
Managing your
health and
healthcare
36. My health professionals and I
work together to manage my health
What were you thinking about when you
responded to this question?
Who do you consider to be "health
professionals"?
Managing your
health and
healthcare
n/a
Were there any (other) questions or
concepts in this section that you found to
be confusing?
Your
experience of
health care
51. Do you have a single professional
who takes responsibility for
coordinating your care across the
services that you use?
What were you thinking about when you
responded to this question?
How did you decide on your answer?
If answer was yes, who is the person you
were thinking of?
Your
experience of
health care
60. Do you have a care plan that
takes into account all your health and
wellbeing needs?
What were you thinking about when you
responded to this question?
How did you decide on your answer?
What does the term "care plan" mean to
you?
APRIL 29, 2021 | 50
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Survey Section
Your
experience of
health care
Question Text
Probes
69. The next few questions are about
your experience with your primary
care centre in the last 12 months.
What were you thinking about when you
responded to this question?
How did you decide on your answer?
What does the term "care records" mean
to you?
Were there times when you had to
repeat information that should have
been in your care records?
Your
experience of
health care
69. Were there times when you had
to repeat information that should
have been in your care records?
What were you thinking about when you
responded to this question?
How did you decide on your answer?
What does the term "care records" mean
to you?
Your
experience of
health care
70. How often do you believe you
have had any safety problems in your
primary care centre (such as not
getting an appointment when needed;
wrong or delayed diagnosis or
treatment; or problems with
communications between healthcare
professionals)?
What were you thinking about when you
responded to these questions?
How did you decide on your answer?
What does the term "primary care centre"
mean to you?
Your
experience of
health care
74. The next questions are about the
online services that your primary
care centre may offer to patients like
you, such as a website, smartphone
app or video consultations:
What were you thinking about when you
responded to these questions?
How did you decide on your answer?
What does the term "primary care centre"
mean to you in this question?
How easy is it for you to use your
primary care centre’s website to look
for information or access services?
Your
experience of
health care
n/a
Were there any (other) questions or
concepts in this section that you found to
be confusing?
Other health
services
n/a
Were there any questions or concepts in
this section that you found to be
confusing?
About yourself
99. Are you…?
- Female
- Male
- Other
- Prefer not to say
What did you think about when I asked
this question?
How did you decide on your answer?
APRIL 29, 2021 | 51
NORC
| MCBS Task 9.30 PaRIS Cognitive Testing Report
Survey Section
Question Text
Probes
About yourself
100. This question is about your
sexual orientation. Do you identify
as:
- Heterosexual or straight (that is,
attracted to the opposite sex)
- Homosexual (gay or lesbian, that is,
attracted to the same sex)
- Bisexual (attracted to both sexes)
- Other
- Prefer not to say
What did you think about when I asked
this question?
How did you decide on your answer?
Were there any terms that you were
uncertain about? Which ones?
About yourself
107. Were you born in the United
States of America?
What did you think about when I asked
this question?
How did you decide on your answer?
About yourself
108. Are you a citizen of the United
States of America?
What did you think about when I asked
this question?
How did you decide on your answer?
About yourself
109. Which of these options best
describes the place you live in?
- A village, hamlet or rural area
(fewer than 3,000 people)
- A small town (3,000 to about
15,000 people)
- A town (15,000 to about 100,0000
people)
- A city (100,000 to about 1,000,000
people)
- A large city (with over 1,000,000
people)
- Don’t know
What were you thinking about when you
responded to these questions?
How did you decide on your answer?
What does “hamlet” mean to you?
About yourself
n/a
Were there any questions or concepts in
this section that you found to be
confusing?
APRIL 29, 2021 | 52
| File Type | application/pdf |
| File Title | MCBS Task 9.30 PaRIS Cognitive Testing Report |
| Subject | MCBS; questionnaire; cognitive testing report |
| Author | NORC at the University of Chicago |
| File Modified | 2021-05-12 |
| File Created | 2021-05-06 |