Cognitive and Field Testing Evaluation of PRAMS Social Determinants of Health Supplement

Cognitive and Field Testing Evaluation of the
Pregnancy Risk Assessment Monitoring System (PRAMS),
Social Determinants of Health Supplement

Prepared for
Council of State and Territorial Epidemiologists (CSTE)
Centers for Disease Control and Prevention (CDC), Division of Reproductive Health
Prepared by
CareHalo Research Center (CRC)

Cooperative Agreement number 1 NU38OT000297

Contents
Introduction ........................................................................................................................................ 3
Methodology ....................................................................................................................................... 3
Testing Emphasis Topics.................................................................................................................. 4
Sampling and Respondent Demographics ....................................................................................... 4
Tables 1 – 2a: Demographic Summary of Respondents for each round of testing......................... 6
Cognitive Testing Demographics: ............................................................................................... 6
Field testing Demographics: ........................................................................................................ 7
Results ................................................................................................................................................. 8
Cognitive Testing Question-by-Question Analysis .......................................................................... 8
Field Testing Debriefing Question-by-Question Analysis ............................................................. 24
Summary Results by Testing Emphasis Topics .............................................................................. 26
Timeframe .................................................................................................................................. 26
Order of the Scaled Options ....................................................................................................... 26
5-point vs. 3-point Likert-scale Options .................................................................................... 26
Appendices ........................................................................................................................................ 27
Appendix A - SDoH Cognitive Testing Instructions ..................................................................... 28
Appendix B - Cognitive and Field Testing Pre-screening Questionnaire...................................... 29
Appendix C - SDoH Cognitive Testing Instruments November 15, 2021 .................................... 30
Appendix D - CareHalo Research Center (CRC) Cognitive Testing Consolidated Brief ............. 31
Appendix E - Field Testing Debrief Instructions ........................................................................... 32
Appendix F - Field Testing Questionnaire..................................................................................... 33
Appendix G - SDoH Field Testing Instruments, December 17, 2021 ........................................... 34
Appendix H - CareHalo Research Center (CRC) Field Testing Consolidated Brief ..................... 35

2

Introduction
This report documents cognitive and field testing results on the social determinants of health (SDOH)
supplement as part of the Pregnancy Risk Assessment Monitoring System (PRAMS), the Centers for Disease
Control and Prevention (CDC). The Council of State and Territorial Epidemiologists (CSTE) partnered with
PRAMS (https://www.cdc.gov/prams/index.htm), a near-national surveillance project that collects
jurisdiction-specific, population-based data on maternal attitudes and experiences before, during, and shortly
after pregnancy to support the implementation of the SDOH supplement. The supplement will use the
existing PRAMS surveillance infrastructure to identify and help address gaps in understanding the impact of
social determinants of health on maternal and infant health.
CSTE and CDC PRAMS sought to conduct cognitive and field testing to ensure the validity and
appropriateness of proposed supplement. CareHalo Research Center (CRC) was contracted to lead the testing
activities, utilizing guidance provided by CSTE and CDC. CRC specializes in creating and executing
customized research, data collection, testing and analysis and has a collective 20+ years of related project
experience, including similar work with the National Institute of Health (NIH) survey development projects.
Testing was conducted on four variations of the SDOH supplement: English and Spanish in both Phone and
Mail. Testing was designed to be conducted in two phases (cognitive and field testing).
The following section describes the methodology of cognitive and field testing, including the procedures for
sampling interview respondents, interview questions, incentives and the interview technique itself.
Lastly, the report presents a question-by-question analysis of the supplement and a summary of general
findings. Discussion of each question includes descriptive data, detailing the respondent’s interpretation of
question intent and explanations of identified issues/problems. Where possible, comments have been
included.

Methodology
The CRC team obtained an exemption for the testing as non-research from an Institutional Review Board
(IRB), Biomedical Research Alliance. Interviewers were then trained on the testing protocol utilizing
materials provided by Centers for Disease Control and Prevention (CDC) to conduct both the cognitive
testing and field testing.
Interviewer training included review of confidentiality measures, familiarization of survey tools
(questionnaire and debriefing questions), and how to properly administer probing techniques. They were also
instructed to take copious notes while being cautious not to document any identifiable information.
CRC coordinated with two local pediatric clinics to identify eligible survey participants. Both practices have
large patient volume and high patient diversity. In efforts to reduce low to missed connections and increase
participation rates, providers extended invitations to their patients to participate in the survey. An office
representative identified eligible participants with scheduled appointments within the field testing windows.
Testing was conducted in two rounds/phases, cognitive testing and field testing. Cognitive testing is a
valuable resource used when developing questionnaire items. This commonly used qualitative methodology
can provide researchers with rich data for the analysis and interpretation of survey questions. It is used to
ensure respondents ability to understand standardized terminology and ensure the survey questions capture
the scientific intent. Cognitive testing methodology weigh heavily on the respondent narratives, which are
collected using verbal probing techniques.

3

Field testing survey instruments provides data that assist with refining and improving the questionnaire
layout and survey questions. The field test results assist with the identification of needed revisions to avoid
vague or unclear text, bias, and redundancy of the questions, or to adjust word/terminology choice.
Testing was spread across four questionnaire versions: English Mail and Phone, and Spanish Mail and
Phone. The testing emphasis included obtaining data on the phrasing of the timeframe, and the use of Likertscale responses. The following section, “Testing Emphasis Topics” includes details of these testing goals.

Testing Emphasis Topics
CDC provided directives on the testing emphasis/focus of the cognitive phase. The first included testing the
phrasing of the timeframe of several questions. Results would assist with determining the importance of the
use of consistent language throughout the survey in efforts to decrease confusion for respondents. The initial
cognitive testing survey included several variations (“During the past 12 months”, “During the last 12
months”, and “In the last 12 months”).
The second testing emphasis/focus was to assess the use of Likert-scale options present in seven questions.
Scaled response options include a set of ordered responses with one that is greater or more preferential than
another. Generally these response sets include between 3-5 options. Testing results will provide insight on
respondent’s ability to decide between “sometimes” and “usually” when presented with these options on a 5point scale. Results will also determine the efficacy of the use of the options “never”, “sometimes” and
“always” on a 3-point scale. Interviewers questioned participants about the use of these scales, to ensure
understanding an ease of selection when presented with each. Testing will allow for insight on the best way
to present these options and the necessity to use consistent size (5-point or 3-point) throughout the survey.
Interviewers were provided with detailed instructions for each question’s intent and how to capture the
necessary data from respondents on their interactions. Respondents were asked questions about their
understanding of the content, feedback on layout and formatting, specifically how it impacted their ability to
easily answer the question. Respondents were asked if options were appropriately captured and if not, what
were their recommendations. Spanish respondents were asked about the translation of specific terms and
words. Interviewers were trained to probe respondents for specific information on each of the 20 questions.
For more information, please see Appendix A, Cognitive Testing Instructions.

Sampling and Respondent Demographics
Testing was conducted in two Atlanta based, provider offices/clinics – A total of 19 interviews were
conducted, nine in the cognitive phase, and ten in field testing phase. While women representing diverse
demographic groups were recruited, sampling was not designed to draw conclusion/analysis of sub-groups.
Testing emphasis was not focused on survey population, rather coverage of survey questions (format,
text/terms, etc.).
The sample for cognitive and field testing targeted women who delivered a live born infant and who are ≤12
months postpartum. In efforts to obtain a variety of interpretations, women representing diverse
demographics (race/ethnicity, education/reading levels and native language) were recruited and sought.
Trained CRC interviewers scheduled specific dates/times with clinics to conduct testing on-site in the
practice location. Recruitment consisted of CRC staff approaching women identified by the clinical office
representative or women waiting for their appointments in the clinic. A pre-screening questionnaire was
utilized to confirm participant eligibility. Questions were designed to collect demographic information, age
of the child, education level, and native/preferred language (see Appendix B, Pre-screening Questionnaire).
Personal Identifiable Information (PII) was not collected from study participants and responses will be
utilized by CDC PRAMS staff only and not disseminated externally for any reason.
Potential participants were provided with: (1) a letter describing the purpose of the study including a brief
description of the Pregnancy Risk Assessment Monitoring System (PRAMS) (2) an informed consent that
4

specifies voluntary participation, risk/benefits (i.e., no effect on her appointment or services), confidentiality,
and (3) incentive information.
After consenting, participants were provided with one of the questionnaire versions. “Phone” surveys were
read to participants, while “Mail” surveys were printed and given to participants to be filled out on their own,
answering at their pace. All interviews were conducted with participants while they waited for their
appointment, in the provider’s office.
CRC interviewers reviewed the participant’s responses with them, utilizing the probes provided by CDC,
immediately following the completion of the survey. The follow up or debriefing portion of the testing
allowed the interviewer to compare information given narratively with related answers to survey questions
and to document any contradictory information given by the participant. This information will demonstrate
points of confusion and/or misunderstanding. Probes were careful outlined in the interview instructions by
CSTE/CDC. Probes were useful for obtaining additional pre-identified areas of concern for questions in the
supplement. Capturing the narratives of the respondents, provides an opportunity to explore unanticipated
issues respondents may have encountered while completing the survey. This quantitative data has proven to
be extremely instrumental when analyzing how respondents interpret the intent of the question and meaning
of terms and phrases utilized.
Each respondent received $100 Visa Gift card for their participation. Pediatric clinics received $200 for each
woman from their practice that participated.
At the conclusion of the cognitive testing phase, the results were submitted to CSTE/CDC. Following the
review and analysis of the cognitive test results, adjustments to the testing instruments were made, as deemed
necessary by CSTE/CDC. The updated instruments were used to conduct the next phase (field testing). The
same procedures/methodology detailed above were used to conduct this second round of testing which
included 20 questions. Once again, interviewers debriefed with respondents immediately after they
completed the survey. The questions/probes below were developed by CSTE/CDC and used throughout the
field testing process (See Appendix C, SDoH Cognitive Testing Instruments).
Field Testing Debriefing Questions - English
1. Were the questions hard or easy to understand? What made the question(s) hard to understand? Was it
the skips, reading level was too high, language problems, etc.?
2. Did you have any trouble choosing between the answer choices: Have her explain or give examples.
3. Were the instruction clear: If not, have her show you an example of poorly worded instructions. How
would she word it differently?
4. Do you have any other comments or suggestions for improving the survey?
Field Testing Debriefing Questions - Spanish
1. Fueron las preguntas dificiles o faciles de entender? Que hizo que las (s) pregunta (s) sea dificil de
entender? Fu eel salto, el nivel de lectura fue demasiado alto, problemas de lenguaje, etc.?
2. Tuviste algun problema para elegir entre las opciones de reapuesta? Haga que explique o de ejemplos.
3. Fueron claras las instrucciones? Si no, pidale que le muestre un ejemplo de instrucciones mal redactadas.
Como lo expresaria de manaera diferente?
4. Tiene algun otro comentario o sugerencia para mejorar la encuesta?
The following tables include demographic characteristics of the respondents for cognitive and field testing
(See Tables 1, 1a, 2 and 2a). A summary of the results of the cognitive and field testing can be found in the
“Results” section starting on page 8.

5

Tables 1 – 2a: Demographic Summary of Respondents for each round of testing
Cognitive Testing Demographics:
Nine women participated in the cognitive testing phase. Most of the participants, 62.5%, were under the age
of 30. When reporting education levels, 62.5% reported their education as Less than High School, 25%
completed High School and 12.5% reported Greater than High School. Over half (56%) of the participants
reported their ethnicity as Hispanic, with 33% identifying as Non-Hispanic White and 11% as Non-Hispanic
Black. Fifty-seven percent reported their income as between “25k and 50k, 28.6% reported their income as
“Less than $25k” and 14.3% reported earning an income of $85k+. Over half (56%) of the respondents
reported English as their native language, while 44% reported Spanish. Seventy-five percent of the
participants reported their youngest child was between 0-6 months, while 25% reported 7–12 months.
NOTE: This summary excludes missing data for participants who elected not to answer their age, education,
income, or age of youngest child.
Table 1. Demographic Summary of Cognitive Testing Participants
% (n)
N=8
62.5 (5)
37.5 (3)
0 (0)

Age (years)
Under 30
30-39 years
40 and over
Education
Less than High School
High School
Greater than High School

N=8
62.5 (5)
25.0 (2)
12.5 (1)

Race/Ethnicity
NH-White
NH-Black
Hispanic
Other

N=9
33.0 (3)
11.0 (1)
56.6 (5)
0 (0)

Income
Less than $25,000
$25,001 - $50,000
$50,000 - $85,000
$85,000+

N=7
28.6 (2)
57.1 (4)
0 (0)
14.3 (1)

Language
English
Spanish

N=9
56.0 (5)
44.0 (4)

Age of youngest child
<2
2-6
7-12 months

N=8
37.5 (3)
37.5 (3)
25 (2)

Table 1a. Cognitive Testing Respondents - Case-by-Case
Subject ID
1945
21133
3243
4422
1900
21117
1
2
3

Race/Ethnicity
Hispanic
Hispanic
White
Black
Hispanic
Hispanic
Hispanic
White
White

Language
Spanish
Spanish
English
English
Spanish
Spanish
English
English
English

Age
24
22
30
22
32
34
25
28

Education

Income

Less than High School
Less than High School
Less than High School
Less than High School
High School
Greater than High School
High School
Less than High School

$25 - $50
Less than $25
Less than $25
$25 - $50
$25 - $50
$85+
$25 - $50
-

Age of
youngest child

4 months
1 month
1 month
3 months
1 month
4 months
11 months
9 months

Survey
Spanish Mail
Spanish Phone
English Mail
English Phone
Spanish Phone
Spanish Mail
English Phone
English Phone
Spanish Mail

6

Field testing Demographics:
Ten women participated in the field testing phase. Most of the participants, 60%, were between ages of 30
and 34. When reporting education levels, “Less than High School” and “Greater Than High School” were at
40% with 20% reporting they completed High School. Half (50%) of the participants reported their ethnicity
as Hispanic, with 30% identifying as Non-Hispanic White and 20% as Other. Sixty percent of the
participants reported their income as “less than $25k”, 30% reported earning an income of $85k+ and 10%
reported their income range as between $25k and $50k. Language spoken by respondents was reported
equally, with 50% reporting English as their native language and 50% reporting Spanish. Seventy percent of
the participants have children ages 0-6 months, while 30% have children between 7–12 months.
Table 2. Demographic Summary of Field Testing Respondents
% (N)
N=10
20.0 (2)
60.0 (6)
20.0 (2)

Age (years)
Under 30
30 -35 years
36 - 40
Education
Less than High School
High School
Greater than High School

N=10
40.0 (4)
20.0 (2)
40.0 (4)

Race/Ethnicity
NH-White
NH-Black
Hispanic
Other

N=10
30.0 (3)
0 (0)
50.0 (5)
20.0 (2)

Income
Less than $25,000
$25,001 - $50,000
$50,000 - $85,000
$85,000+

N=10
60.0 (6)
10.0 (1)
0 (0)
30.0 (3)

Language
English
Spanish

N=10
50.0 (5)
50.0 (5)

Age of youngest child
<2
2-6 months
7-12 months

N=10
20.0 (2)
50.0 (5)
30.0 (3)

Table 2a. Field Testing Respondents - Case-by-Case
Subject ID
Race/Ethnicity Language Age
OF-HL-001
OF-HL-002
OF-HL-003
OF-HL-004
OF-HL-005
BB-AK-001
BB-AK-002
BB-AK-003
BB-AK-004
BB-AK-005

Black/White
White
White
Other
White
Hispanic
Hispanic
Hispanic
Hispanic
Hispanic

English
English
English
English
English
Spanish
Spanish
Spanish
Spanish
Spanish

22
37
38
31
35
30
32
35
30
24

Education

Income

High School
Greater than High School
Greater than High School
Greater than High School
Greater than High School
Less than High School
Less than High School
Less than High School
High School
Less than High School

Less than $25
$85+
$85+
Less than $25
$85+
Less than $25
Less than $25
Less than $25
$25 - $50
Less than $25

Age of
youngest child

2 weeks
12 months
10 months
1 month
2 months
9 months
6 months
5 months
6 months
6 months

Survey
English Mail
English Mail
English Mail
English Phone
English Mail
Spanish Phone
Spanish Mail
Spanish Phone
Spanish Mail
Spanish Mail

7

Results
Cognitive Testing Question-by-Question Analysis
1.

What is your living situation today? Check ONE answer
I have a steady place to live
I have a place to live today, but I am worried about losing it in the future
I do not have a steady place to live (I am temporarily staying with others, in a hotel, in a shelter,
living outside on the street, on a beach, in a car, abandoned building, bus, or train station, or in a
park)
¿Cuál es su situación de vivienda hoy? Marque UNA respuesta
Tengo un lugar estable para vivir
Tengo un lugar para vivir hoy, pero me preocupa perderlo en el futuro
No tengo un lugar estable para vivir (me estoy quedando temporeramente con otras personas, en
un hotel, en un refugio, viviendo afuera en la calle, en una playa, en un automóvil, en un edificio
abandonado, en una estación de autobús o de tren, o en un parque)

Discussion:
This question was designed to capture housing instability. This question did not elicit suggestions or
recommendations from respondents. There were no reports of difficulty answering the question or
expressions of confusion or misunderstandings. A Spanish Mail respondent highlighted a spelling error and
recommended the word “temporeramente” be changed to “temporalmente”.
Recommendation:
• Confirm spelling of text on the Spanish Mail and Phone Surveys, paying specific attention to the word
“temporeramente”.
Question Modification: None
________________________________________________________________________________________________________________________

2.

Think about the place you live. Do you have problems with any of the following?
Check ALL that apply
Pests such as bugs, ants, or mice
Mold
Lead paint or pipes
Lack of heat
Oven or stove not working
Smoke detectors missing or not working
Water leaks
None of the above
Piense en el lugar donde vive. ¿Tiene problemas con alguno de los siguientes?
Marque TODAS las que correspondan
Plagas como insectos, hormigas o ratones
Moho
Plomo en la pintura o tuberías
Falta de calefacción
El horno o la estufa no funcionan
Faltan detectores de humo o no funcionan
Goteos de agua
Ninguno de los anteriores
8

Discussion:
Interviewers were instructed to probe phone respondents that answered no to all of the options or if they
experienced problems that were not listed. When respondents were asked if the list was complete, the
majority confirmed the list was complete and did not recommend additions be made. One respondent
recommended “plumbing” be added to the list. Another recommended capturing additional responses by
including an “Other: Please explain, fill-in” option.
Respondents also provided feedback on the question format/layout. They were specifically asked if a yes/no
format would make the question easier to answer. Two respondents reported they preferred a yes/no,
table/grid format, instead of the current “Check all that apply” format. One added that the formatting update
would encourage participants to read each option thoroughly and discourage “scanning”.
Some of the feedback provided by respondents on this question illustrated the potential for data error due to
confusion regarding question intent and vague/unclear text. One respondent reported that she found the
text/phrasing confusing. She reported having a physical intolerance with mold and she was unsure if she was
to report personal issues or the status of her home while considering the options.
Spanish Mail/Phone: Several Spanish respondents shared insight on use of specific terms and meanings. In
general, the translation worked well. However, some respondents experienced difficulty with a few words
and phrase. During the administration of the Spanish Phone, one respondent had difficulty understanding the
term “moho”. After being given the example of “Black mold/mildew”, the respondent was able to
understand. Another respondent expressed some confusion with the same term while completing the Spanish
Mail survey. She recommended the term include the additional descriptor “negro” (black) to assist with
understanding. This respondent also expressed confusion with the term “plomo en la pintura o tuberias” and
“calefaccion” She noted the term “calentador” is the term she uses for heat (calefaccion) and recommended
the term be updated to, “falta de un calentador/calentador electrico”.
Another Spanish Phone respondent noted that the response option “2e, el horno o la estufa no funcionan”
(oven or stove not working) was confusing. She initially thought that checking “yes” would indicate that her
oven/stove was in working condition and not that she was experiencing issues.
Recommendation:
• Review the terms used in the Spanish Mail and Phone, paying close attention to the noted areas of
confusion (moho, plomo en la pintura o tuberias” and “calefaccion”)
• Consider adding probes or additional descriptors providing respondents with instructions to answer the
question for issues their having in the place they are living
• Consider using yes/no format instead of “Check ALL that apply”
• Consider adding “in the place you live” to the end of the last sentence, to emphasize the intent of the
question
Question Modification:
• Question layout changed from ‘Check all that apply”, to grid format with yes/no response option.
• Additional description (like black mold) added after the option b “Mold”
________________________________________________________________________________________________________________________

3.

During the last 12 months, was there a time when you were not able to pay your mortgage, rent,
or utility bills?
No
Yes

Go to Question 5

9

En los últimos 12 meses, ¿hubo algún momento en el que no pudo pagar su hipoteca, alquiler o
facturas de servicios?
No
Sí

Pase a la Pregunta 5

Discussion:
Respondents were asked about the need to include examples of “utility bills” Overall respondents reported
understanding the question. Respondents followed skip instruction without difficulty.
Spanish Translation: Respondents were asked to provide specific feedback on the translated term for “utility
bills”, since there is no equivalent in the Spanish language. One Spanish Mail respondent reported not
understanding the meaning of “hipoteca’ or “alquiler”. She added that she uses the term “la renta”. Another
Spanish respondent reported that she did not understand meaning of utility bills translated in Spanish to
facturas de servicios to mean “light bill, water bill, etc”. Respondent noted she refers to these as the bill or
“el bill” in Spanish “el bill de agua, bill de luz”.
Recommendation:
• Consider updating terms used in Spanish question to ensure appropriateness of meaning
• Research Spanish text to ensure the appropriate standardized terms have been incorporated
Question Modification: None
_______________________________________________________________________________________
4.

In the last 12 months, have you or anyone in your household applied for emergency rental
assistance through your state or local government to cover your unpaid rent or utility bills?
Applied and received assistance
Applied and waiting for a response
Applied, but the application was denied
Did not apply
En los últimos 12 meses, ¿usted, u otra persona en su hogar, ha solicitado asistencia de
emergencia para alquiler a través de su gobierno estatal o local para cubrir el alquiler no
pagado o facturas de servicios no pagados?
Se solicitó y se recibió la asistencia
Se solicitó y en espera de una respuesta
Se solicitó, pero la aplicación fue denegada
No fue solicitada

Discussion:
The question was designed to measure the prevalence of seeking government assistance with housing.
Respondents skip this question if they answer “no” to the question above (Q#3). This question did not
produce substantive feedback. Respondents answered the questions without expressing confusion or
misunderstandings. When asked, respondents reported the options listed were adequate and no additions
were necessary.
Recommendation:
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. “In the
last 12 months” was updated to “During the last 12 months”
_______________________________________________________________________________________

10

5.

During the last 12 months, was there a time when an electric, gas, oil, or water company
threatened to shut off services?
No
Yes
Durante los últimos 12 meses, ¿hubo algún momento en que una compañía de electricidad, gas,
petróleo o agua amenazó con cortar los servicios?
No
Si

Discussion:
This question was designed to capture financial stability. Testing focus of this question included the use of
“threaten to” or other types of notice. Overall, the majority of the respondents report understanding the intent
of the questions. Several noted that the term “threatened” was strong and recommended a new term be
identified. Suggestions included “notified/notifico”, “advised/aviso” and “warning. A Spanish respondent
suggested “con cortar” be dropped, leaving “amenazo” as a standalone term. Another Spanish respondent
noted the inclusion of examples of utility bills was not necessary, adding that “utility bills” alone was
sufficient to understand meaning of electric, water, gas bills.
Recommendation:
• Consider replacing the term “threatened” with one of the suggestions offered by respondents or another
alternative
Question Modification:
• The word “threatened” was removed and “issued a notice or warning that they were going” was added.
________________________________________________________________________________________________________________________

6.

Thinking about the last 12 months, when you are at home, how much does each of the following
bother, disturb, or annoy you?
For each item, check:
NA if not at all
M if moderately, or
E if extremely
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
k.

Noise from cars, trucks, trains, or other traffic
Smells or dirt from road traffic
Smoke, gas, or bad smells from anything else
Litter or poorly kept up housing
Noise from aircraft
My neighbors’ noise or other activities
Undesirable business, institutional, or industrial property
A lack of parks or green spaces
Inadequate public transportation
The amount of neighborhood crime
Poor city or county services

Piense en los últimos 12 meses. Cuando usted está en su hogar, ¿qué tanto le molesta, perturba o
fastidia lo siguiente?
Para cada una, marque:
PN, si para nada
M, si moderamente o
E, si extremadamente

11

a. Ruido de automóviles, camiones, trenes, u otro tráfico
b. Olores o sucio del tráfico en la carretera
c. Humo, gas o malos olores de otra cosa
d. Basura o viviendas en mal estado
e. Ruido de aviones
f. El ruido u otras actividades que hacen mis vecinos
g. Negocios o propiedades institucionales o industriales indeseables
h. Falta de parques o zonas verdes
i. Transporte público inadecuado
j. La cantidad de crimen o delitos en el vecindario
k. Malos servicios de la ciudad o del condado
Discussion:
The intent of this question is to capture living environment. Interviewers were instructed to probe
respondents to ensure their understanding of the response options, specifically “poorly kept up housing” and
“undesirable business, institutional, or industrial property”. Respondents were also asked to provide feedback
on options that may be missing. The majority of the respondents reported feeling like the list was complete
and did recommend additional response options, however, one suggested “lack of adequate street lighting”
be added. Instead of selecting “PN”, one respondent left several answers blank, to indicate the options were
not a problem. She reported being confused by the answer choices and suggested the format be changed to
yes/no options. Similar suggestions were made by other respondents.
The majority of respondents demonstrated clear understanding of the terms “poorly kept up housing” and
“undesirable business, institutional, or industrial property”, however a few had difficulty providing examples
when asked about undesirable businesses. These respondents benefited from the probes and additional
information provided by the interviewers. This may indicate the need to consider revising the language of
this response option (option g). Another respondent had difficulty understanding the term “inadequate”
(option i). This respondent shared that because she had not finished high school, she has difficulty
understanding specific words or terms. This comment may indicate the reading level for this option needs to
be adjusted.
Spanish Translation: Several respondents noted misunderstandings related to the translation of terms/words
within the question. One respondent noted that “viviendas en mal estado” translated as houses in poor
condition. She also noted that text used in option g translated as businesses that sell/promote products or
services, including illegal products or services (i.e., nightclubs). One respondent noted that text used in
option i, translated as public transportation that is broken down or unclean, with unkind employees that are
rude.
Recommendation:
• Consider deleting or rewording option “g”
• Revise description of timeframe to maintain standardized language throughout survey
• Consider replacing the word “inadequate” with the term “lack of” (option i)
• Review Spanish translation in options “g” and “i” to ensure accuracy
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey
“Thinking about” was removed and replaced with “During” the last 12 months
• Option “g”, “Undesirable business, institutional, or industrial property” was removed
________________________________________________________________________________________________________________________

12

7.

How safe from crime do you consider your neighborhood to be? Would you say…
Extremely safe
Safe
Unsafe
Extremely unsafe
¿Qué tan seguro considera que es su vecindario con respecto a la delincuencia? Usted diría que
es...
Muy seguro
Seguro
Inseguro
Extremadamente inseguro

Discussion:
This question was designed as an alternative to the above question regarding living environment. The intent
is to gauge perceived neighborhood safety. Interviewers were instructed to probe respondents about their
definition of crime. One respondent reported considering her sense of safety in and immediately surrounding
her home. She added that her answer would not be “not safe” when considering several streets away from her
home. Respondents may consider various ideas about the size and parameters when considering their
neighborhoods. Including additional descriptors may decrease confusion and provide richer data.
Spanish Translation: A few respondents reported being confused by the question and not certain of the intent.
When asked, several respondents did not have an understanding or definition for crime (Spanish delincuencia or crimen). When probed further, one respondent reported crimes were “robberies and theft”.
Other respondents’ reported the definition of crime as the following: stealing, smoking, breaking into cares
or homes, shootings, delincuencia/violence, murders, and minor car accidents.
Recommendation:
• Consider providing details or guidance on the radius or “neighborhood” and consider adding definition
or examples of crime to decrease confusion.
Question Modification: None
_______________________________________________________________________________________
8.

During the past 12 months, how often did the food that you bought not last, and you didn’t have
money to get more? Was that…
Always
Usually
Sometimes
Rarely
Never
Durante los últimos 12 meses, ¿con qué frecuencia no duraron los alimentos que compró y no
tuvo dinero para comprar más? Ha sido...
Siempre
Usualmente
A veces
Rara vez
Nunca

13

Discussion:
This question was designed to test the use of the 5-point scale and determine if it possible to use a 3-point
scale instead. Interviewers were also asked to probe respondents about their difficulty choosing between
“always” and “usually”.
Several respondents noted a preference for the 3-point scale, stating the decrease in options, and addition of
examples would help participants understand the intent of the question. However, it should be noted that
some respondents expressed their preference for the 5-point scale, indicating the same rationale, having more
options increases understanding.
When respondents were asked about the use of “usually”, interviewers noted varying interpretations. One
respondent interpreted the meaning of usually as “more often” and sometimes as “once in a while/not as
common”. One reported having a clear understanding of the terms and not experiencing difficult
differentiating between the two. Another perceived the choices as too similar and recommended removing
“usually” from the response list and maintaining the other four responses (always, sometimes, rarely, never).
Spanish Translation: Several respondents expressed confusion with the term “usualmente”. One noted that
the use of the term usually (usualmente) is unnecessary and recommended it be removed from the response
options, and suggested maintaining the other 4 responses choices, (siempre, a veces, rara vez, and nunca).
The same respondent also suggested that reducing even further, to a 3-point scale (siempre, a veces, nunca),
would simplify the mental processing required when selecting a response.
Recommendation:
• Consider utilizing 3- or 4-point scale
• Consider removing “usually” from the list of response options
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. The
word “past” was removed and replaced with “last”.
________________________________________________________________________________________________________________________

9.

During the past 12 months, how often were you unable to afford to eat balanced meals? Was
that….
Always
Usually
Sometimes
Rarely
Never
Durante los últimos 12 meses, ¿con qué frecuencia no pudo permitirse comer comidas
balanceadas? Ha sido…
Siempre
Usualmente
A veces
Rara vez
Nunca

Discussion:
The intent of this question is to capture the ability to afford nutritious food and determine the necessity of
including the definition of “balance meal” Several respondents, both Spanish and English, expressed
difficulty understanding the term “balanced meals”. One respondent defined “balanced meals” as the ability
to equally distribute food over a certain amount of time. Another respondent defined the term as meals eaten
14

for weight loss or weight management. Respondents agreed that providing the definition, assisted them with
understanding the word/term.
Several respondents reported limiting the response options to a 3-point scale over the 5- point scale
would decrease the time it takes to process their response and make the question easier to answer. One
respondent reported “always” and “rarely” were unnecessary responses for this question. A Spanish Mail
respondent also reported the term “usualmente” was not necessary and recommended it be removed.
Recommendation:
• Consider providing description/definition of “balanced meals” to the question instructions
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “past” was replaced with “last” (During the last 12 months)
• Minor format change - the word “unable” is underlined
• The NHANES definition of “a balanced meal” was added to the instructions of the question
________________________________________________________________________________________________________________________

10.

In the last 12 months, has lack of reliable transportation kept you from medical appointments,
meetings, work, or from getting things needed for daily living?
No
Yes
En los últimos 12 meses, ¿la falta de transporte confiable le ha impedido acudir a citas médicas,
reuniones, trabajo o conseguir las cosas necesarias para la vida diaria?
No
Sí

Discussion:
This question was designed to capture access and ease of access to transportation. Respondents were asked to
share if they preferred this over Q#11. The overall response to the question was positive. When respondents
were asked if they had a preference, the majority reported both questions were adequate. Several
respondents recommended offering a detailed list of options, stating that it would provide them with the
opportunity to share more details versus offering no/yes options only.
Respondents shared varying interpretations of the question intent. Some reported their interpretation of the
question as transportation that is working order and not in jeopardy of “breaking down”, transportation that
“always gets you where you need to be”, and transportation that guarantees you will arrive on time. While
other respondents’ interpretations focused on personal relationships. For example, respondents reported their
interpretation as the ability to travel without depending on others for transportation to and from
appointments, and always having readily available transportation as needed. If these interpretations do not
match the intent of the question, the answers could be considered cases of response error.
Spanish Translation: Several respondents noted their interpretation of the question to mean whether the
transportation being utilized could be trusted, as in safe and not reliable or easily accessible. This may
indicate an issue with the translation of “reliable transportation” to “transporte confinable”. Respondents
reported that the term, “confinable” is more commonly used to convey trust and sense of safety. Suggested
update: En los últimos 12 meses, la falta de acceso al transporte le ha impedido acudir a citas médicas,
reuniones, trabajo o conseguir las cosas necesarias para la vida diaria?

15

Recommendation:
• Consider providing additional instructions to the questions to increase the clarity of possible
interpretations of the term “reliable”
• Consider adding a probe that include examples of the term “reliable”
• Consider editing the Spanish translation for the term “reliable transportation”
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “In the last” was replaced with “During the last”
________________________________________________________________________________________________________________________

11.

In the last 12 months, has lack of transportation kept you from medical appointments, meetings,
work, or from getting things needed for daily living? Check ALL that apply
No
Yes, it has kept me from medical appointments
Yes, it has kept me from non-medical meetings, appointments, or work
Yes, it has kept me from getting things needed for daily living
En los últimos 12 meses, ¿la falta de transporte le ha impedido acudir a citas médicas, reuniones,
trabajo o conseguir las cosas necesarias para la vida diaria? Marque TODAS las que
correspondan
No
Sí, me ha impedido acudir a las citas médicas
Sí, le ha impedido asistir a reunions o citas que no eran médicas, o trabajo
Sí, me ha impedido asistir a reuniones, citas o trabajos no medicos
Sí, me ha impedido obtener las cosas necesarias para la vida diaria

Discussion:
This question is an alternative to the transportation question above (Q#10). Respondents were asked to share
their preference. Generally, there were no comprehension issues reported with this question. One respondent
noted she interpreted the question to mean whether the transportation being utilized could be trusted as in
“can I trust the person that is driving this taxi?” and not whether she had access to transportation or if it was
easily accessible. This may illustrate possible error and need to provide additional descriptors to ensure
respondents understand the intent of the question.
Another respondent reported the question caused her to think of all public transportation versus forms of
personal transportation. When the interviewer engaged the participant with the probes, the respondent shared
that she had experienced some mechanical issues with her personal vehicle with the past 12 months and this
impacted her ability to get her daughter to school on time.
Spanish Translation: A Spanish respondent reported the translation of “reliable transportation/transporte
confinable” was problematic, sharing that the word “confinable” is more commonly used to convey trust and
a sense of safety. Another respondent suggestion the word “reliable” be removed, like in Q11.
Recommendation:
• Consider including additional descriptors to the question instructions to ensure respondents understand
the intent of the question.
• Consider including a “Please explain” option so respondents can provide additional information.
• Confirm translation of Spanish terms/words, specifically term used for “reliable transportation”
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “In the last” was replaced with “During the last”.
________________________________________________________________________________________________________________________

16

12.

In the last 12 months, have you delayed getting medical care because of the cost?
No
Yes
En los últimos 12 meses, ¿usted ha retrasado obtener atención médica debido al costo?
No
Sí

Discussion:
This question was designed to capture delay in getting medical care for the respondents, not their infants.
Interviewers were asked to probe respondents on if they made the decision NOT to get medical care or
delayed receiving preventive services. A few respondents reported answering the question regarding delaying
medical care for themselves. Several respondents reported not delaying care due to cost, while one
respondent shared they purposefully delayed personal care due to cost and chose to receive care in her native
country. One respondent suggested using “not getting medical care because of cost” instead of “delayed
getting medical care”. Because the intent of the question is to measure the “delay” of medical care, using the
term “put off” may be a stronger alternative.
It should also be noted that interviewers captured lengthy narrative responses. This may indicate the question
needs a “Please explain” option or another way for respondents to share additional information.
In general, this question did not present as problematic. Overall, respondents reported not experiencing
confusion when answering the question. A Spanish respondent expressed confusion with the translation of
the term “attencion medica”. She interpreted this to mean “going to the pharmacy”. She also did not
understand the meaning of the term “ha retrasado obtener”. This may indicate the need to consider
alternative translation of the term(s) to avoid possible data errors.
Recommendation:
• Consider including additional descriptors to the probes (question instructions) to ensure respondents
understand the intent of the question.
• Consider reducing reading level of the word “delay”.
• Consider including a “Please explain” option so respondents can provide additional information.
• Confirm translation of Spanish terms/words, specifically for “attencion medica” and “ha retrasado
obtener”.
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “In the last” was replaced with “During the last”
________________________________________________________________________________________________________________________

17

13.

In the last 12 months, how often did your doctors, nurses, or other health professionals explain
things about your health in a way that was easy to understand?
Never
Sometimes
Usually
Always
En los últimos 12 meses, ¿con qué frecuencia sus médicos, enfermeras y otros profesionales de la
salud le explicaron cosas sobre su salud de una manera fácil de entender?
Nunca
Algunas veces
Usualmente
Siempre

Discussion:
The intent of this question is to assess health literacy/health accessibility (understanding of information
shared with them). The question was also designed to test respondent’s tolerance of the change in the order
of the 4-point scale options. Respondents provided mixed feedback when asked if they noticed the change in
response scale. Some reported not noticing and having no preference, while others noted the change and even
found it “strange” or “confusing”.
Several respondents skipped out of this question, so it wasn’t extensively tested. However, when responders
were questioned about their understanding of the intent, most noted their interpretation as whether or not
their health care professionals explained things adequately during their prenatal visits.
Recommendation:
• Standardize scale options to ensure consistency throughout survey and avoid possible confusion.
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “In the last” was replaced with “During the last”
• Scaled response items were re-ordered to maintain consistency throughout the survey (Always, Usually,
Sometimes, Rarely, Never)
________________________________________________________________________________________________________________________

14.

Since your new baby was born, have you felt that you have needed mental health services such as
counseling, medications, or support groups to help with feelings of anxiety, depression, grief, or
other issues?
No
Yes

Go to Question 17

Desde que nació su nuevo bebé, ¿ha sentido que servicios de salud mental, medicamentos o
grupos de apoyo para ayudarle con sus sentimientos de ansiedad, depresión, dolor u otros
problemas?
No
Sí

Pase a la Pregunta 17

Discussion:
This question was designed to test respondents’ preference for the use of the term “mental health services”
versus “mental health treatment”. When asked their preference for mental health services versus treatment
the majority of the respondents preferred mental health services.
18

Spanish Translation: One respondent noted the translation of mental health services (servicios de salud) and
suggested it be used instead of the “tratamientos de salud mental” because the meanings of both are very
different. The respondent reported that “servicios” means services available to an individual. While the
translation presented on the survey for treatment means an individual who is actively receiving or is under
the care of a provider.
Recommendation:
• Consider using mental health services/servicios de salud instead of mental health treatment/ tratamientos
de salud mental
Question Modification: None
_______________________________________________________________________________________
15.

Were you able to get the mental health services that you needed?
No
Yes

Go to Question 17

¿Ha podido obtener los servicios de salud mental que necesitaba?
No
Sí

Pase a la Pregunta 17

Discussion: See Q#14 (above)
Recommendation: See Q#14 (above)
Question Modification: See Q#14 (above)
________________________________________________________________________________________________________________________

16.

Which of these statements explains why you did not get the mental health treatment or
counseling you needed? Check ALL that apply
I couldn’t afford the cost
My health insurance does not cover any type of mental health treatment or counseling
My health insurance does not pay enough for mental health treatment or counseling
I did not know where to go to get services
I was concerned that the information I gave the counselor might not be kept confidential
I didn’t want others to find out that I needed treatment
I was concerned that I might be committed to a psychiatric hospital or might have to take medicine
I had no transportation, treatment was too far away, or the hours were not convenient
I didn’t have time (because of job, childcare, or other commitments)
Some other reason or reasons
Please tell us: ________________________________
¿Cuál de estas afirmaciones explica por qué no recibió los tratamientos de salud mental o la
consejería que necesitaba? Marque TODAS las que correspondan
No podía pagar lo que costaba
Mi seguro médico no cubre ningún tipo de tratamiento o consejería de salud mental
Mi seguro médico no paga lo suficiente por el tratamiento o la consejería de salud mental
No sabía a donde ir para obtener servicios
Me preocupaba que la información que le daría al consejero no se mantuviera confidencial
No quería que otros se enteraran de que necesitaba tratamiento
Me preocupaba que me internaran en un hospital psiquiátrico o que tuviera que tomar
medicamentos
No tenía transporte, el tratamiento estaba demasiado lejos o el horario no era conveniente
No tenía tiempo (por el trabajo, cuidado de niños u otros compromisos)
Otra razón o razones Por favor, escríbala:
19

Discussion:
The intent of this question is to measure why respondents did not receive the mental health treatment they
needed. Several respondents skipped out of this question after answering “no” to questions 14 or 16. Of the
respondents that did answer the question, several suggested the response option detailing the psychiatric
hospital and medicine, be split into two separate responses. As with questions 14 and 15, respondents
reported they preferred “mental health services” over “mental health treatment”.
Recommendation:
• Consider following respondents’ suggestion to separate the option detailing concern about being
committed to a hospital or taking medicine. This will avoid potential data error and allow the two topics
to be tracked and documented separately
Question Modification:
• Option “g”, “… concerned that you might be committed to a psychiatric hospital or might have to take
medicine” was separated into two options (“…concerned that you might be committed to a psychiatric
hospital” and “…concerned that you might have to take medicine”)
________________________________________________________________________________________________________________________

17.

During your life, how often have you felt that you were treated badly or unfairly because of your
race or ethnicity?
Never
Rarely
Sometimes
Most of the time
Always
Durante tu vida, ¿con qué frecuencia has sentido que te trataron mal o injustamente por tu raza
u origen étnico?
Nunca
Rara vez
Algunas veces
La mayor parte del tiempo
Siempre

Discussion:
This question was designed to assess the prevalence of racial/ethnic discrimination. Interviewers were asked
to probe respondents about their understanding of the terms “treated badly” or “treated unfairly”. The
majority of the respondents report understanding and relating to racism. Respondents also reported their
interpretation of “unfairly/badly” to mean being treated negatively because of your race/color of skin, racial
discrimination, being mistreated by someone who is racist, experiencing rude comments, being dismissed
and not being provided the same opportunities as others.
Recommendation:
• Standardize scale options to ensure consistency throughout survey and avoid possible confusion
Question Modification:
• Scaled response items were re-ordered to maintain consistency throughout the survey (Always, Usually,
Sometimes, Rarely, Never)
• Replaced “Most of the time” with “Usually”
_______________________________________________________________________________________

20

18.

During your life, how often have you seen your parents or other family members treated badly
or unfairly because of the color of their skin, language, accent, or because they are from a
different country or culture?
Never
Rarely
Sometimes
Most of the time
Always
A lo largo de tu vida, ¿con qué frecuencia has visto a tus padres, a otros familiares tratados mal
o injustamente por el color de su piel, idioma, acento o porque son de otro país o cultura?
Nunca
Rara vez
Algunas veces
La mayor parte del tiempo
Siempre

Discussion:
The question was designed to assess the prevalence of pervasive discrimination in their life and whether it
extends beyond race to language, culture, etc. One respondent became visibly uneasy after reading the
question. The interviewer reports that the respondent’s answer was not consistent with her body language.
Other respondents reported understanding the intent of the question was to measure thoughts related to
racism and discrimination. When respondents were asked for their definition of “unfairly/badly” their
responses were similar to what was shared above (Q#17). Additional responses included: being treated
poorly and disrespectfully, name calling, and body sharing. Although one respondent selected “never” when
asked if she had been treated badly, she shared how individuals have talked about her appearance and called
her names. She recommended the question be changed to read, “During your life, do you recall being treated
badly or unfairly because of your race or ethnicity?”. She felt the change would allow respondents to think
about specific incidents they have experienced.
Recommendation:
• Standardize scale options to ensure consistency throughout survey and avoid possible confusion
• Consider offering a “Please explain” option to capture quantitative data on women’s specific experiences
Question Modification:
• Scaled response items were re-ordered to maintain consistency throughout the survey (Always, Usually,
Sometimes, Rarely, Never).
________________________________________________________________________________________________________________________

19.

During the last 12 months, how often would you say you get the social and emotional support you
need?
Never
Rarely
Sometimes
Most of the time
Always

21

¿Con qué frecuencia obtiene el apoyo social y emocional que necesita?
Nunca
Rara vez
Algunas veces
La mayor parte del tiempo
Siempre
Discussion:
The intent of this question is to assess the prevalence of the respondents who are receiving emotional and
social support as they need it. Interviewers were instructed to probe about what time period respondents were
focusing on when answering the question and to ask them for their definition of social and emotional
support. When respondents were asked about the time period used when responding to the questions,
responses included: beginning of her pregnancy to present day, entire life, recent, within the last 5 years, her
6th month of pregnancy, and since giving birth to her child. One respondent reported using “random times
and within the last year” as the time period used when considering the options. She added that her choice was
based on the timeframe pattern she noted throughout the survey.
When respondents were asked to provide their definition or interpretation of social/emotional support,
answers included: support from family and/or her partner, services from others or her doctor’s office, group
meetings, socializing with, or helping others, receiving help during difficult moments. One respondent
became tearful when recalling the difficulties of being pregnant without the support of her partner and
potentially having to explain to her child about the father's absence.
Recommendation:
• Standardize order of scale options to ensure consistency throughout survey and avoid possible confusion.
• Consider including specific timeframe for respondents to use when answer the question
• Revise description of timeframe to maintain standardized language throughout survey
Question Modification:
• Language describing the timeframe was updated to maintain consistency throughout the survey. In this
question, “In the last” was replaced with “During the last”
• Scaled response items were re-ordered to maintain consistency throughout the survey (Always, Usually,
Sometimes, Rarely, Never)
________________________________________________________________________________________________________________________

20.

Stress means a situation in which a person feels tense, restless, nervous, or anxious, or is unable
to sleep at night because their mind is troubled all the time.
Within the last 30 days, how often have you felt this kind of stress?
Never
Rarely
Sometimes
Most of the time
Always
El estrés es una situación en donde una persona se siente tensa, inquieta, nerviosa o ansiosa, o no
puede dormir por la noche porque su mente está perturbada todo el tiempo.
En los últimos 30 días, ¿con qu frecuencia ha sentido este tipo de estrés?
Nunca
Rara vez
Algunas veces
La mayor parte del tiempo
Siempre
22

Discussion:
The intent of this question is to measure how often respondents are experiencing stress that is impacting their
quality of life. Interviewers were asked to probe respondents about if their stress is acute or chronic. When
respondents were probed about the status of their stress their responses were mix. Some reported answers
about their chronic stress, while others reported their stress as acute. Examples of acute stress provided by
respondents included: a recent and unexpected job loss, financial and family responsibilities, and caring for a
newborn. Respondents demonstrated a clear understanding of the difference between acute and chronic
stress.
Recommendation:
• Standardize scale options to ensure consistency throughout survey and avoid possible confusion.
Question Modification:
• Scaled response items were re-ordered to maintain consistency throughout the survey (Always, Usually,
Sometimes, Rarely, Never)

23

Field Testing Debriefing Question-by-Question Analysis
1. Were the questions hard or easy to understand? What made the question(s) hard to understandwas it the skips, reading level was too high, language problems, etc.?
Fueron las preguntas dificiles o faciles de entender? Que hizo que las (s) pregunta (s) sea dificil de
entender? Fu eel salto, el nivel de lectura fue demasiado alto, problemas de lenguaje, etc.?
Discussion:
When respondents were asked to share their thoughts on the difficulty level of the questions, the majority
reported the questions were easy to understand. Respondents reported the questions were well developed
and included topics that were relevant to their own or other people's personal experiences.
During the testing phases, most did not struggle with interpreting the information. Respondents that did
express confusion, reported finding the examples and additional definitions helpful.
While the majority reported little to no confusion, there were some that provided feedback for specific
questions that were in need of improvement:
• Q#6 (option h): Confusion with the intent of option h. The respondent reported the option did not
address transportation support from others.
• Q#6 (option d): The respondent had difficulty deciding whether the question was asking if she
was bothered by trash inside or outside of her home or in her neighborhood.
• Q#7: The respondent reported this question was difficult to answer because she was not aware of
crime levels in her neighborhood and did not watch the local news.
• Q#14: The respondent reported having difficulty understanding the question and suggested
removing the term “felt” and replacing it with “seeking”.
• Q#19. The respondent interpretated the question as asking how often she went somewhere to
obtain social and emotional support; rather than how often did she receive social and emotional
support from others.
• Q#20: The respondent reported she felt this question was difficult but did not provide examples
to support her opinion.
When interviewers asked respondents that reported experiencing difficulty to describe the areas of
confusion, some reported difficulty understanding the meaning of terms that they had never heard before.
They also expressed difficulty understanding the intent of the question.
Several Spanish respondents needed assistance with translation of specific terms throughout the survey.
These instances were documented in the Cognitive Testing Question-by-Question Analysis section.
Spanish respondents offered specific questions that were difficult:
• Q#6: Respondent did not understand the term “transporte public pobre”
• Q#6 (option j): Respondent did not understand the meaning of the term “servicios malos de la
ciudad o del condado”. The interviewer utilized the probes and additional definitions to assist the
respondent with understanding. The respondent was able to comprehend after hearing the types
of city services and the meaning of these services.
Recommendation:
• Review instructions and response options for questions #6, #7, #14, #19 and #20, to ensure terms and
intent are clearly stated
• Consider revising the Spanish translation to correct the identified grammatical error.
Correct translation: Durante los últimos 12 meses, con qué frecuencia diría usted que obtuvo el
apoyo emocional y social que necesitaba
_____________________________________________________________________________________

24

2. Did you have any trouble choosing between the answer choices: Have her explain or give examples.
Tuviste algun problema para elegir entre las opciones de reapuesta? Haga que explique o de
ejemplos.
Discussion:
When respondents were asked if they had trouble selecting from the available response options some
reported that they did not experience difficulty, and the options were easy to interpret, and meanings
were clear and easy to follow. However, several respondents provided the following examples of
questions that were problematic:
• Q#6: The respondent found the response options (moderadamente and extremadamente) in this
question difficult to choose from. She also expressed preference for Yes/No answer choice as
opposed to other options offered throughout the survey. For example, Not At All (Para Nada),
Sometimes (A veces), Always (Siempre).
• Q#14: The respondent reported the response options were not adequate and the yes/no response did
not fit well with the options. The respondent suggested the word “seeking” as an alternative term to
be used (Since your new baby was born, are you seeking mental health services).
• Q#20: The respondent shared that the question’s intent was confusing. She was not certain if while
answering she should consider family stress, financial stress or the stress of being a mother.
Recommendation:
• Review instructions and response options for questions #6, #14, and #20, to ensure terms and intent
are clearly stated
________________________________________________________________________________________________________________________

3. Were the instruction clear: If not, have her show you an example of poorly worded instructions.
How would she word it differently?
Fueron claras las instrucciones? Si no, pidale que le muestre un ejemplo de instrucciones mal
redactadas. Como lo expresaria de manaera diferente?
Discussion:
When the respondents were asked if the question instructions were clear, the majority
reported the instructions were clear and easy to understand and they did not have suggestions for
improvements. One respondent providing the following feedback:
● Q#14: Respondent misinterpreted the instructions, selected “no” and moved directly Q#17.
Recommendation:
• Review instructions for question #14, to ensure terms and intent are clearly stated
________________________________________________________________________________________________________________________

4. Do you have any other comments or suggestions for improving the survey?
Tiene algun otro comentario o sugerencia para mejorar la encuesta?
Discussion:
Most of the respondents did not share additional comments or suggestions for improving the survey.
Most stated that the questionnaire was clear and well developed. However, one respondent shared that
the survey could be improved by providing women with an option to answer the questions in a digital
format (phone/tablet/computer). Another respondent shared that she experienced confusion with two
questions (Q#3 and Q#14). The respondents left both questions blank. After the interviewer provided
additional definitions/probes, she answered “no” to both questions. Initially she interpreted the
instructions to mean she could leave the answers blank. She made no additional comments or
suggestions about the survey.
Recommendation:
• Review instructions for question #14, to ensure terms and intent are clearly stated
25

Summary Results by Testing Emphasis Topics
Timeframe
CSTE/CDC was interested in examining potential issues surrounding the use of alternating the timeframe
phrasing (During the past 12 months, During the last 12 months, and In the last 12 months) in seven
questions within the supplement. Participants were asked how they felt about the changes and asked if they
found it confusing. Overall the majority of the participants shared they did not experience confusion with the
alternating timeframes. Some respondents reported they did not notice the changes and simply focused on the
term “12 months”, ignoring other descriptors. Several respondents suggested utilizing the same terminology
to reduce any possible confusion. After analyzing the feedback from the cognitive testing phrase,
CDC/CSTE decided to standardize timeframe phrasing, using only “During the last 12 months” throughout
field testing survey instruments. Field testing respondents did not report issues or difficulty understanding the
timeframe phrasing.
Order of the Scaled Options
CSTE/CDC sought to assess the use of Likert-scale options present in seven questions. These scaled
responses included either 3 or 5-point options. Cognitive testing instruments alternated the order of these
scaled options. Participants were asked to share their opinion on the use of the different patterns. Overall
respondents expressed confusion regarding the alternating order of the scaled responses. Respondent’s
comments
• Respondent noticed alternating pattern and shared that the changes caused her to consider whether
there was a preferred answer (did the scale start with “never” if it was a bad situation and “always” if
it the topic involved a positive situation).
CSTE/CDC modified the scale options to appear in the same order: Always, Usually, Sometimes, Rarely,
Never, throughout the field testing survey instruments.
5-point vs. 3-point Likert-scale Options
CSTE/CDC sought to analyze the efficacy of the use of the options, the best way to present these options and
the necessity to use consistent size (3 or 5-point) throughout the survey.
While respondents’ feedback was mixed, the majority of respondents reported noticing the differences and
expressed clear preference for the simpler, 3-point scale option. Respondent’s comments:
• Reduce question responses to a 4-point scale, keeping always, sometimes, rarely, and never
(siempre, a veces, rara vez, and nunca) or reducing to a 3-point scale. keeping always, sometimes,
never in Spanish (siempre, a veces, nunca).
• Preference: 3-point scale options reduce mental processing needed to answer the question.
• Preference: Condense the options to a 3-point scale and add detailed examples to assist with the
understanding of the question intent.
• Preference: 5-point scale - more options make it easier to understand and answer the questions.
• 3-point scale is sufficient, it is not necessary to use the 5-point scale to answer the questions.
• The change was not noticed and reducing to 3-point scale was not needed or preferred.
• The change was noticed, and it seemed “strange”. Caused respondent to double check previous
answers.
• Preference: 3-point scale is easier to understand and more direct.
Testing results provided insight on respondent’s ability to decide between “sometimes” and “usually” when
presented with these options on a 3 or 5-point scale. When respondents were asked their opinion on the use
of the terms “usually” and “sometimes” the feedback was mixed. Some respondents did not experience
difficulty while others reported that both terms should not be presented as options because the meanings
were too similar. Respondent’s comments:
• “Always” and “rarely” are not necessary to provide a clear response to the question
• Including fewer options, such as usually, sometimes, and never is more easily understood
• “Never”, “sometimes” and “always” are sufficient
26

Appendices

27

Appendix A - SDoH Cognitive Testing Instructions

28

Appendix B - Cognitive and Field Testing Pre-screening Questionnaire

29

Appendix C - SDoH Cognitive Testing Instruments November 15, 2021

30

Appendix D - CareHalo Research Center (CRC) Cognitive Testing Consolidated
Brief

31

Appendix E - Field Testing Debrief Instructions

32

Appendix F - Field Testing Questionnaire

33

Appendix G - SDoH Field Testing Instruments, December 17, 2021

34

Appendix H - CareHalo Research Center (CRC) Field Testing Consolidated
Brief

35