2010, 2011 & 2012 Forms

American Community Survey Methods Panel Tests

Attachment H 2010 ACS Content Test Cognitive Interviewing Final Report

2010, 2011 & 2012 Forms

OMB: 0607-0936

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0607-0936 can be found here:

Document [pdf]

Download: pdf | pdf

August 12, 2009
Contract No. 50-YABC-2-66053 TO 13
RTI Project Number 0209182.013
Deliverable 19

Cognitive Testing of the American
Community Survey Content Test Items
Final Report
Submitted to:
U.S. Census Bureau
Department of Commerce
Washington, DC 20233

Prepared by:
Marjorie Hinsdale, RTI International
Emily McFarlane, RTI International
Stacey Weger, RTI International
Alisú Schoua-Glusberg, Research Support Services
Jeffrey Kerwin, Westat

Submitted by:
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709-2194

Table of Contents

Table of Contents
Executive Summary .......................................................................................................................... ES-1
Methodology and Respondent Characteristics ............................................................................. ES-2
Recommendations by Module ..................................................................................................... ES-3
Conclusion................................................................................................................................ ES-25
1. Introduction ...................................................................................................................................... 1
1.1 Items Tested in Cognitive Interviews ...................................................................................... 2
1.2 Schedule................................................................................................................................. 2
2. Methodology..................................................................................................................................... 3
2.1 Initial Planning for Staffing and Interviewing ......................................................................... 3
2.2 Development of Cognitive Interview Protocols and Forms...................................................... 4
2.3 Translation Methodology for the Cognitive Interview Protocols.............................................. 5
2.4 Cognitive Interview Training .................................................................................................. 5
2.5 Conducting the Cognitive Interviews ...................................................................................... 6
2.6 Reporting Results ................................................................................................................... 7
3. Recruiting ......................................................................................................................................... 8
3.1 Recruiting Targets .................................................................................................................. 8
3.2 Recruiting of English-speaking Respondents........................................................................... 9
3.3 Recruiting of Spanish-speaking Respondents ........................................................................ 12
3.4 Recruitment Outcomes ......................................................................................................... 15
4. Computer and Internet: Summary of Findings and Recommendations ............................................. 17
4.1 Findings from the Computer and Internet Module ................................................................. 22
4.2 Version Preference and Recommendations for the Computer and Internet Module ................ 26
5. Food Stamps: Summary of Findings and Recommendations............................................................ 31
5.1 Findings from the Food Stamps Module ............................................................................... 32
5.2 Version Preference and Recommendations for the Food Stamps Module............................... 35
6. Parental Place of Birth: Summary of Findings and Recommendations ............................................. 36
6.1 Findings from the Parental Place of Birth Module................................................................. 38
6.2 Version Preference and Recommendations for the Parental Place of Birth Module ................ 42
7. Veteran Identification: Summary of Findings and Recommendations .............................................. 45
7.1 Findings from the Veteran Identification Module .................................................................. 47
7.2 Version Preference and Recommendations for the Veteran Identification Module ................. 51
8. Veteran Period of Service: Summary of Findings and Recommendations ........................................ 53
8.1 Findings from the Veterans Period of Service Module........................................................... 56
8.2 Version Preference and Recommendations for the Veterans Period of Service Module.......... 57
9. Wages and Salary: Summary of Findings and Recommendations .................................................... 59
9.1 Findings from the Wages and Salary Module ........................................................................ 60
9.2 Version Preference and Recommendations for the Wages and Salary Module ....................... 63
10. Interest and Dividends: Summary of Findings and Recommendations ............................................ 65
10.1 Findings from the Interest and Dividends Module ................................................................. 67
10.2 Version Preference and Recommendations from the Interest and Dividends Module ............. 71
11. Cash Public Assistance: Summary of Findings and Recommendations ............................................ 74
11.1 Findings from the Cash Public Assistance Module................................................................ 76
11.2 Version Preference and Recommendations for Cash Public Assistance.................................. 78

iii

Table of Contents
12. Conclusions, Lessons Learned, and Recommendations for Future Research .................................... 80
12.1 Lessons Learned ................................................................................................................... 80
12.2 Recommendations for Future Research ................................................................................. 80

Table of Contents

List of Tables
Table 2-1. Modules by Version and Mode .................................................................................. 4
Table 3-1. English Recruiting Targets and Interviews by Demographic Characteristic .............. 10
Table 3-2. English Recruiting Targets and Interviews by Module ............................................. 11
Table 3-3. Spanish Recruiting Targets and Interviews by Demographic Characteristic ............. 13
Table 3-4. Spanish Recruiting Targets and Interviews by Module............................................. 14
Table 3-5. English Screening and Participant Recruitment ........................................................ 15
Table 3-6. Spanish Screening and Participant Recruitment ....................................................... 15
Table 4-1. ACS Questions Tested for Computer and Internet.................................................... 17
Table 4-2. Proposed Alternate Wordings for the Computer and Internet Question..................... 27
Table 5-1. ACS Questions Tested for Food Stamps .................................................................. 31
Table 5-2. Proposed Versions for Parental Place of Birth and Ethnicity Questions .................... 35
Table 6-1. ACS Questions Tested for Parental Place of Birth.................................................... 36
Table 6-2. Proposed Versions for Parental Place of Birth and Ethnicity Questions .................... 43
Table 7-1. ACS Questions Tested for Veteran Identification..................................................... 45
Table 7-2. Proposed Alternate Wordings for the Veteran-Identification Question ..................... 52
Table 8-1. ACS Questions Tested for Veteran Period of Service............................................... 54
Table 8-2. Proposed Alternate Wordings for the Veteran-Identification Question ..................... 57
Table 9-1. ACS Questions Tested for Wages and Salary........................................................... 59
Table 9-2. Proposed Alternate Wording for the Wages and Salary Question ............................. 64
Table 10-1. ACS Questions Tested for Property Income ........................................................... 65
Table 10-2. Proposed Question Wordings for the Interest and Dividends Questions.................. 72
Table 11-1. ACS Questions Tested for Public Assistance ......................................................... 74
Table 11-2. Proposed Question Wordings for the Public Assistance Questions ......................... 72

Appendices

Appendices
Appendix 1 – Self-administered English Protocol Guides and ACS Interviews
1A – Version 1 Protocol Guide (English, Self-administered)
1B – Version 1 ACS Questionnaire (English, Self-administered)
1C – Version 2 Protocol Guide (English, Self-administered)
1D – Version 2 ACS Questionnaire (English, Self-administered)
1E – Version 3 Protocol Guide (English, Self-administered)
1F – Version 3 ACS Questionnaire (English, Self-administered)
Appendix 2 – Interviewer-administered English Protocol Guide
2A – Version 1 Protocol Guide (English, Interviewer-administered)
2B – Version 1 ACS Questionnaire (English, Interviewer-administered)
2C – Version 2 Protocol Guide (English, Interviewer-administered)
2D – Version 2 ACS Questionnaire (English, Interviewer-administered)
2E – Version 3 Protocol Guide (English, Interviewer-administered)
2F – Version 3 ACS Questionnaire (English, Interviewer-administered)
Appendix 3 – Crosswalk of Cognitive Interview Probes
3A – Self-administered
3B – Interviewer-administered
Appendix 4 – Self-administered Spanish Protocol Guide (Stateside Version)
4A – Version 1 Protocol Guide (Spanish, Self-administered)
4B – Version 1 ACS Questionnaire (Spanish, Self-administered)
4C – Version 2 Protocol Guide (Spanish, Self-administered)
4D – Version 2 ACS Questionnaire (Spanish, Self-administered)
4E – Version 3 Protocol Guide (Spanish, Self-administered)
4F – Version 3 ACS Questionnaire (Spanish, Self-administered)
Appendix 5 – Interviewer-administered Spanish Protocol Guide (Stateside Version)
5A – Version 1 Protocol Guide (Spanish, Interviewer-administered)
5B – Version 1 ACS Questionnaire (Spanish, Interviewer-administered)
5C – Version 2 Protocol Guide (Spanish, Interviewer-administered)
5D – Version 2 ACS Questionnaire (Spanish, Interviewer-administered)
5E – Version 3 Protocol Guide (Spanish, Interviewer-administered)
5F – Version 3 ACS Questionnaire (Spanish, Interviewer-administered)
Appendix 6 – Self-administered Spanish Protocol Guide (Puerto Rico Version)
6A – Version 1 Protocol Guide (Puerto Rico, Self-administered)
6B – Version 1 ACS Questionnaire (Puerto Rico, Self-administered)
6C – Version 2 Protocol Guide (Puerto Rico, Self-administered)
6D – Version 2 ACS Questionnaire (Puerto Rico, Self-administered)
6E – Version 3 Protocol Guide (Puerto Rico, Self-administered)
6F – Version 3 ACS Questionnaire (Puerto Rico, Self-administered)
vi

Appendices

Appendix 7 – Interviewer-administered Spanish Protocol Guide (Puerto Rico Version)
7A – Version 1 Protocol Guide (Puerto Rico, Interviewer-administered)
7B – Version 1 ACS Questionnaire (Puerto Rico, Interviewer-administered)
7C – Version 2 Protocol Guide (Puerto Rico, Interviewer-administered)
7D – Version 2 ACS Questionnaire (Puerto Rico, Interviewer-administered)
7E – Version 3 Protocol Guide (Puerto Rico, Interviewer-administered)
7F – Version 3 ACS Questionnaire (Puerto Rico, Interviewer-administered)
Appendix 8 – Informed Consent Form (English)
8A – English
8B – Stateside Spanish
8C – Puerto Rico Spanish
Appendix 9 – Screening Questions (English, Stateside Spanish, Puerto Rico Spanish)
9A – English
9B – Stateside Spanish
9C – Puerto Rico Spanish
Appendix 10 – Computer and Internet Module: Final Briefing Recommendations
Appendix 11 – Food Stamps Module: Final Briefing Recommendations
Appendix 12 – Parental Place of Birth Module: Final Briefing Recommendations
Appendix 13 – Veterans Identification Module: Final Briefing Recommendations
Appendix 14 – Veterans Place of Birth Module: Final Briefing Recommendations
Appendix 15 – Salary and Wages Module: Final Briefing Recommendations
Appendix 16 – Interest and Dividends Module: Final Briefing Recommendations
Appendix 17 – Cash Public Assistance Module: Final Briefing Recommendations

vii

Acknowledgements

The authors gratefully acknowledge the assistance of staff members from the U.S. Census
Bureau for their guidance and review in conducting the study and in preparing this report. We
are particularly grateful to Susan Schechter, Susan Ciochetto, Mary Davis, John Chestnut, Todd
Hughes, Debbie Klein, Ivonne Pabón-Marrero, Jennifer Tancreto, Leticia Fernandez, Patti
Goerman, Tony Tersine, Kelly Holder, Mary Frances Zelenak, Yan Jiang, Ed Welniak, Patricia
de la Cruz, Tracy Loveless, Kirby Posey, John Hisnanick, and Ann Dimler.
The authors also acknowledge the contributions of the analysis team member: Mandy Sha, and
reviewers: Rachel Caspar, Emilia Peytcheva, Kerry Levin, and Martha Stapleton Kudela.
Particular thanks are also extended to the cognitive interviewers for this study, Emily McFarlane,
Tim Flanigan, Emilia Peytcheva, Georgina McAvinchey, Rosanna Quiroz, Sonia Rodriguez,
Liliana Aguayo-Huerta, Manuel Borobia, Yolanda Fowler, Patricia Marnell, Cecilia Avison,
Cristina Golab, Joshua Hantman, Jeff Kerwin, Brett McBride, Alicia Norberg, Jorge Restrepo,
and Debra Stark.

viii

Executive Summary

Executive Summary
The American Community Survey (ACS) is a household survey conducted by the United
States Census Bureau that is designed to meet the needs of federal government agencies for that
require data on the American public. As part of the initial step for the next ACS Content Test, the
U.S. Census Bureau developed a Statement of Work designed to pretest new and revised
questions that were proposed by Sponsoring Agencies. The Pretest design involved cognitive
testing methods to evaluate the current ACS question wording against experimental versions of
the questions and to evaluate alternative versions of new questions under consideration for the
ACS. The results of the cognitive interviews conducted as part of the pretest were used to inform
the decision regarding the versions of the questions that will be tested in a production version in
the field in the fall of 2010. This report documents all aspects of the pretest contract, Cognitive
Testing of the American Community Survey Content Test Items, which was conducted from
March 20, 2009, to August 12, 2009.
The primary objective of the pretest cognitive testing was to evaluate eight question
topics as part of the ACS survey. Six of the eight question topics were current questions in the
ACS while two were new question topics that were proposed as additions for a future version of
the ACS. These question topics that were tested were organized into modules for the cognitive
interviews. The modules that were proposed as new questions for the ACS were the following
two modules: Computer and Internet Usage and Parental Place of Birth. The modules with
revised questions were the following: Veterans Identification, Veterans Period of Service, Food
Stamps, Public Assistance Income, Wages and Salary Income and Property Income (Interest and
Dividends).
The modules selected by the Census Bureau for the pretest were chosen for a variety of
reasons. In general, legislation and quality issues identified during a review of earlier ACS data
led to the identification of some of the question revisions that are proposed as well as to the need
for additional data to be collected. In the Housing section of the ACS, there were two modules
that were identified for testing. The Computer and Internet Use questions are proposed additions
because the Federal Communications Commission (FCC) is interested in tracking deployment of
broadband use and ownership of computers. The food stamps questions are being tested because
there was a name change to this program and the new name, Supplemental Nutritional
Assistance Program, was included in the questions in different ways to determine which was
most widely recognized and would lead to the most accurate reporting. In the Detailed Person
section of the ACS, there were six module tested. The Parental Place of Birth questions have
been included in a previous Census questionnaire, but have never been part of the ACS. The
Veterans Identification and Period of Service questions were revised because data users wanted
to simplify the categories for periods of service. The Wages and Salary Income, Property
Income, and Public Assistance questions are being tested to determine the best way to bridge the
gap between the ACS reported data and administrative records data.
As part of the pretest, two or three alternate versions of the questions were tested using
three different modes (self-administered, interviewer-administered in-person, and intervieweradministered by telephone) and two languages (English and Spanish). In the final
recommendations based on the cognitive testing, the best of the alternative versions were

ES-1

Executive Summary

selected to be administered in the next phase of the Content Test, the field test. For modules that
are currently included in the ACS, the recommendation made in this report is for the one best
alternative version of the questions to be evaluated against the current version. In the case of the
new question topics, two versions are recommended for evaluation in the field test.
To implement this pretest of the new and revised questions, the U.S. Census Bureau
contracted with RTI International* (RTI) to conduct the cognitive interviews and make the
recommendations for the versions to be tested in the field. In order to maintain the schedule for
this work, RTI subcontracted with two other research organizations, Research Support Services
(RSS) and Westat. Each organization conducted a specified number of the cognitive interviews
and contributed to the analysis and reporting.
The Cognitive Testing of the ACS Content Test Items included a total of 220 cognitive
interviews, 115 interviews of which were conducted in English and 105 interviews of which
were conducted in Spanish. Cognitive interviews were conducted primarily in May and June
2009 with a small number of interviews conducted in early July 2009.
Methodology and Respondent Characteristics
The cognitive interview protocols were developed by RTI in consultation with RSS and
Westat and in accordance with the U.S. Census Bureau’s Statement of Work for the Task Order.
As such, the protocols were developed to include nearly the complete ACS interview as well as
follow-up probes for specified questions to be tested. The probing was primarily retrospective,
but the probes were administered concurrently within a section of the ACS interview; that is, the
probes for the housing section were administered after the housing questions and the probes for
the detailed person questions were administered following that part of the interview. The only
exception to this was for the Food Stamps module because these questions were nearly always
paired with the Public Assistance Module, which was part of the detailed person section of the
ACS. The protocol began with a scripted introduction and the informed consent for the
interview. The administration of the ACS interview was pre-selected for either the selfadministered version, the face-to-face interviewer-administered version, or the telephone
interviewer-administered version. Probes were scripted as part of the protocol, but the
interviewers were also encouraged to administer spontaneous probes as appropriate.
Interviews were conducted in six states (Colorado, Florida, Maryland, North Carolina,
Illinois, and Virginia), as well as the District of Columbia and Puerto Rico. While this research is
based on a non-probability sample, the sites were selected to represent the diversity of the
Hispanic population residing in the United States. Among the eight sites selected, respondents
who met the recruitment targets for geographical areas (both urban and rural) and for diverse
nationalities among Spanish-speaking respondents (Mexican, Puerto Rican, Central and South
American descent) could be found. All interviews were conducted by professional staff trained to
conduct the interviews and were conducted at a location that assured sufficient privacy. All
respondents were asked for their consent to audiotape the cognitive interviews. Respondents
were paid $40 for participation.

RTI International is a trade name of Research Triangle Institute.

ES-2

Executive Summary

Spanish interview respondents were native Spanish-speakers with little or no knowledge
of English. English interview respondents were primarily native English-speakers; however,
some respondents who spoke English well but as a second language were also interviewed.
There were recruitment targets for the overall demographics of all respondents as well as specific
target characteristics for each module. The targeted demographic characteristics for the overall
recruitment included a mix of categories for gender, age, education level, Hispanic origin or
race, and income. In addition to these typical demographic targets for respondents, we also had
specific recruitment targets for the individual modules such as Wi-Fi users for the Computer and
Internet module, active duty, national guard, and reservists for the Veterans modules, various
types of income for salary and property income questions, public assistance and food stamps
recipients from multiple states. These module-specific recruitment targets were specified so that
the cognitive interview data would include a mix of respondents question-characteristic targets
for each of the modules tested.
Recommendations by Module
Overall, the cognitive interview findings revealed that for each module tested, there were
specific recommendations that could improve the respondents’ understanding of the intent of the
questions. A brief summary of each module tested is provided in this section. Detailed
descriptions of the findings and recommendations by module are provided in Sections 4 through
11 as well as in the corresponding appendices (10 through 17) of this report.
Computer and Internet Module

A set of new computer and internet questions in the ACS, were tested in three versions in
the cognitive interviews, both in interviewer- administered and self-administered modes, in
English and Spanish. The questions covered household internet access, internet service, and
equipment used to access internet. Of the 84 interviews conducted, 42 were in English, and 42 in
Spanish.
The purpose of this sequence of questions is to elicit data on type of computer equipment
in the household, internet access, and type of service through which the internet is accessed. In
Version 1, the information is elicited in that order, whereas Version 2 first elicits data on devices
in the household that can access internet, followed by which service the household has. Finally,
in Version 3 the order is internet access, service used, and equipment owned or used. The
motivation for inclusion of such questions in the ACS is the FCC’s interest in tracking
deployment of broadband use and ownership of computers.
The questions in their self-administered English version read as follows:
Version 1
9. At this house, apartment, or mobile home – do you or any member of this household currently
own or use any of the following computers or related devices?
§

Desktop, laptop, netbook, or notebook computer

Handheld computer, smart mobile phone, or other handheld wireless computer

ES-3

Executive Summary

Some other type of computer

10. At this house, apartment, or mobile home – do you or any member of this household
currently access the Internet?
§

Yes, with a subscription to an Internet service

Yes, without a subscription to an Internet service -> Skip to Question 12

No, Internet access at this house, apartment, or mobile home -> Skip to Question 12

11. At this house, apartment, or mobile home – do you or any member of this household
currently subscribe to the Internet using
§

Dial-up service?

DSL service?

Cable modem or fiber-optic service?

Wireless Internet service, including a mobile Internet plan (exclude in-house
Wi-Fi)?

Satellite service?

Some other service?

Version 2
9. Which of the following devices that could access the Internet are currently owned or used in
this house, apartment, or mobile home? Mark (X) one or more boxes.
§

Desktop, laptop, netbook, or notebook computer

Handheld computer, smart mobile phone, or other wireless device

Some other device: Specify type of device ______________________________

None

10. How do you or any member of this household subscribe to the Internet at this house,
apartment, or mobile home? Mark (X) one or more boxes.
§

No subscription to an Internet Service Provider

Dial-up service

DSL service

Cable-modem or fiber-optic service

Wireless Internet service, including a mobile Internet plan (exclude in-house Wi-Fi)

Satellite service

Some other service: Specify service ______________________________

ES-4

Executive Summary

Version 3
9. Do you or any member of this household subscribe to the Internet at this house, apartment, or
mobile home?
§

Yes

10. What type of Internet service do you or any member of this household have at this house,
apartment, or mobile home? Mark (X) one or more boxes.
§

Dial-up service

DSL service

Cable-modem or fiber-optic service

Wireless Internet service, including a mobile Internet plan (exclude in-house Wi-Fi)

Satellite service

Some other service – Specify service ______________________________

11. Do you or any member of this household currently own or use a computer or related device
at this house, apartment, or mobile home?
INCLUDE desktops, laptops, netbooks, smart mobile phones, hand-held computers, or other
types of computers.
EXCLUDE GPS devices, digital music players, and devices with only limited computing
capabilities, for example: household appliances.
The recruitment of respondents for this module was developed to include a mix of people
who would likely answer these questions differently based on their own situations. Specific
groups of respondents who were targeted included the following types of respondents:
§

living in urban/suburban areas

living in rural areas

living in areas where municipal Wi-Fi service is available

from a variety of income levels and age brackets (including people over age 60).

Virtually without exception, the main intent of each question in all three versions was
clearly understood. Respondents generally realized that these questions were asked at the
household level and not at the individual level. (It is worth keeping in mind, however, that most
respondents interviewed lived with other family members or by themselves.) The questions on
specific equipment owned or used by the household were also interpreted as intended, even if
some of the equipment or devices interviewers probed about (e.g. smartphones, handhelds) were
not known to respondents. They also understood and were able to answer questions about
whether their household had internet access with or without a subscription.

ES-5

Executive Summary

Knowledge of systems and services—and therefore comprehension of related
questions—was equally distributed across age groups, income levels, and gender. More of the
women mentioned that other members of the household “take care of such things” when asked
about internet access and services, and the relatives they mentioned were mostly male. No clear
response patterns emerge when we compare the answers of respondents in rural and
urban/suburban areas. Systems use and internet access appear—at least in our non-probability
sample—evenly distributed across both types of residence.
Some respondents had difficulty answering questions regarding the type of service by
which their households access the internet. Some respondents exhibited limited understanding of
different services or types of access, and some did not have the information because someone
else in the household takes care of such matters. The combined effect of technical confusion and
how responsibilities are distributed among household members led some respondents to great
uncertainty. These issues were true across versions: the main problem respondents had in
answering the questions was due to the lack of knowledge many of them have about the type of
internet service the household has. For example, several respondents reported access through a
variety of services in rather unlikely combinations. Upon probing, it became evident that they
were not certain at all about what they really had. This issue was slightly more prevalent among
Spanish-speaking respondents.
Probes on specific types of equipment and their classifications also showed that use of
technology by many households is ahead of household members’ understanding of how the
technology works and how it should be classified. For instance, in probes about different types of
devices, very consistently, one-third of the respondents in each version—28 in total— either
considered a videogame system as a computer without further qualifying their answers or
defined a videogame as a computer as long as it can access the Internet.
Despite this confusion and uncertainty uncovered in probing, the questions as worded in
any version do not require that respondents classify the equipment they have or use—simply that
they report having it under one of the lists of devices offered. As we understand it, the aim of the
questions is to determine if and how households access the internet. As long as the “other”
category is phrased in the most inclusive way, no data should be missed for lack of finding the
right place in the lists. For instance, in Version 1, instead of “some other type of computer,” the
response choice could read, “some other type of computer or related device.” The one type of
internet-access device that might be missed if asking only for computers are cell phones that
have internet access but no other computing capabilities and are not classified as smart phones or
handheld computers by respondents. Respondents found the lists of equipment to be
comprehensive.
Although some respondents reported thinking about computer usage in general, when it
came to specific responses, they discussed only their home situation as far as computer
equipment and internet access were concerned. Only a handful of respondents mentioned
accessing the internet in public places in addition to the home (e.g., public library, coffee shop).
Because the ACS specifically asks about internet access at the house, apartment or mobile home,
the interpretation of the overwhelming majority of respondents was as intended. Generally,
respondents answered not just for themselves but also for all members of the household (as

ES-6

Executive Summary

indicated above, this may have been different if we had interviewed more respondents living
with roommates).
The concepts of “access” (in Spanish, “acceso”/”conectarse”) and “subscription” (in
Spanish, “suscripción”) to internet were well understood. The terminology employed worked
well both in English and Spanish. Respondents made clear distinctions between subscribing and
accessing without a subscription. Subscribing was widely defined as paying for the service,
whereas accessing without a subscription was generally defined as connecting with internet but
without having a paid service. None of the respondents considered that someone who connects to
internet through a neighbor should answer that they access internet with a subscription.
Over one-fourth of the respondents (n=24) did not know what Wi-Fi is or how it works,
even in very general terms. Even among those who are more familiar with the concept, many did
not know what “in-house Wi-Fi” is and how it differs technically from wireless internet service.
Respondents did not distinguish between wireless internet service and Wi-Fi.
Ten English-language respondents lived in areas with municipal Wi-Fi. Generally, they
were not very knowledgeable about the municipal service. For households who access
community Wi-Fi, the question on whether they subscribe to internet or not could be interpreted
and answered differently depending on whether they consider themselves subscribers or not.
Among our 10 respondents in such areas, only 1 was using community Wi-Fi for internet access.
Even within the same area, respondents do not agree on whether the free municipal Wi-Fi would
constitute accessing with or without a subscription. Others were not familiar enough with the
municipal Wi-Fi in their area to have an opinion.
No specific differences by mode were detected in testing. As for language, the Spanish
and English wording exhibited similar problems. However, Spanish-speaking respondents
generally had less knowledge of the topics in the questions, as more of them had no computer
equipment in the household or had no internet access. The question on internet access was
equally non-problematic for Spanish speakers. However, in the internet service questions
Spanish speakers were slightly more highly represented among those who were uncertain about
the type of service their household subscribes to, which led to response error.
English-language participants did not express a strong liking for any one version. Spanish
language respondents did express a preference for Version 2 or Version 3, as they found them
clearer than Version 1. Respondents liked the include/exclude lists in Question 11. In Version 2,
those with no internet subscription liked that such a choice was explicitly offered. None of the
three versions appeared to lead to substantially higher response error or difficulty. In Version 1,
the Question 9c response option, however, was confusing to 4 respondents, all English speakers.
These respondents were baffled by this question, thinking that 9a and 9b had already covered all
possible devices.
Version 3 stood out as working particularly well, and although respondents could not
necessarily explain what was meant by “limited computing capability,” many were able to list
devices that can be appropriately included there (e.g., i-Pods, GPS devices, stove thermostats).
On the basis of the testing, we recommend Versions 2 and 3 be tested for inclusion in the ACS
with slight modifications in response options. One modification to Version 2 is to have a “None”

ES-7

Executive Summary

response option as the first choice, to allow those who do not have any computing devices to
easily find an answer. The second modification to both versions is to replace the Wireless
Internet Service option with ‘Mobile broadband plan for a computer or cellular phone’ to avoid
the confusion with Wi-Fi. The English version, shown here to illustrate, would be:
Proposed Version 2
9.
Which of the following devices that could access the Internet are currently owned or used
in this house, apartment, or mobile home? Mark (X) one or more boxes.
§

None

Desktop, laptop, netbook, or notebook computer

Handheld computer, smart mobile phone, or other wireless device

Some other device: Specify type of device __________________________

10. How do you or any member of this household subscribe to the Internet at this house,
apartment, or mobile home? Mark (X) one or more boxes.
§

No subscription to an Internet Service Provider

Dial-up service

DSL service

Cable-modem or fiber-optic service

Mobile broadband plan for a computer or cellular phone

Satellite service

Some other service: Specify service __________________________

Proposed Version 3
9.
Do you or any member of this household subscribe to the Internet at this house, apartment, or
mobile home?
§

Yes

10.
What type of Internet service do you or any member of this household have at this house,
apartment, or mobile home? Mark (X) one or more boxes.
§

No service

Dial-up service

DSL service

Cable-modem or fiber-optic service

Mobile broadband plan for a computer or cellular phone

ES-8

Executive Summary

Satellite service

Some other service – Specify service ____________________________

Yes

Table 4-2, Recommendations for Final Versions for Computer and Internet
Questions, in Chapter 4 shows the specific wording of the questions, instructions, and answer
categories proposed for self-administered and interviewer-administered wording for both English
and Spanish.
Food Stamps Module

The Food Stamps question in the ACS is being revised with the goal of increasing the
accuracy with which households report receiving food stamps. The Census Bureau was
interested in testing two versions to determine which would be best to be included in the field
test as a comparison to the current question. The two versions of the new Food Stamps questions
were tested in English and Spanish and in three interview modes (self-administered, face-to-face
interviewer-administered and interviewer-administered by telephone). A total of 87 interviews
were conducted with this module, 43 interviews were conducted with Spanish-speaking
respondents and 44 were conducted with English-speaking respondents.
Although both test versions incorporated the new name of the Food Stamp program,
“Supplemental Nutrition Assistance Program (SNAP),” one placed greater emphasis on this new
name than the other version. Another difference is that the version giving emphasis to this new
program name also explicitly stated that SNAP and Food Stamps are the same program, whereas
the other version did not. The two test versions were as follows (only English is shown):
§

Version 1– In the past 12 months, did you or any member of this household receive a
government benefit from the Supplemental Nutrition Assistance Program (SNAP)—
formerly known as the Food Stamp Program—that can only be used to buy food? Do not
include WIC or the National School Lunch Program.

Version 2– In the past 12 months, did you or any member of this household receive a
government benefit that can be used to buy food? Include Food Stamps and Supplemental
Nutrition Assistance Program (SNAP). Do not include WIC or the National School
Lunch Program.

In order to ensure that the revised Food Stamps questions function properly, it was
important to recruit respondents with certain characteristics, especially with regard to when they
first began receiving food stamps (due to the recent change in the name of the program) and

ES-9

Executive Summary

whether they are receiving other public assistance. Specific groups of respondents that were
targeted included respondents from a variety of states who:
§

First received food stamps more than 2 years ago

First received food stamps within the past 6 months

Received both food stamps and public assistance

Received food stamps only

Overall, the results of the cognitive interviews revealed that the test Food Stamps
questions were well understood by the most respondents in both Spanish and English. However,
of the 87 participants probed on a version of the Food Stamps question, 17 (10 English-speakers,
7 Spanish) experienced noteworthy problems in answering or interpreting it. Twelve participants
appeared to have answered the question incorrectly: 6 participants answered “no” (or did not
answer) when they should have answered “yes,” whereas 6 participants answered “yes” when
they should have answered “no.” The problems observed seemed to be largely independent of
interview mode and form version. Furthermore, we observed no problems that were unique to
either language of the instrument. The following examples illustrate the types of problems
observed with the Food Stamps question:
§

Three respondents reported having received food stamps on the basis of having received
food from a food pantry or similar service. Two of these respondents were Spanishspeaking persons and the third (a recent African immigrant) did not speak English well.

Three other respondents reported having received food stamps on the basis of other forms
of aid. One appeared to have answered “yes” on the basis of SSI because she uses this
money to buy food. Another based his “yes” answer on the fact that his daughter receives
WIC (he saw the instruction to exclude WIC but did not read it closely). In fact, he said
that when he saw the reference to WIC, he assumed he was to count it. A third
respondent reported receiving food stamps on the basis of the school lunch program (she
neglected to read the instruction to exclude it).

Two respondents incorrectly answered “no” due to issues with the reference period. One
interpreted “In the past 12 months” to mean all 12 months, incorrectly answering “no” to
the target question because she had received food stamps only for the previous 2 months.
Another respondent who answered “No” incorrectly did so because she overlooked the
phrase “In the past 12 months.” She thought the question was asking if she currently
received food stamps, and she had stopped getting food stamps 2 months ago.

One respondent incorrectly answered “no,” apparently due to the social stigma of food
stamps. Probing revealed that she had, in fact, received food stamps and the interviewer
noted that she seemed rather embarrassed about it

Perhaps the most important finding from the cognitive interviews was that there was very
little recognition of the Supplemental Nutrition Assistance Program (SNAP) among the
participants. Only about one-third indicated that they had heard of the program, and many (even
among those claiming to have heard of it) assumed it was something other than food stamps,
such as a program devoted to improving nutrition. Furthermore, when asked to compare the two
versions, many participants criticized Version 1 specifically for its emphasis on SNAP, although
a few did appreciate that it points out that the SNAP and Food Stamps are the same program.
ES-10

Executive Summary

Finally, it should be noted that many participants discussed with us how they receive their Food
Stamps benefit on a card which they use at the grocery store. It is our understanding that all
states now make use of an Electronic Bank Transfer (EBT) for providing Food Stamp benefits.
The findings lead us to recommend and altered form of Version 2 for the field test
(English version shown here to illustrate):
In the past 12 months, did you or any member of this household receive a
government benefit card that can only be used to buy food? Include Food Stamps,
now known as the Supplemental Nutrition Assistance Program (SNAP). Do NOT
include WIC, the National School Lunch Program, or assistance from food banks.
Table 5-2, Recommendations for Final Versions for Food Stamps Questions, in
Chapter 5 shows the specific wording of the questions proposed for self-administered and
interviewer-administered wording for both English and Spanish.
Parental Place of Birth Module

The Parental Place of Birth questions are newly proposed questions for the ACS.
Because these questions had never been included in the survey before, the Census Bureau was
interested in testing three versions to determine which two would be best to be included in the
field test. The three versions of the Parental Place of Birth questions were tested in English and
Spanish and in three interview modes (self-administered, face-to-face interviewer-administered
and interviewer-administered by telephone). A total of 67 interviews were conducted with this
module, 38 interviews were conducted with Spanish-speaking respondents and 29 were
conducted with English-speaking respondents.
Each of the three test versions included the same questions about a person’s father’s
country of origin, his or her mother’s country of origin, and the person’s own ancestry or ethnic
background. The question about ancestry of the person was the same in all three versions. The
differences in the versions appeared in the questions about the parents’ places of birth. The
differences in the wording were as follows (shown in English):
§

Version 1 – In what country was (your/this person’s) (father/mother) born?

Version 2 – Was (your/this person’s) (father/mother) born in or outside of the United
States? In what country was (your/this person’s) (father/mother) born? (Included a
clarification statement about U.S. Territories being considered outside of the U.S.)

Version 3 – Was (your/this person’s) (father/mother) born in the United States? In what
country was (your/this person’s) (father/mother) born? (Included a clarification statement
about U.S. Territories being considered outside of the U.S.)

In order to ensure that the newly added Parental Place of Birth questions function
properly for all types of respondents, it was important to recruit respondents from varying
backgrounds. Specific groups of respondents that were targeted included respondents whose:
§

Parents were born in a foreign country

Parents were born in the United States

ES-11

Executive Summary

Parents were born in a U.S. territory (Puerto Rico, Guam, etc.)

Households contained adoptees, step relatives, or foster children.

Overall, the results of the cognitive interviews revealed that the questions about parental
place of birth and ethnicity were well understood by the most respondents in both Spanish and
English. Of the 67 respondents, 61 were able to answer the questions accurately. Of the six
respondents (four in Spanish, two in English) who answered the questions incorrectly, three
respondents reported that their parents had been born in the U.S. when in fact they had been born
in Puerto Rico; one respondent answered the Ethnic Origin question about his father instead of
himself; and the remaining two respondents answered incorrectly because they were adopted and
did not have information on their biological parents.
One of the issues that the Census Bureau was interested in testing was the impact of
question order for the newly added parental place of birth questions. Specifically, if these
questions preceded the ethnicity question, would it cause confusion for the respondents because
they might think the third question was also about the parents rather the person. The analysis of
these questions indicated that while this was not a wide-spread problem, it was noted by several
respondents that they had misinterpreted the ethnicity question to be about their parents.
A second finding that revealed a potential problem with the understanding of the
questions was that respondents were somehow confused about whether the U.S. territories
should be considered as part of the U.S. or not. While the self-administered versions did include
a clarification statement about the U.S. territories, the interviewer-administered version did not
so some respondents initially reported that their parents had been born in the U.S. until the
interviewer clarified. Without a probe, these responses would have been incorrectly recorded as
born in the U.S.
A third significant finding for this module was that respondents who were adopted, had
household members who had been adopted, or were foster children or step children did not
always understand if they should report the biological parents or the parents with whom they
lived. Most reported for their biological parents, but several indicated through the probe
questions that they were not certain what the survey expected them to do. Additional probes
revealed that some would not have been able to report the place of birth for their biological
parents at all because this was not known.
Spanish-speaking respondents preferred Version 1, and English-speaking respondents
preferred Version 3. However, all versions of the questions presented problems for respondents,
especially for respondents with parents born in a U.S. territory. As a result, we suggest using a
revised question for the field test that specifically asks respondents if they were born “in the
United States, in a U.S. territory, or outside of the United States.” Providing all three options
reduced the ambiguity in deciding how to answer if respondents’ parents were born in a U.S.
territory. In addition, for one of the versions to be field tested, we recommend asking the Ethnic
Origin question before the Parental Place of Birth question to determine if the question-order
impact problem identified in the cognitive interviews can be alleviated with that change. The
revised question order and wording for the parents’ place of birth (English intervieweradministered shown as an example), we recommend is:

ES-12

Executive Summary

13.

What is (your / [NAME]’s) ancestry or ethnic origin?

14a. Was (your / [NAME]’s) father born in the United States, in a U.S. territory, or outside the
United States?
§

In the United States – go to Question 15a

In a U.S. territory

Outside the United States.

14b.

In what (country/U.S. territory) was (your / [NAME]’s) father born?

15a. Was (your / [NAME]’s) mother born in the United States, in a U.S. territory, or outside
the United States?
§

In the United States – go to 16

In a U.S. territory

Outside the United States.

15b.

In what (country/U.S. territory) was (your / [NAME]’s) mother born?

Our recommendation is to include Version 3 and the newly revised wording for the field
test. Table 6-2, Recommendations for Final Versions for Parental Place of Birth Questions,
in Chapter 6 shows the specific wording of the questions, instructions, and answer categories
proposed for self-administered and interviewer-administered wording for both English and
Spanish.
Veterans Identification Module

The Veterans Identification question is used to determine an individual’s veteran status.
The Census Bureau was interested in testing two revised versions and the current ACS version of
this question. The three versions of the Veterans Identification question were tested in English
and Spanish in three interview modes (self-administered, face-to-face interviewer-administered,
and interviewer-administered by telephone). A total of 47 interviews, 31 in English and 16 in
Spanish, were conducted with this module.
The Veterans Identification question asks individuals if they have ever served on active
duty in the military and provides a definition of what is included and excluded in active duty.
The interviewer-administered Versions 1 and 2 of this question were identical. The selfadministered Versions 1 and 2 were very similar.
•

Version 1 - “Has this person ever served on ACTIVE DUTY in…/¿Ha estado esta
persona alguna vez en el SERVICIO MILITAR ACTIVO en…”

•

Version 2 - “What is this person’s ACTIVE DUTY military status in…/¿Cuál es
el estatus del SERVICIO MILITAR ACTIVO de esta persona en…”

•

Version 3 - differed from the other versions in the clarification of active duty. In
Versions 1 and 2 the clarification said what to include first (i.e. federal
activation/activación federal) and then what to exclude (i.e. Reserve or National

ES-13

Executive Summary

Guard training/ entrenamiento para la Reserva Militar o Guardia Nacional). In contrast
Version 3 listed the exclude criteria first and then the include criteria.
To ensure that the Veterans Identification question functions properly for all types of
respondents, it was important to recruit respondents from varying backgrounds. For the English
interviews, respondents were recruited to meet five different characteristics:
§

Current active military

Reserves

National Guard

Military veterans

Non-military individuals who are members of a household containing military/veterans.

The ability to read and write in English is required for U.S. military service, therefore
Spanish-speaking respondents were recruited for different categories than were the Englishspeaking respondents:
§

Non-military individuals who are members of a household with military/veterans

Non-military individuals.

One issue that the Census Bureau was concerned about is whether respondents thought
active duty included weekend training for the Reserves and National Guard and their
understanding of “active duty for training.” Thirteen out of 47 respondents (8 English-speaking
and 5 Spanish-speaking) thought that training was active duty. Respondents found the
clarification confusing and contradictory because the question said to exclude training, but they
considered training to be active duty. Despite the confusion, 11 out of these 13 answered the
question accurately because they had actually served on active duty. They were either regular
military or if they were in the Reserves or National Guard, they had been activated. The two
respondents who did answer the question incorrectly had not been activated, and had only
received training for the Reserves or National Guard, yet answered that they were active duty.
This indicates the question is potentially problematic for individuals in the Reserves or National
Guard who received training, but were never activated.
Another issue the Census Bureau was concerned about was the terminology used in the
question including “active duty,” “federal activation,” and “U.S. Armed Forces.” Aside from
considering training as active duty, most English-speaking and Spanish-speaking respondents
understood the terms used in the questions. Five respondents (2 English and 3 Spanish) did have
some difficulty with the phrase “federal activation/activación federal.” However, none of these 5
respondents were in the military themselves, and it did not affect how they answered the
question.
The Census Bureau was also concerned about the phrase “the military Reserves/la
Reserva Militar” versus “the Reserves/la Reserva” and wanted to know which phrase was more
clearly understood and preferred by respondents. The English-speaking respondents were fairly
evenly split on their preferences. However, Spanish-speaking respondents emphatically preferred

ES-14

Executive Summary

“la Reserva Militar” because the shorter phrase “la Reserva” has several different meanings in
Spanish, including “reservations.”
The clarification of active duty was the only part of the question that appeared to be
problematic for respondents. As a result, we recommend the clarification for active duty either be
removed or revised. In addition, we recommended changing the response category for “Only
training for the Reserves or National Guard” to “Only active duty training for the Reserves or
National Guard” so it will be less ambiguous to respondents. We also recommend maintaining
the use of the phrase “la Reserva Military” as opposed to using the shorter phrase “la Reserva.”
The revised wording for the interviewer-administered version of the questions (shown in
English as an example), we recommend is:
28a.

(Has / Have you) ever served on active duty in the U.S. Armed Forces,
military Reserves, or National Guard? Do not include active duty training for the
Reserves or National Guard, but do include activation, mobilization, or
deployment for service in the United States or overseas.

28b.

Are you currently on active duty?

28c.

Have you ever been in the U.S. military Reserves or the National Guard?

The recommended revised wording for the self-administered version of the questions is:
28.

Has this person ever served on active duty in the U.S. Armed Forces, military
Reserves, or National Guard? Do NOT include active duty for training (ADT) for
the Reserves or National Guard, but DO include activation, mobilization, or
deployment for service in the U.S. or overseas. Mark (X) ONE box.
Never served in the military
Only active duty for training (ADT) for the Reserves or National Guard
On active duty in the past, but not now
Now on active duty

Veterans Period of Service Module

Testing a revised version of the Veterans Period of Service question, which included a
shorter set of response options, was another topic of interest for the Census Bureau. The two
versions of the Veterans Identification question (revised and control) were tested in English and
Spanish in three interview modes (self-administered, face-to-face interviewer-administered, and
interviewer-administered by telephone). A total of 47 interviews, 31 in English and 16 in
Spanish, were conducted with this module.
Both versions of the question asked respondents to select the time periods during which
they served on active duty. The difference between the two versions is that the revised question,
Version 1, combined some of the response categories used in the control, Version 3. Specifically,
the following two periods in Version 3, “May 1975 to August 1980” and “September 1980 to
ES-15

Executive Summary

July 1990” were changed to “May 1975 to July 1990” in Version 1. Similar, these two periods in
Version 3, “February 1955 to July 1961” and “March 1961 to July 1964” were changed to
“February 1955 to July 1964” in Version 3.
The Veterans Period of Service question was tested with the same respondents who were
recruited for the Veterans Identification question. Of the 47 respondents, 38 understood the
question as intended and answered accurately. Nine respondents demonstrated some type of
confusion or misunderstanding with the question. Of these, five answered incorrectly because
they included times when they were in training only; two answered incorrectly because they
could not accurately recall the dates; and two answered incorrectly because they did not review
the answer choices thoroughly enough before answering.
Overall, respondents reacted very favorably toward the response categories with no
differences noted between the two versions. However, several respondents commented that the
categories were not formatted consistently—some had dates first and some had verbal
descriptions first. In addition, many respondents indicated that they included time spent in
training when they selected their answers. If it is important that respondents exclude this time,
we suggest adding an instruction to exclude time spent in training. Otherwise, RTI simply
recommends reformatting the response categories so they consistently list the dates first with the
verbal description in parentheses, which will help ensure respondents notice the dates as well as
the names of the categories.
The recommended revision for the interviewer-administered version of the questions
(shown in English as an example), is:
29. [Using Card A, please tell me each period in which ( / you) served on
active duty, even if it was just for part of the period. / Did ( / you) serve on
active duty during any of the following periods?] Do not include time spent in
training for the military Reserves or National Guard.
September 2001 or later
August 1990 to August 2001 (including Persian Gulf War)
May 1975 to July 1990
August 1964 to April 1975 (Vietnam Era)
February 1955 to July 1964
July 1950 to January 1955 (Korean War)
January 1947 to June 1950
December 1941 to December 1946 (World War II)
November 1941 or earlier
The recommended revised wording for the self-administered version of the questions is:
29. When did this person serve on active duty in the U.S. Armed Forces? Do not
include time spent in training for the Reserves or National Guard. Mark (X) a box
ES-16

Executive Summary

for EACH period in which this person served, even if he or she served just for
part of the period.
September 2001 or later
August 1990 to August 2001 (including Persian Gulf War)
May 1975 to July 1990
August 1964 to April 1975 (Vietnam Era)
February 1955 to July 1964
July 1950 to January 1955 (Korean War)
January 1947 to June 1950
December 1941 to December 1946 (World War II)
November 1941 or earlier

Salary and Wages Module

The Salary and Wages questions in the ACS, which include questions about earned
income – salary, wages, bonuses, tips and commissions, were tested in two versions in the
cognitive interviews. The current ACS question asks about all of these types of earned income
in a single question, and the Census Bureau had concerns about order effects caused by the
presentation of the list of types of earnings. Therefore, the revised version was designed to ask
the question in two steps: first ask about salary and wages, and then ask about additional earned
income in bonuses, tips and commissions. In addition, a question on self-employment income
was explicitly added to further separate different types of earnings and make sure respondents
remembered to mention that income as well. The reason for the proposed change to these earned
income questions was to determine if having respondents report the sources of income separately
would lead to more complete reporting. Only the two interviewer-administered modes were
tested: telephone and fact-to-face. There were two versions of the Salary and Wages module
tested in the interviewer-administered mode, with 13 interviews conducted in English and 18 in
Spanish for a total of 31 interviews.
Both versions tested included an introduction that established the reference period: “The
next few questions are about income DURING THE PAST 12 MONTHS…” Following that
introductory statement, the current ACS question (Version 2) asks if the respondent received
“any wages, salary, tips, bonuses, or commissions” and, if so, how much was received before
taxes and other deductions. The alternative questions tested separating the different types of
earnings. After the introductory statement questions ask if the respondent received any wages or
salary, and if so, how much they received from all jobs before taxes and other deductions.
Respondents are then asked whether they received any additional tips, bonuses or commissions
during the past 12 months and if so, how much the person received from all jobs before taxes and
other deductions.
To ensure that the Salary and Wages questions function properly for all types of
respondents, it was important to recruit respondents from varying backgrounds to test the
ES-17

Executive Summary

questions both in English and Spanish. Respondents were recruited for the following
characteristics:
§

Earned income includes a significant (at least 10%) portion from tips,

Earned income includes a significant (at least 10%) portion from bonuses or
commissions,

Multiple jobs but no self-employment, and

Multiple jobs and self-employment.

Across versions, there was a general tendency for respondents to report what they
considered substantial or important, and not to report smaller amounts or earned income from
jobs they considered not to be regular jobs. Respondents did not necessarily think of themselves
as self-employed when doing small or odd jobs. Among the five respondents with multiple jobs
and no self-employment, three included the wages and salary from all of their jobs; the
remaining two did not. One of the respondents reported income from her main job only because,
as she explained, the series of questions preceding this focused on last week’s job, and that form
of questioning led her to focus only on her main job. The other respondent also reported only the
wages from his “basic” employment. Of the seven respondents screened as having multiple jobs
and also engaging in self-employment, six confirmed self-employment and a job. Of these six
respondents, five reported their job income, including the self-employment income, collectively
under “wages and salary.” This is problematic because a separate self-employment question
appears later in the questionnaire and there can be double reporting of the self-employment
earnings. The Wages and Salary question precedes the question about self-employment. As a
result, at least three respondents who were self-employed reported their self-employment income
in this section. All three respondents also reported their income under the Self-Employment
questions in the ACS survey, effectively double-counting their income.
Respondents who had only one source of income from a regular paycheck did not appear
to have difficulty deciding what to include, regardless of version. However, some respondents
had a broader interpretation and included bonus, tip, or commission income under wages and
salary.
All but a handful of respondents felt it would be much easier to report their earned
income if they were interviewed soon after tax-return preparation. This suggests that respondents
were thinking of—and possibly reporting about—the prior calendar year rather than the past 12
months. This was confirmed from the probes as we indicate below. In addition, almost all
respondents reported it would be easier to report gross income before deductions. This was
particularly true of respondents who received a regular paycheck. Only a few respondents
indicated it would be easier to report pretax earnings.
For both versions of these questions, some respondents were not thinking of the correct
reference period (i.e., the past 12 months) when they provided their responses. Of the 14
respondents in Version 1, only 5 (1 in English and 4 in Spanish) seemed to use the correct
reference period. In two other English-language cases, it is not clear whether the respondents
were relying on their tax returns or not. A third case refused to answer the Wages and Salary
question. Of the 17 respondents in Version 2, only 2 (1 in English and 1 in Spanish) clearly kept

ES-18

Executive Summary

the reference period in mind. When respondents answered incorrectly, they tended to either
report for only part of the reference period, report for the 2008 calendar year instead of the past
12 months, or use their current salary information to create an estimate for the past 12 months.
The questions tested in this module were simply worded in both English and Spanish.
The English terms such as “salary,” “wages,” “bonus,” “tips,” “commissions,” “taxes,” and
“deductions” are clear concepts that respondents did not exhibit problems with, whether or not
they received them. The same was true in Spanish for the translated terms (“jornal,” “sueldo
salario,” “bonos,” “propinas,” “comisiones,” “impuestos,” and “deducciones”).
While the English-language respondents divided almost evenly in their preference for one
version over the other, about two-thirds of the Spanish-language respondents preferred Version
2. Because the Spanish-language respondents typically had lower education levels than the
English-language respondents, it is possible their preference was based on the shorter sequence
of text to listen to and process. Those who preferred Version 2 particularly liked the fact that it
consisted of a shorter sequence of questions. Some felt the early mention of bonuses,
commissions, or tips might have helped them remember to report these types of income. Those
who preferred Version 1 particularly liked the fact that the sequence of questions asked about the
different types of earnings separately. The respondents who received bonuses also had a
preference for Version 1.
Of the 14 respondents in Version 1, there were 8 (5 in Spanish and 3 in English) who
seemed to understand the questions as intended and to answer them correctly. Five respondents
(1 in Spanish, 4 in English) answered the questions inaccurately, and the remaining respondent
refused to answer. Those with salary or wages only were more likely to understand the question.
While no particular demographic patterns seemed to be associated with the respondents who had
problems in all cases, the problems occurred with respondents who had more complicated
earnings than just a salary or regular wages. The problems detected included difficulty recalling
earnings, reporting commissions or bonuses under wages and salary, double reporting of
earnings, and earnings entirely left out.
Of the 17 respondents in Version 2, there were 11 (5 in English and 6 in Spanish) who
seemed to understand the questions as intended and to answer them correctly. There were no
consistencies across the demographic characteristics of respondents observed in this group.
Neither did the group include more of any specific type of income recipient. Six respondents had
different types of problems that made them misreport the amounts earned or report them under
the wrong category of earnings. In addition, the combination of tips, bonuses, and commissions
with wages and salary made some respondents selectively hear only some of these types of
income.
For this reason, we recommend keeping Version 1 with some modifications designed to
stress the reference period and to reduce double reporting of tips, bonuses, and commissions.
One modification is to restate the reference period in each question. A second modification is to
add ‘in total from all jobs’ to stress the need to be fully inclusive. The third modification puts
parentheses around ‘additional’ so that it is only read to those who answer Yes in 49a1a. These
modifications will help both the Spanish-speaking and English-speaking respondents. The
English version of the recommended questions, shown here to illustrate, is:

ES-19

Executive Summary

The next few questions are about income DURING THE PAST 12 MONTHS. . .
49a1a. Did you receive any wages or salary during the past 12 months?
§

Yes

No à Skip to Question 49a2a

49a1b. How much did you receive in total for all wages and salary from all jobs in the past 12
months before taxes and other deductions?
49 a2a. Did you receive any (additional) tips, bonuses, or commissions DURING THE PAST 12
MONTHS?
§

Yes

No à Skip to Current Question 47b

49a2b. How much did you receive in total for all tips, bonuses, or commissions from all jobs in
the past 12 months before taxes and other deductions?
Table 9-2, Recommendations for Final Versions for Salary and Wages, in Chapter 9
shows the specific wording of the questions proposed for self-administered and intervieweradministered wording for both English and Spanish.
Interest and Dividends Module

The property income questions in the ACS, which include questions about interest and
dividends as well as net rental income, income from royalties, estates and trusts, were tested as
separate questions for the cognitive interviews, The reason for the proposed change to these
income questions is to determine if having respondents report the sources of income separately
leads to more accurate reporting. Based on previous data collected by the Census Bureau for
these questions, the interviewer-administered and self-administered modes were not producing
similar results. For the cognitive interviews, only the two interviewer-administered modes were
tested: telephone and fact-to-face. There were two versions of the Interest and Dividends module
tested in the interviewer-administered mode. A total of 33 interviews were conducted using this
module. About half of the interviews were conducted in English (n=17) and about half in
Spanish (n=16).
Both of the test versions of the ACS questions included clarification statements following
the question asking if income from interest or dividends were received. The clarification
statements were included to ensure that even small amounts of income would be recorded. The
text in both versions was very similar, but Version 2 had an additional point of clarification
added. The Version 1 text read, “Report even small amounts credited to an account.” The
Version 2 text read, “Report even small amounts credited to a checking or savings account.”
The recruitment of respondents for this module was developed to include people a mix of
people who would likely answer these questions differently based on their own situations.
Specific groups of respondents who were targeted included:
§

People who reported having income from interest or dividends only,

ES-20

Executive Summary

People who reported receiving income from rental income, royalty income, income from
estates or trusts, and

People who had none of these types of income.

The majority of respondents could answer, without difficulty, the questions that asked
whether they received any interest or dividends and the amount received. Among the respondents
who reported having received interest or dividends during the screening process, there was no
confusion about the initial question that asks if they receive this type of income. Some of these
respondents did, however provide responses to the follow-up question about the amount received
that would either cause over-reporting or under-reporting. The two primary reasons for
misreporting were related to the 12-month reference period or reporting for other household
members.
Respondents expressed some confusion about the 12-month reference period and a
number of respondents who had property income to report indicated they would report the annual
income for the previous year rather than reporting for the past 12 months as intended. Some
were confused about the intention of the questions while other simply said they would report
what was on their previous year’s taxes for income. Adding the specific months to the questions
would help clarify this question for those respondents who were confused. This recommendation
is included in the final version proposed for the field test.
While the Spanish-speaking respondents showed more difficulty with these questions, the
primary issues for them seemed to be related to familiarity with these types of income rather than
the language or accuracy of the translation. This unfamiliarity with the topic of these questions
was related to the lower education levels and lower income among the Spanish-speaking
respondents who participated in this module. Although there were fewer cases with lower
income and education among the English-speaking respondents, they showed the same tendency
to express confusion. When these respondents who expressed confusion about the questions
explained what they were thinking, it was clear that many of them had a general idea of what
was being asked but they did not have this type of income to report. There were a few
respondents who seemed not to have a clear idea of the intent of the questions, but they too
clarified through the probing that they did not have this type of income to report. Despite
apparent confusion for some respondents, most respondents seemed to be able to answer the
questions correctly. Most respondents who did not understand the terminology reported no
income from interest or dividends, which was correct.
The only terminology that was problematic, “net rental income” (“ingreso neto de
rentas”), appeared in the question about types of income such as rent, royalty income, or income
from estates and trusts. Respondents mostly believed they understood the term correctly, but
many described something altogether different from the intended meaning and would have
answered the question incorrectly. This was the case for both English-speaking and Spanishspeaking respondents in this module. A recommendation was made to define the term “net” in
the version of this question included for the field test.
For the versions of the question about interest or dividends that were tested, respondents
had a slight preference for Version 2 because it provided an example of one type of interest
payment to be considered. However, the analysis of the comments revealed that the clarification
ES-21

Executive Summary

caused some respondents to misinterpret the question. The final recommendation for the version
to test in the field was a modified Version 2. One modification added clarification that the
example included (“checking or savings accounts”) was only an example and that other interest
or dividends should also be included. A second modification clarifies the reference period by
specifying the time period to be included in the past 12 months. The third modification
addressed the confusion about the term “net rental income.” The English version, shown here to
illustrate, would be:
Q49c1a. [The next few questions are about income DURING THE PAST 12 MONTHS,
which is from to . .]
Did [/you] receive any interest or dividends [DURING THE PAST 12
MONTHS]? Report even small amounts. For example, report any interest or
dividends credited to a checking or savings account as well as any other income
from interest or dividends
Yes
No à Skip to Question 49c2a
Q49c1b. What was the amount received? ________
Q49c2a. Did [/you] receive any net rental income, royalty income, or income
from estates and trusts [DURING THE PAST 12 MONTHS]? Net rental
income is the amount earned after expenses.
Yes
No à Skip to Question 49d1
Q49c2b. What was the amount received?
Table 10-2, Recommendations for Final Versions for Interest and Dividends, in
Chapter 10 shows the specific wording of the questions proposed for self-administered and
interviewer-administered wording for both English and Spanish.
Cash Public Assistance Module

The Public Assistance question in the ACS is being revised with the goal of increasing
the accuracy with which respondents report receiving public assistance (i.e., welfare payments).
The Census Bureau was interested in testing two versions to determine which would be best to
include in the field test as a comparison against the current question. The two versions of the new
Public Assistance questions were tested in English and Spanish and in three interview modes
(self-administered, face-to-face interviewer-administered and interviewer-administered by
telephone). A total of 75 interviews were conducted with this module, 38 interviews were
conducted with Spanish-speaking respondents and 37 were conducted with English-speaking
respondents.
Both test versions of the new Public Assistance question included phrasing to encourage
respondents to report public assistance as an income source even if they had received it only
once during the 12 month reference period. One version did this by stating “even if for only one
month.” The other version stated “even if for only one payment.” Within the self-administered

ES-22

Executive Summary

mode, a further variation concerned the ordering of two key phrases within the question: “even if
for only one month/payment” and “for this person or any children in this household.” Thus, the
two self-administered versions (only English is shown) were as follows:
Mark (X) the “Yes” box for each type of income this person received, and give your best
estimate of the total amount during the PAST 12 MONTHS…
Version 1: 49f1. Any welfare payments or cash assistance from the state or local welfare office,
even if for only one month, for this person or any children in this household. Do not
include benefits from food, energy, or rental assistance programs.
Version 2: 49f1. Any welfare payments or cash assistance from the state or local welfare office for
this person or any children in this household, even if only one payment. Do not include
benefits from food, energy, or rental assistance programs.
[IF YES] 49f2. TOTAL AMOUNT for past 12 months:_______

In order to ensure the revised Public Assistance question functions properly, it was
important to recruit respondents with certain characteristics. Specific groups of respondents that
were targeted for the cognitive testing included respondents who:
§

Received public assistance only

Received both public assistance and food stamps

Received food stamps only

Receiving public assistance on behalf of child under age 15.

Overall, the results of the cognitive interviews revealed that the test Public Assistance
questions were well understood by most respondents in both Spanish and English. However, of
the 75 participants probed on a version of the Public Assistance question, 17 (11 Englishspeaking, 6 Spanish) experienced noteworthy problems in answering or interpreting it. In our
judgment, 12 to 13 respondents appeared to have answered the question incorrectly: 7
respondents answered “yes” (i.e., that they or another household member had received public
assistance) when they should have answered “no.” We believe two respondents (possibly three)
answered “no” when they should have answered “yes.” The problems observed seemed to be
largely independent of interview mode and form version. Furthermore, we observed no
problems that were unique to either the Spanish or the English versions of the instrument. The
following illustrates the types of problems observed with the Food Stamps question:
§

Four respondents answered “yes’ incorrectly on the basis of other unrelated benefits. One
did so on the basis of unemployment compensation. One answered “yes” on the basis of
SSI payments (unfortunately, the previous SSI item was skipped for this person due to
time constraints, so this may have contributed to the misreporting). One reported
receiving public assistance on the basis of her son receiving Medicaid, and another
because he receives food stamps. Finally, a few other similar problems of interpretation
should be noted here. A couple of respondents (who answered correctly) believed that
child support may be relevant to the question. Another person asked, after hearing the
question, whether it includes unemployment compensation; this respondent specifically
pointed to the phrase, “Include all assistance” in the question as making her think she
ES-23

Executive Summary

should perhaps report it as public assistance. One person answered correctly but
wondered if her SSI payments were relevant to the question (even though she had
reported them in the previous item specifically for SSI).
§

One respondent received public assistance but incorrectly reported that the other two
household members received it as well, since the benefit “is for the entire family.”
Another person (mentioned previously as reporting public assistance based on her son’s
Medicaid) reported that both she and her husband receive public assistance. She noted
the question asks whether each person receives assistance for a child, and, as she sees it,
they both do.

At least two respondents incorrectly reported that they had not received public assistance
within the past 12 months when in fact they had. In each case the problem was due to the
respondent not realizing that the benefit they receive under a given state program name
for welfare (e.g., TANF, Workfirst) was the type of public assistance being asked about
in the question

We probed participants on their interpretation of key terms in the target question,
including “welfare payments” and “cash assistance” (“pago de bienestar público” and “asistencia
en dinero en efectivo” in Spanish). Almost everyone seemed to have an appropriate
understanding of these terms.
The findings of this study did not clearly point to one version being a better form of the
question than the other. The problems and difficulties we observed were almost evenly
distributed across the two versions and unrelated to the wording variations of interest. Our main
recommendation is to place more emphasis on the instruction not to include benefits from other
programs, and the comments obtained from a few participants voicing their preference between
the two versions suggests that referring to “month” would be a safer choice than referring to
‘payment.” Therefore, our recommendation for the field test is as follows (English only is
show):
Interviewer-administered:
DURING THE PAST 12 MONTHS, did [/you] receive any welfare
payments or cash assistance from the state or local welfare office for
[/yourself] or any children in this household, even if for only one month?
Do NOT include benefits from any other type of assistance, such as SSI, food,
energy, or rental assistance programs.
Self-administered:
Any welfare payments or cash assistance from the state or local welfare office,
even if for only one month, for this person or any children in this household? Do
not include benefits from any other type of assistance, such as SSI, food, energy,
or rental assistance programs.
Table 11-2, Recommendations for Final Versions for Cash Public Assistance, in
Chapter 11 shows the specific wording of the questions proposed for self-administered and
interviewer-administered wording for both English and Spanish.
ES-24

Executive Summary

In addition, since a few respondents did not seem to realize that “TANF” is considered
welfare, we suggest creating a show card for interviewers with the name of the TANF program
in each state, similar to the card created for food stamps, so that interviewers can determine
whether the program the respondent mentions is correct
Conclusion
The cognitive interviewing and analysis for this task order identified both effective and
problematic aspects of the proposed question wordings that were tested across the 115 English
interviews and 105 Spanish interviews conducted. The issues that have been identified in this
research are highlighted in Sections 4 through 11 of this report, which summarize the specific
question series modules that were tested. In many cases, respondents identified problematic
wording, which led to the recommendations for possible revisions that could increase
comprehension and consistency of understanding for Spanish-speaking and English-speaking
respondents for the Field test of the American Community Survey. These recommendations for
the versions to be further tested in the field are noted in Sections 4 through 11 as well.

ES-25

1. Introduction

Introduction

The Cognitive Testing of the American Community Survey (ACS) Content Test Items
was designed as an initial step to help the Census Bureau pretest new and revised questions for
the ACS. Testing these proposed revisions is part of a continued effort to meet the needs of the
federal government agencies that use the ACS data. As required by the Census Bureau guidelines
for pretesting, field testing, and implementing new content changes, this task order contract was
established to cognitively test both English and Spanish versions of questions before they are
implemented in the field.
The Census Bureau contracted with RTI International* (RTI) through a Task Order
Contract that included subcontracts to two other research organizations, Research Support
Services (RSS) and Westat. Working collaboratively to complete the cognitive testing, RTI,
RSS, and Westat conducted 220 cognitive interviews and provided final recommendations for
the question wording to be included in the next phase of the ACS Content Test, which is
scheduled to be field tested in 2010.
The goal of the cognitive testing for the content test items was to determine which of the
new or revised questions should be used in the subsequent field test. In order to ensure that the
revised questions work equally well in all modes of administration for the survey, cognitive
interviews were conducted to specifically test the self-administered versions, the face-to-face
interviewer-administered version, and the telephone interviewer-administered version. The
interviews were also conducted using two different versions of the Spanish interview: stateside
Spanish, which is used in the United States, and the Spanish version of questions used in Puerto
Rico as part of the Puerto Rico Community Survey (PRCS).
Using cognitive interviewing methodology, eight new or revised question topics, with
multiple versions of each question, were tested. With two-to-three versions of each question
module, three modes or administration, and three language versions of the instruments, a total of
18 versions of the ACS/PRCS were tested. A unique cognitive interview protocol was developed
for each of the 18 versions of the questions. Each protocol included a scripted informed consent,
the proposed questions to be tested embedded into the context of the ACS survey, scripted
probes to be administered for selected modules, and a final section of general debriefing
questions for the respondents. These cognitive interviews provided a means of determining any
problems with the questions directly from respondents.
This report documents all aspects of this task order, including the development of the
protocol, translation of the protocol, recruitment of appropriate research participants, cognitive
testing, analysis of data, a summary of the findings and recommendations by question module,
and final conclusions and recommendations for future research. The specific research activities
include the following:

RTI International is a trade name of Research Triangle Institute.

1. Introduction

1.1

Preparing cognitive interview protocols in English and two versions in Spanish (stateside
and Puerto Rican), including administration details, consent forms, incentive receipts, and
other materials required for the cognitive interviews (e.g., interviewer guide)

Translating interview protocols, consent forms, and incentive receipts into Spanish once
the English protocols were finalized

Preparing a recruitment plan in accordance with the criteria provided by the Census
Bureau, which included methods of recruiting, assessing, and selecting respondents;
number and characteristics of respondents who will be interviewed for each question
module; protocol for incentive payment, and the screening questionnaire to be used to
recruit respondents

Recruiting participants (English- and Spanish-speaking) with diverse levels of
educational attainment and diversity by race/ethnicity, age, and gender as well as other
recruitment characteristics specific to the modules that were tested

Conducting cognitive interview training for English and Spanish interviewers

Conducting 45- to 90-minute cognitive interviews in English and Spanish in three modes
(paper, face-to-face and telephone modes)

Preparing the interview summaries, recommendations briefing reports, and this final
research report, based on 220 completed interviews covering 361 question modules.
Items Tested in Cognitive Interviews

Of the eight question modules tested, six modules included revisions to existing questions
and two were new question modules. The cognitive interviews included revised question topics
for the following six modules: veterans identification, veterans period of service, food stamps,
public assistance income, wages income, and property income (interest and dividends); and new
questions for the following two modules: computer and internet usage, and parental place of
birth. Two alternative versions of each question were tested for current questions and three
alternate versions were tested for new questions. Based on the analysis of the respondent data,
recommendations for one test version for revised questions and two test versions for new
questions were made for the field test.
1.2

Schedule

Protocol development took place in early April 2009. Interviewers were trained on April
20, 2009. Interviews were conducted primarily in May and June 2009, with a few interviews
conducted in early July 2009. A draft recommendations report was submitted on July 6, 2009,
with a final recommendations report submitted on July 22, 2009. A final briefing meeting was
held on July 23, 2009, to present and discuss the recommendations. This draft Final Report was
submitted on July 29, 2009, and the Final Report will be submitted on Aug 12, 2009.

2. Methodology

Methodology

The goal of the Cognitive Testing of the American Community Survey (ACS) Content
Test Items task order was to conduct between 174 and 232 cognitive interviews in English and
Spanish in order to determine the most effective versions of questions from among two to three
alternate versions being tested. In total, 320 questions were tested across the three versions, three
modes, and two languages for the eight modules. In order to complete the desired number of
interviews per module, more than one module was tested in some interviews.
This chapter outlines the procedures followed by the researchers in order to accomplish
the research goals?ranging from the identification of geographic locations for theinterviews; to
the selection of staff assigned; to protocol development, training, and implementation of the
cognitive interviews.
2.1

Initial Planning for Staffing and Interviewing

Interviews were conducted across eight geographic locations, including Colorado,
Florida, Maryland, North Carolina, Illinois, and Virginia, as well as the District of Columbia and
Puerto Rico. Some sites were selected specifically because they are close to the research teams
geographically, which helped with logistics and in controlling costs. Other sites were selected to
ensure adequate coverage of the recruitment target characteristics, such as Internet Wi-Fi access,
rural geographic areas, or sufficient concentrations of Hispanic populations, including Puerto
Rican, Mexican, and other Spanish-speaking respondents.
Because all steps in the protocol required extensive knowledge of cognitive interviewing
as well as language skills for the Spanish interviews, the first step was to confirm the teams of
cognitive interviewers for each subcontractor. As the task order contract was operating under a
very tight schedule, identifying the interviewing staff early in the contract period was critical. All
staff were required to complete security clearance paperwork and receive special sworn status
from the U.S. Census Bureau before conducting interviews. They also completed the Title 13
training before conducting any work on the project. Delays in acquiring security clearance
created a challenge for some of the teams, but in the end, the interviews were completed in
sufficient time to be included in the Final Briefing Recommendations Report.
The qualifications and experience considered in assembling the cognitive interview teams
for Spanish included native-speaker language competence, education and work experience in the
target culture, and knowledge of and experience with cognitive interviewing. For English
interviewers, education, work experience, and knowledge of and experience conducting
cognitive interviews were considered.
RTI, RSS, and Westat operated independently for recruitment and interviewing, but
frequent communication took place to ensure that consistency was maintained. Each interview
team comprised one lead researcher and a team of Spanish and English interviewers. A total of
18 cognitive interviewers were trained and conducted interviews, including two of the lead
researchers who also conducted interviews.

2. Methodology

2.2

Development of Cognitive Interview Protocols and Forms

The Census Bureau provided all versions of the ACS questionnaires to be tested. Twelve
different versions were provided to cover the three versions of the questions, two interview
modes (self- and interviewer- administered), and three languages. The new questions to be tested
had up to three possible question wordings to be pretested, and the revised questions had up to
two possible revised question wordings. Table 2-1 shows the versions and modes that were
tested for each module. For all versions and modes shown, the modules were tested in both
English and Spanish.
Table 2-1. Modules by Version and Mode
Self-administered
Modules

Interviewer-administered

Computer and Internet Use

Food Stamps

—

-—

Parental Place of Birth

Veterans Identification

Veterans Period of Service

—

Wages and Salary

—

Interest and Dividends

—

Cash Public Assistance

—

The cognitive interview protocols were initially developed by RTI then reviewed by RSS
and Westat before being submitted to the Census Bureau for final review and approval. Scripted
cognitive interview protocols were developed in English and reviewed by staff at the U.S.
Census Bureau, including members of the ACS language team (see Appendices 1, Selfadministered English Protocol Guides (A-F) and 2, Interviewer-administered English Protocol
Guides (A-F) for the three versions of the self-administered and three versions of the
interviewer-administered English protocol guides and their corresponding ACS instruments).
Appendix 3, Crosswalk of Cognitive Interview Probes, provides a crosswalk of the English
probes used in each version of the protocol for both the self-administered and intervieweradministered modes. Development of the Spanish versions of the protocols is documented in
Section 2.3.
The cognitive interview protocol documented the administration details, consent forms,
and materials required for the cognitive interviewing, including a list of standard probes and
special instructions to be used, and a guide for the interviewers to follow during interviews and
reporting. Because the protocol included scripted instructions to be read to the respondent, it also
served as a guide for the administration of consent forms and to confirm the point in time when
audio recording of the interview should begin. The protocol was designed to standardize
implementation of the cognitive interviews in English and Spanish.
Although the cognitive interview protocol was developed to include the majority of the
questions from the ACS, the protocol guide focused on probing the specific question modules
that were of interest for this project. For each module tested, a debriefing section was developed

2. Methodology

to include questions designed to evaluate the alternative version(s) of the question topics that the
respondent was not asked during the interview.
The interview protocols were tested and timed before they were finalized. However,
because there was great variability in the length of administration depending on the mode (selfor interviewer-administered), the reading skills of the respondent, the number of household
members, and the specific modules selected for an interview, the overall interview time varied
significantly. In order to ensure that no interviews were excessively long, interviewers were
instructed to reduce the number of household members asked about in the survey or to skip
sections of the survey that were not being probed if the interview was running long.
Once finalized, all documentation, including the procedures that were followed for the
translation of the protocols into Spanish, was submitted to RTI’s Human Subjects Institutional
Review Board (IRB) for approval. The Census Bureau obtained a waiver from the Office of
Management and Budget (OMB) for this work prior to the start of interviewing.
2.3

Translation Methodology for the Cognitive Interview Protocols

The versions of the ACS questionnaires that were tested had been translated previously
through another Census Bureau contract. The translation task for this contract was to translate
the cognitive interview protocols and interview materials, including the consent form and
respondent incentive receipt (see Appendices 4A-4F and 5A-5F for the stateside Spanish and
Appendices 6A-6F and 7A-7F Puerto Rican Spanish protocol guides and ACS instruments).
The RSS language team conducted committee translation of the cognitive interview protocols
and forms. Once developed, the Spanish protocols were reviewed by RTI and Westat, then
submitted to the Census Bureau for review and final approval.
Team or committee approaches to translation have been used since the 1960s (Nida,
1964) and in the translation of data collection instruments (Brislin, 1976; Schoua-Glusberg,
1993; Guillemin, Bombardier, and Beaton, 1993; Acquadro, Jambon, Ellis, and Marquis, 1996).
In recent years, survey researchers’ and survey translators’ dissatisfaction with traditional
translation and assessment methods (e.g., back translation) has led to the wider adoption of team
approaches. The U.S. Census Bureau Expert Panel on Translation and the Translation Task Force
for the European Social Survey have indicated that back translation is not a satisfactory
approach. The Census Bureau Guidelines for Survey Translation recommends following a team
or committee approach (Pan and de la Puente, 2005).
In addition to the cognitive interview protocols, the language team also translated the
interview consent forms and incentive receipts. Finalized and approved versions of the protocol
guides and interview materials were submitted to the IRB before interviewers were conducted in
Spanish.
2.4

Cognitive Interview Training

After finalizing the cognitive testing protocols, a cognitive interviewing training session
was held with all of the subcontractor team members. This initial training session was held at the
Census Bureau’s office in Suitland, MD, on April 20, 2009, and consisted of both
methodological and substantive issues and provided the specific cognitive interviewing
5

2. Methodology

methodologies to be used in this research. The following topics were covered in sequence during
the one-day training:
§

Welcome and Introductions

Background of Project

Overview of Protocol

Highlights of Protocol Guides

Demonstration of Interview (Interviewer-Administered Version 1)

Paired Mock (Interviewer-Administered Version #1)

Review of Differences for Self-Administered Version #1

Paired Mock (Self-Administered Version #1)

Question and Answer Session Following Mocks

Demonstration of Probing

Paired Mock (Interviewer-Administered Version 2)

Summary Reports Overview and Exercise

Final Gathering/Questions and Answers.

The training was designed to outline the research goals and objectives to review the
correct administration of the prepared probes as documented in the protocol guides. For the
Spanish language–specific follow-up training, the goal was to cover the specific language/culture
wording and Spanish translations. An important part of the training for the Spanish-speaking
interviewers involved negotiating appropriate questions and probes in Spanish. During the
Spanish follow-up sessions, each team member practiced administering the protocol guides in
Spanish. Teams also had an opportunity to discuss the impact of culturally-driven perceptions
and topics in the ACS materials. As mentioned previously, all interviewers were experienced in
conducting cognitive interviews in Spanish.
Following a round of revisions to the instruments to facilitate the use of the ACS
instruments with the protocol guides, a follow-up training session was held locally for each of
the subcontractors to review the revisions and allow time for additional practice. The follow-up
training for the Spanish-language interviewers was also held locally by each subcontractor once
the Spanish protocols were finalized.
2.5

Conducting the Cognitive Interviews

An individual folder was created for each sampled interview respondent. The following
materials were included in the interview folder:
§

Two copies of the consent form (one for the respondent and one for records)

Protocol guide (mode-specific version that matched other documents)

Roster (part of ACS questionnaire – separate document for CAI-administered versions
only)

2. Methodology

ACS survey (mode-specific version)

Veterans show card (if Veterans module selected and face-to-face interview only)

Supplemental Nutritional Assistance Program (SNAP) list (if Food Stamps module
selected and face-to-face interview only)

Alternate question sheets (only for modules to be probed – mode-specific version)

Incentive payment and receipt.

Prior to beginning the interview, each participant was assigned to one of the protocols.
The protocol began by providing the participant with an explanation of the research and having
the participant review and sign the informed consent document. (See Appendix 8, Informed
Consent Form.) If the participant agreed, the interviewer tape-recorded the interview.
Interviewers observed the participants while they answered the questions, noting any specific
signs of difficulty, confusion, hesitation, or annoyance. Interviewers asked probing questions to
determine the cause of any observed or spoken confusion or concern on the part of the
participants. Following the final debriefing, the interviewers closed out the interviews and
provided the incentive payment to the participants.
2.6

Reporting Results

Interview summary reports for each interview were prepared and reviewed by the lead
researchers on a flow basis. On June 23, 2009, all summary reports that were completed at the
time were compiled by the three subcontractors, and interview data were disseminated to the lead
researchers and analysts responsible for preparing the draft reports. Each contractor analyzed the
results from all interviews, including English, stateside Spanish, and Puerto Rican Spanish, for a
particular module. RTI analyzed four of the eight modules, and RSS and Westat analyzed two
modules each. A draft briefing recommendations report was submitted to the Census Bureau on
July 6, 2009. As the remaining interviews were completed and documented, the summary reports
were forwarded to the analysts on a flow basis to be included in the final briefing
recommendations report, which was submitted on July 22, 2009.
A Final Recommendations Meeting was held on July 23, 2009, at the Census Bureau
offices in Suitland, MD. Representatives from RTI, RSS, and Westat presented the findings from
the interviews as well as recommendations for the selection of the versions of the questions to be
included in the field test. In addition to Census Bureau staff from the ACS team and the
Statistical Research Division, invited guests from other Federal Agencies with interest in the
ACS Content Test Items attended the meeting.

4. Computer and Internet: Summary of Findings and Recommendations

Recruiting

Participant recruitment for target language interviews was carried out under the
responsibility and direction of RTI, RSS, and Westat as specified in Section 3.2, Recruiting of
English-speaking Respondents, and Section 3.3, Recruiting of Spanish-speaking Respondents,
below. For both languages, the recruitment process followed the same general process for
screening participants. All recruitment information was maintained in a consistent manner, and
all team members followed the security protocol developed to protect potential participants. To
reach the targeted number of participants, however, each subcontractor’s team of recruiters
utilized a cadre of recruiting techniques to determine what was most effective for the targeted
demographics and other recruitment criteria. This section details the consistently applied—as
well as the unique—protocols for recruiting for English and Spanish-speaking respondents.
3.1

Recruiting Targets

As an initial step in the planning process, a Staffing and Recruiting Plan was developed
and approved by the Census Bureau. This document, which was submitted as a separate
deliverable, outlined the recruiting targets for specific demographics and respondent
characteristics for both English and Spanish interviews. As part of the Staffing and Recruiting
Plan, three screening forms were developed (see Appendix 9, Screening Questions). The
screening forms were designed to gather information about respondents’ eligibility for each of
the modules to be tested as well as information about the respondents’ language skills.
Once an interview candidate expressed interest in participating in the cognitive interview,
one of the designated recruiters or interviewers used the scripted questions to complete the
screening form. The eligibility criteria embedded in the screening questions allowed us to target
a mix of age groups, race and ethnicity categories, education levels, income levels, gender
distribution, and other specific characteristics relevant for the ACS questions being tested.
Candidates for cognitive interviews were asked to self-report their language competency. For
Spanish interviews, only those who said they spoke and read Spanish as native-speakers and
could read or speak English less than well were eligible for the cognitive testing.
All procedures were developed in accordance with U.S. Census Bureau’s Policy Office
and Legal Office working plan for the implementation of this task order. All language teams
used a paper document to record recruiting information. This document was designed to track
basic demographics of the respondents and other eligibility data asked during the screening
interview. Each respondent was given an ID number, which was then used in the interview
summaries for identification purposes.
In accordance with the requirements of Title 13, respondent name, address, phone
number, e-mail address, demographic data, and other personal identifying information (PII) of all
household members were not stored in an electronic file. PII from all respondents and all
household members were recorded on paper or audiotape, and all appropriate Title 13
requirements were followed when handling these data. In addition, hard-copy documents
containing respondent PII were not stored or distributed with corresponding cognitive interview
data.

4. Computer and Internet: Summary of Findings and Recommendations

3.2

Recruiting of English-speaking Respondents

Recruiting of the English-speaking cognitive interview participants was conducted by
RTI in North Carolina and Florida, by RSS in Illinois, and by Westat for the interviews in the
Washington, DC metro area, Virginia, Maryland, and Colorado. Typical methods for
recruitment, such as posting flyers, posting notices on the Internet, contacting community
outreach organizations, and inviting respondents to identify others who may be eligible and
willing to participate were used.
To recruit for the specific respondent characteristics required for the modules to be tested,
additional means of outreach were required. Such efforts included the following:
§

Posting specific recruitment targets on Craigslist sites for each of the geographic
locations where interviewing took place.

Contacting community organizations and leaders via e-mail, telephone, and in-person
visits.

Posting advertisements in local newspapers.

Notifications were posted on RTI’s internal website for Research Triangle Park, NC;
Washington, DC; and Chicago to target RTI employees’ friends or family members who
might be interested and eligible.

Snowballing referrals after the completion of the screening questions and/or the
interviews.

Conducting intercept recruitment in North Carolina; Florida; Puerto Rico; Maryland;
Washington, DC; and Virginia, which involved recruiting respondents in person by going
to locations where the targeted groups are likely to be found and completing the screeners
in person. Typically, this sort of recruitment was conducted at community centers, local
Hispanic or Latino markets, and organizations that assist low income and/or immigrant
populations.

In-person posting and canvassing at retail outlets and community grocery stores, hair
salons, independent restaurants, veterans centers, and local shops.

Additionally, Westat used an existing EurekaFacts participant database for potential
candidates, and EurekaFacts conducted telephone recruitment with the aid of targeted
lists of residents of Cumberland, MD and Washington, DC. Candidates who had recently
participated in other research studies were not eligible for the ACS Content Test Items
interviews.

Recruiting targets were based on specifications from the Statement of Work.
Demographic characteristics for English-speaking respondents were tracked separately from the
other characteristics required for the specific questions to be administered in the ACS interview.
Table 3-1 displays the recruiting targets and outcomes for the demographic characteristics for
the English interviews. Table 3-2 displays the recruiting targets and outcomes for the
characteristics specific to the questions by module.

4. Computer and Internet: Summary of Findings and Recommendations
Table 3-1. English Recruiting Targets and Interviews
by Demographic Characteristic
Target

Interviews
Completed

90–120

115

Male

45–60

Female

45–60

18–35

30–40

36–59

30–40

60+

15–20

Less than high school
High school/some college

10–20
40–52

10
49

College graduate

30–42

White
Black

51–68
24–36

62
36

American Indian/Alaska Native

0–16

Asian
Native Hawaiian/Other Pacific Is.

0–16
0–16

6
0

Other

0–16

Low (< $30,000)
Medium ($30,000–$75,000)

36–48
27–36

51
46

High (> $75,000)

15–20

Total
All cases
Sex

Age

Education

Race

Income

4. Computer and Internet: Summary of Findings and Recommendations

Table 3-2. English Recruiting Targets and Interviews by Module

Computer and
Internet

Target

Interviews
Completed

All cases

Urban/suburban

Rural

Wi-Fi

18–35

36–59

60+

Low (< $30,000)

Medium ($30,000–$75,000)

High (> $75,000)

21–28

Parents born in foreign country
Parents born in U.S.

5–7
5–7

9
6

Parents born in U.S. territory

6–7

All cases
Parental
Place of Birth

Household with adoptees, step, foster kids

5–7

24–32
4

31
4

Reserves

6–8

National guard
Veterans

6–8
4–6

6
7

All cases
Currently active
Veterans

Household with veteran/military

4–6

12–16

Income from tips

3–4

Income from bonuses, commission

2–3

Multiple jobs, no self employment

3–4

Multiple jobs and self employment
Other

2
2–3

3
2

All cases
Wages and
Salary Income

All cases
Property Income

12–16

From interest or dividends only
From rental/royalty, etc.

6–8
2

8
5

Neither

4–6

4. Computer and Internet: Summary of Findings and Recommendations

Target

Interviews
Completed

34–44

Receive Food Stamps only (< 6 months)

6–8

Receive Food Stamps only (> 2 years)

6–8

Public Assistance & Food Stamps (any time)

12–16

Public Assistance only

10–12

All cases

Food Stamps
and
Cash Public
Assistance

Child recipient households

State = NC
State = MD

9–11
6–8

9
9

State = VA

3–5

State = IL
State = DC

6–8
6–8

7
10

State = CO

State = FL

1
131–164

1
184

All Cases

3.3

Recruiting of Spanish-speaking Respondents

Recruitment of Spanish-speaking cognitive interview participants was conducted by RTI
in North Carolina, Florida, and Puerto Rico by RSS in Illinois and Florida and by Westat in the
Washington, DC metro area, Virginia, and Maryland. Typical methods for recruitment
documented above for English-speaking cognitive interview participants were also used for
Spanish-speaking respondents; however, the community outreach organizations and word-ofmouth recruitment were the most effective for the Spanish-speaking population.
As with English-speaking participant recruitment, the Spanish-speaking recruiting targets
were based on specifications from the Statement of Work. Demographic characteristics for
Spanish-speaking respondents were tracked separately from the other characteristics required for
the specific questions to be administered in the ACS interview. Table 3-3 displays the recruiting
targets and outcomes for the demographic characteristics for the Spanish interviews. Table 3-4
displays the recruiting targets and outcomes for the characteristics specific to the questions, by
module, for the Spanish-speaking respondents.

4. Computer and Internet: Summary of Findings and Recommendations
Table 3-3. Spanish Recruiting Targets and Interviews
by Demographic Characteristic
Target

Interviews
Completed

84–112

105

Male

42–56

Female

42–56

18–35

28–37

36–59

28–38

60+

28–37

Less than high school
High school/some college

36–45
36–45

27
47

College graduate

12–22

Mexico
Puerto Rico

28–38
21–28

32
25

Other

35–46

Income
Low (< $30,000)

34–46

Medium ($30,000–$75,000)

25–33

High (> $75,000)

25–33

Total
All cases
Sex

Age

Education

Place of birth

4. Computer and Internet: Summary of Findings and Recommendations

Table 3-4. Spanish Recruiting Targets and Interviews by Module

Computer and
Internet

Target

Interviews
Completed

All cases

Urban/suburban

Rural

Wi-Fi

18–35

36–59

60+

Low (< $30,000)

Medium ($30,000–$75,000)

High (> $75,000)

21–28

Parents born in foreign country
Parents born in U.S.

6–7
3–5

15
1

Parents born in U.S. territory

9–11

All cases
Parental
Place of Birth

Household with adoptees, step, foster kids
Veterans

Wages and Salary
Income

3–5

All cases
Non-veteran/military

18–24
4–6

16
6

Household with veteran/military

14–18

All cases
Income from tips

12–16
3–4

20
3

Income from bonuses, commission

2–3

Multiple jobs, no self employment

3–4

Multiple jobs and self employment
Other
All cases
Property Income

From interest or dividends only
From rental/royalty, etc.
Neither

Food Stamps
and
Cash Public
Assistance

2–3

12–16

4–6
4–6

4
5

All cases
Receive Food Stamps only (< 6 months)

34–44
6–8

45
8

Receive Food Stamps only ( > 2 years)

6–8

Public Assistance & Food Stamps (any time)

12–16

Public Assistance only
Child recipient households

10–12
8

12
7

State = NC

2–4

State = MD
State = VA

5–6
5–6

6
4

State = IL

6–8

State = DC

4–5

State = CO
State = FL

1
4

2
4

Puerto Rico

8–10

126–156

177

All Cases

4. Computer and Internet: Summary of Findings and Recommendations

3.4

Recruitment Outcomes

The English recruitment outcomes were met with one adjustment to the original target for
child public assistance. (The targeted number of 8 interviews was changed to 7 interviews.) All
other targets for demographic and modular respondent characteristics were reached for the
English interviews. Table 3-5 shows the number of candidates screened and interviews for the
English interviews.
Table 3-5. English Screening and Participant Recruitment
Screening /Recruitment Outcome

Number of Cases

Total cases screened

428

Cases ineligible

Cases recruited and scheduled

131

Cases interviewed

115

Table 3-6 shows the number of candidates screened and interviewed for the Spanish
interviews.
Table 3-6. Spanish Screening and Participant Recruitment
Screening /Recruitment Outcome

Number of Cases

Total cases screened
Cases ineligible (spoke English)

327
49

Cases recruited and scheduled

130

Cases interviewed

105

For the Spanish interviews, several of the recruitment targets proved to be quite difficult
to find and/or to secure the interview. With the exception of municipal Wi-Fi interviews, there
were at least one or more respondents with each characteristic. The specific recruitment goals
that were not reached for the Spanish interviews are documented below:
§

Male respondents (35 of 42 were completed)

Respondents aged 60 or older (18 of 28 were completed)

Education less than high school (27 of 36 were completed)

High income > $75, 000 (2 of 25 were completed; 1 of 8 for Computer/Internet module)

Rural areas for Internet/computer (6 of 10 were completed)

Wi-Fi areas for Internet/computer (0 of 8 were completed)

Parents born in the U.S. (1 of 3 were completed)

Households with military or veterans (10 of 14 were completed)

Child recipient households for cash public assistance (7 of 8 were completed)

Food stamp recipients in Virginia (4 of 5 were completed).

In an effort to reach the recruitment goals, the schedule for interviewing was extended by
nearly 2 weeks. In light of the tight schedule for final reporting, we did not continue the
15

4. Computer and Internet: Summary of Findings and Recommendations

recruiting and interviewing beyond the first week in July, so the specific recruitment for each
module was not reached for some Spanish targets. The overall number of interviews in Spanish,
however, was within the target range. With the exception of the Veterans module, each of the
overall modular targets was also reached for Spanish interviews.

4. Computer and Internet: Summary of Findings and Recommendations

Computer and Internet: Summary of Findings and
Recommendations

Three versions of the Computer and Internet module were tested in both selfadministered and interviewer-administered modes. A total of 84 interviews were conducted using
the Computer and Internet module. Of the 84 interviews, 29 interviews were completed using
Version 1, 27 interviews were completed using Version 2, and another 28 were completed using
Version 3.
Interviews were conducted in both English and Spanish. Of the 84 interviews, 42 were
conducted in English, and 42 were conducted in Spanish. Respondents were recruited to meet
three different characteristics. Of the 84 interviews, 24 English and 37 Spanish interviews were
with respondents living in urban/suburban areas; 12 English and 7 Spanish interviews were with
respondents living in rural areas, and the remaining 10 English interviews were with respondents
living in areas where municipal Wi-Fi service is available.
Table 4-1 shows the question wording of each version tested.
Table 4-1. ACS Questions Tested for Computer and Internet
Version/
Language
Version 1
English

Self-administered
9.

Interviewer-administered

At this house, apartment, or mobile
home – do you or any member of this
household currently own or use any of
the following computers or related
devices? YES/NO

9a.

At this ,
do you or any member of this household
currently own or use a desktop, laptop,
netbook, or notebook computer?

9b.

(At this )
Do you or any member of this household
currently own or use a handheld
computer, smart mobile phone, or other
handheld wireless computer?

9c.

(At this )
Do you or any member of this household
currently own or use some other type of
computer?

10a.

At this ,
do you or any member of this household
currently access the Internet?

10b.

(At this )
Do you or any member of this household
currently access the Internet with or
without a subscription to an Internet
service?

11.

At this ,
do you or any member of this household
currently subscribe to the Internet using

Desktop, laptop, netbook, or notebook
computer
Handheld computer, smart mobile phone, or
other handheld wireless computer
Some other type of computer
10.

At this house, apartment, or mobile
home -- do you or any member of this
household currently access the
Internet?
Yes, with a subscription to an Internet service
Yes, without a subscription to an Internet
service ? Skip to current Q12
No Internet access at this house, apartment,
or mobile home ? Skip to current Q12

11.

At this house, apartment, or mobile
home – do you or any member of this
household currently subscribe to the
Internet using – YES/NO

dial-up service?

Dial-up service?

DSL service?

a cable-modem or fiber-optic service?

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language

Self-administered

Interviewer-administered

Cable modem or fiber-optic service?

wireless Internet service, including a mobile
Internet plan but excluding in-house Wi Fi?

Wireless Internet service, including a mobile
Internet plan (exclude in-house Wi Fi)?

satellite service?

Satellite service?

some other service?

Some other service?
Version 1
Spanish
(stateside
and
Puerto
Rico)

Actualmente en esta casa,
apartamento o casa móvil, ¿tiene o
usa usted o algún otro miembro de
este hogar alguna de las siguientes
computadoras o equipos
relacionados? SÍ/ NO
Computadora de escritorio, computadora
portátil, miniportátil (netbook) o computadora
personal móvil (notebook)

9a.

Actualmente, en esta [casa, apartamento o
casa móvil], ¿tiene o usa usted o algún
otro miembro de este hogar una
computadora de escritorio, computadora
portátil, miniportátil (netbook) o
computadora personal móvil (notebook)?

9b.

Actualmente, en esta [casa, apartamento o
casa móvil], ¿tiene o usa usted o algún
otro miembro de este hogar una
computadora de mano, smartphone o
alguna otra computadora de mano
inalámbrica?

9c.

Actualmente, en esta [casa, apartamento o
casa móvil] ¿tiene o usa usted o algún
miembro de este hogar algún otro tipo de
computadora?

10a.

Actualmente en esta [casa, apartamento o
casa móvil], ¿tiene usted o algún otro
miembro de este hogar acceso a Internet?

10b.

Actualmente en esta [casa, apartamento o
casa móvil], ¿tiene usted o algún otro
miembro de este hogar acceso a Internet
con o sin una suscripción a un servicio de
Internet?

11.

Actualmente en esta [casa, apartamento o
casa móvil], ¿se suscribe usted o algún
otro miembro de este hogar a Internet
usando

Computadora de mano, smartphone o alguna
otra computadora de mano inalámbrica
Algún otro tipo de computadora
10.

Actualmente en esta casa,
apartamento o casa móvil, ¿tiene
usted o algún otro miembro de este
hogar acceso a Internet?
Sí, con una suscripción a un servicio de
Internet.
Sí, sin una suscripción a un servicio de
Internet ? Pase a la pregunta Q12 actual
No tiene acceso a Internet en esta casa,
apartamento o casa móvil ? Pase a la
pregunta Q12 actual

11.

Actualmente en esta casa,
apartamento o casa móvil, ¿se
suscribe usted o algún otro miembro
de este hogar a Internet usando –
SÍ/NO

servicio de conexión Dial Up?
servicio de DSL?

Servicio de conexión Dial Up?

servicio de cable-modem o de fibra óptica?

Servicio de DSL?

servicio inalámbrico de Internet, incluyendo un
plan de Internet móvil pero excluyendo servicio
WIFI en la casa)?

Servicio de cable-modem o de fibra óptica?
Servicio inalámbrico de Internet, incluyend un
plan de Internet móvil (excluya servicio WIFI
en la casa)?

servicio por satélite?
algún otro servicio?

Servicio por satélite?
Algún otro servicio?
Version 2
English

Which of the following devices that
could access the Internet are currently
owned or used in this house,
apartment, or mobile home? Mark (X)
one or more boxes.

9a.

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language

Self-administered

Interviewer-administered
smart mobile phone, or other wireless
device; some other type of device, or
none? Mark (X) one or more boxes.

Desktop, laptop, netbook, or notebook
computer
Handheld computer, smart mobile phone, or
other wireless device

Desktop, laptop, netbook, or notebook computer

Some other device: Specify type of device
______________________________

Handheld computer, smart mobile phone, or other
wireless device

None

Some other device
______________________________

10. How do you or any member of this
household subscribe to the Internet at
this house, apartment, or mobile
home? Mark (X) one or more boxes.

None

No subscription to an Internet Service
Provider

9b.

What is this other type of device?
_________________________________

10a.

How do you or any member of this
household subscribe to the Internet at this
[house/ apartment/mobile home] - no
subscription to an Internet Service
Provider, dial-up service, DSL service,
cable-modem or fiber-optic service,
wireless Internet service, including a
mobile Internet plan but excluding inhouse Wi Fi, satellite service, or some
other type of Internet service? Mark (X)
one or more boxes.

Dial-up service
DSL service
Cable-modem or fiber-optic service
Wireless Internet service, including a mobile
Internet plan (exclude in-house Wi Fi)
Satellite service
Some other service: Specify service
______________________________

No subscription to an Internet Service Provider
Dial-up service
DSL service
Cable-modem or fiber-optic service
Mobile broadband plan for a computer or cellular
phone
Satellite service

Some other service
______________________________
10b. What is this other service? _____________
Version 2
Spanish
(stateside
and
Puerto
Rico)

¿Cuáles de los siguientes equipos con
los cuales pueden obtener acceso a
Internet tienen o usan en esta casa,
apartamento o casa móvil? Marque (X)
una o más casillas

9a.

Computadora de escritorio, computadora
portátil, miniportátil (netbook) o computadora
personal móvil (notebook)
Computadora de mano, smartphone o algún
otro equipo inalámbrico

¿Cuáles de los siguientes equipos con los
cuales pueden obtener acceso a Internet
tienen o usan en esta [casa, apartamento o
casa móvil]: una computadora de
escritorio, computadora portátil,
miniportátil (netbook) o computadora
personal móvil (notebook); una
computadora de mano, smartphone o
algún otro equipo inalámbrico, algún otro
tipo de equipo o no usa ninguno? Marque
(X) una o más casillas.
Computadora de escritorio, computadora portátil,
miniportátil (netbook) o computadora personal
móvil (notebook)

Algún otro equipo
Especifique el tipo de equipo
________________________

Computadora de mano, smartphone o algún otro

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language

Self-administered

Interviewer-administered
equipo inalámbrico

Ninguno
10.

Algún otro equipo

¿Cómo se suscribe usted o algún otro
miembro de este hogar a Internet en
esta casa, apartamento o casa móvil?
Marque (X) una o más casillas.

Ninguno
9b.

¿Cuál es este otro tipo de
equipo?_____________________________
____

10a.

¿Cómo se suscribe usted o algún otro
miembro de este hogar a Internet en esta
[casa, apartamento o casa móvil]: no tiene
suscripción a un proveedor de servicios
de Internet, servicio de conexión Dial Up,
servicio de DSL, servicio de cable-modem
o de fibra óptica, servicio inalámbrico de
Internet, incluyendo un plan de Internet
móvil pero excluya servicio Wi Fi en la
casa, servicio por satélite, o algún otro
tipo de servicio de Internet? Marque (X)
una o más casillas.

No tiene suscripción a un proveedor de
servicios de Internet
Servicio de conexión Dial Up
Servicio de DSL
Servicio de cable-modem o de fibra óptica
Servicio inalámbrico de Internet, incluso un
plan de Internet móvil (excluya servicio WIFI
en la casa)
Servicio por satélite
Algún otro servicio: Especifique el servicio
__________________________________

No tiene suscripción a un proveedor de servicios
de Internet
Servicio de conexión Dial Up
Servicio DSL
Servicio de cable-modem o de fibra óptica
Servicio inalámbrico de Internet, incluyendo un
plan de Internet móvil (excluya servicio WIFI en la
casa)
Servicio por satélite
Algún otro servicio

Version 3
English

Do you or any member of this
household subscribe to the Internet at
this house, apartment, or mobile
home?

10.

What type of Internet service do you or
any member of this household have at
this house, apartment, or mobile
home? Mark (X) one or more boxes.

10b.

¿Cuál es este otro servicio? ___________

9a.

Do you or any member of this household
subscribe to the Internet at this [house/
apartment/mobile home]?

10a.

What type of Internet service do you or
any member of this household have at this
[house/apartment/mobile home] - dial-up
service, DSL service, cable-modem or
fiber-optic service, wireless Internet
service, including a mobile Internet plan
but excluding in-house Wi Fi, satellite
service, or some other type of Internet
service? Mark (X) one or more boxes.

10b.

What is this service?

11.

Do you or any member of this household
currently own or use a computer or related
device at this [house/apartment/mobile
home]?
INCLUDE desktops, laptops, netbooks,

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language

Self-administered

Interviewer-administered

______________________________

smart mobile phones, hand-held
computers, or other types of computers.
EXCLUDE GPS devices, digital music
players, and devices with only limited
computing capabilities, for example:
household appliances.

¿Se suscribe usted o algún otro
miembro de este hogar a Internet en
esta casa, apartamento o casa móvil?

9a.

¿Se suscribe usted o algún otro miembro
de este hogar a Internet en esta ?

10.

¿Qué tipo de servicio de Internet tiene
usted o algún otro miembro de este
hogar en esta casa, apartamento o
casa móvil? Marque (X) una o más
casillas.

10a.

¿Qué tipo de servicio de Internet tiene
usted o algún otro miembro de este hogar
en esta :
servicio de conexión Dial Up, conexión
DSL, servicio de cable-modem o de fibra
óptica, servicio inalámbrico de Internet,
incluso un plan de Internet móvil (excluya
servicio WIFI en la casa), servicio de
satélite, algún otro servicio. Marque (X)
una o más casillas.

10b.

¿Cuál es este servicio?

11.

Actualmente, ¿tiene o usa usted o algún
otro miembro de este hogar una
computadora o equipo relacionado en esta
?
INCLUYA computadoras de escritorio,
computadoras portátiles, miniportátiles
(netbooks), smartphones, computadoras
de mano, u otros tipos de computadora.
EXCLUYA Sistemas de posicionamiento
global (GPS), reproductores digitales de
música y otros equipos con capacidad
limitada de computación, por ejemplo,
enseres eléctricos.

Servicio de conexión Dial Up
Servicio de DSL
Servicio de cable-modem o de fibra óptica
Servicio inalámbrico de Internet, incluyendo
un plan de Internet móvil (excluya servicio Wi
Fi en la casa)
Servicio por satélite
Algún otro servicio: Especifique el servicio
______________________
11.

Actualmente, ¿tiene o usa usted o
algún otro miembro de este hogar una
computadora o equipo relacionado en
esta casa, apartamento o casa móvil?
INCLUYA computadoras de escritorio,
computadoras portátiles,
miniportátiles (netbooks),
smartphones, computadoras de mano,
u otros tipos de computadora.
EXCLUYA Sistemas de
posicionamiento global (GPS),
reproductores digitales de música y
otros equipos con capacidad limitada
de computación, por ejemplo, enseres
eléctricos.

4. Computer and Internet: Summary of Findings and Recommendations

4.1

Findings from the Computer and Internet Module

Overview

Three versions were tested in the Computer and Internet module. The purpose of this
sequence of questions was to elicit data on type of computer equipment in the household, internet
access, and type of service through which the internet was accessed. The sequence in Version 1
is type of computer equipment, internet access and internet service. Version 2 first asks about
devices in the household that can access internet, followed by type of service the household has.
Finally, in Version 3 the order of questions is internet access, service used, and equipment owned
or used.
Virtually without an exception, the main intent of each question across the three versions
was clearly understood. Respondents generally realized that these questions were asked at the
household level and not at the individual level. However, most respondents lived with family
members or by themselves and we suspect if we had interviewed more respondents who lived
with unrelated roommates, they might have answered just for themselves rather than for the
whole household.
The questions on specific equipment owned or used by the household were also
interpreted as intended, even if some of the equipment or devices interviewers probed about (e.g.
smartphone, handheld computer) were not known to respondents. They also understood and were
able to answer questions about whether their household had internet access with or without a
subscription.
Knowledge of systems and services—and therefore comprehension of related
questions—was equally distributed across age groups, income levels, and gender. More of the
women mentioned that other members of the household “take care of such things” when asked
about internet access and services, and the relatives they mentioned were mostly male. A couple
of the male respondents stated that they had a technical background and therefore understood
things better than others. No clear response patterns emerged when we compared the answers of
respondents in rural and urban/suburban areas. Systems use and internet access appeared—at
least in our non-probability sample—evenly distributed across both types of residence. A
discussion of differences by language appears below.
Internet Service

Some respondents had difficulty answering questions regarding the type of service
through which their households accessed the internet. Some respondents exhibited limited
understanding of different services or types of access, and some did not have the information
because someone else in the household was taking care of such matters. The combined effect of
technical confusion and distribution of responsibilities among household members lead some
respondents to great uncertainty. These issues were true across versions: the main problem for
respondents in answering the questions was the lack of knowledge or understanding many of
them have about the type of internet service the household. For example, several respondents
reported access through a variety of services in rather unlikely combinations. Upon probing, it
became evident that they were not certain at all about what they really had. This issue was
slightly more prevalent among Spanish-speaking respondents.

4. Computer and Internet: Summary of Findings and Recommendations
Computer Equipment

Probes on specific types of equipment and their classifications also showed that use of
technology by many households was ahead of individual household members’ understanding of
how the technology worked and how it should be classified. For instance, in probes about
different types of devices, very consistently, one-third of the respondents in each version (28 in
total) either considered a videogame system a computer without further qualifying their answers
or defined a videogame as a computer as long as it could access the internet.
Despite the confusion and uncertainty uncovered in probing, the questions as worded in
any version did not require respondents to classify the equipment they had or used, but simply to
report having it under one of the lists of devices offered. As we understand it, the aim of the
questions was to determine if and how households accessed the internet. As long as the “other”
category is phrased in the most inclusive way, no data should be missed. For instance, in Version
1, instead of “some other type of computer,” the response choice could read, “some other type of
computer or related device.” The one type of internet-access device that might be missed if
asking only for computers are cell phones that have internet access but no other computing
capabilities and are not classified as smart phones or handheld computers by respondents.
Respondents found the lists of equipment to be comprehensive. Hard pressed by the
probing to come up with “other” devices, a few respondents were able to suggest something else.
This included videogames, GPSs, DVRs, TV (digital TV, Web TV), home phones, regular cell
phones that can access the internet, and e-books. However, these were not – except for rare
exceptions – devices they reported under ‘Other’ in the equipment question; thus, the probe may
simply have ‘forced’ them to try to report something that they would not naturally volunteer in
their answer.
Internet Access

Although some respondents reported thinking about computer usage in general, when it
came to specific responses about computer equipment and internet access, they discussed only
their home situation. Only a handful of respondents mentioned accessing the internet in public
places in addition to the home (e.g., public library, coffee shop). Because the ACS specifically
asks about internet access at the house, apartment or mobile home, the interpretation of the
overwhelming majority of respondents was as intended.
Generally, respondents answered not just for themselves but also for all household
members (as speculated above, this may have been different if we had interviewed more
respondents living with roommates). The exceptions were parents of young children who
reported their children did not yet have access to computers or internet, and a few respondents
who were not familiar with their roommates or boarders’s computer and internet access.
The concepts of “access” (in Spanish, “acceso”/”conectarse”) and “subscription” (in
Spanish, “suscripción”) to the internet were well understood. The terminology worked well in
both English and Spanish. Respondents made clear distinctions between subscribing and
accessing without a subscription. Subscription was widely defined as paying for a service,
whereas accessing without a subscription was defined as connecting to the internet without
having a paid service. Some respondents did not think it was possible to access without a

4. Computer and Internet: Summary of Findings and Recommendations

subscription. Many felt that accessing without a subscription was the equivalent of stealing. In a
few and rare cases, respondents who did not use computers or accessed the internet had vague
notions and could not define these terms when probed; nonetheless, they had no difficulties
correctly answering the survey questions.
None of the respondents considered that someone who connected to internet through a
neighbor should answer that they accessed the internet with a subscription. This included the
three respondents who acknowledged connecting through a neighbor. However, many
participants did not understand how it would be possible to connect to the internet through a
neighbor. Some felt that such a way of connecting would be illegal, like stealing, and therefore
someone accessing internet that way would not be reported it on the questionnaire.
Over one-fourth of the respondents (n=24) did not know what Wi-Fi was or how it
worked. Even among those who were more familiar with the concept, many did not know what
“in-house Wi-Fi” was and how it differed technically from wireless internet service. That is,
participants might have known that they accessed the internet wirelessly, but did not realize there
was an in-home router that received the internet signal through such media as DSL, cable, fiber
optics, and telephone lines, and distributed it wirelessly inside the home. Because so many
respondents were unfamiliar with the concept of Wi-Fi and confused it with the wireless signal
they had at home, it was not surprising that the concept of wireless internet service was also
unclear. Those who seemed to understand these concepts were still unable to comprehend why
the response category for wireless internet service instructed respondents to “exclude in-house
Wi-Fi.” The small group that understood the meaning of the response categories were able to
answer without error.
Municipal Wi-Fi

Ten English language respondents lived in areas with municipal Wi-Fi: four in Florida,
one in North Carolina, and five in Maryland. Three completed a self-administered interview (one
in Version 1 and two in Version 3) and seven were assigned to an interviewer-administered
mode (two respondents received Version 1, one received Version 2 and four received Version 3).
Generally, respondents in these areas were not very knowledgeable about the municipal
service. For those who accessed community Wi-Fi, the question on whether they subscribed to
internet or not could be interpreted and answered differently depending on whether they
considered themselves subscribers or not. Among the 10 respondents in such areas, only 1 was
using community Wi-Fi for internet access. She reported that she did not subscribe to internet,
and clarified that she was using community Wi-Fi. Even within the same area, respondents did
not agree on whether the free municipal Wi-Fi would constitute accessing with or without a
subscription. Other participants were not familiar enough with the municipal Wi-Fi in their area
to have an opinion. Overall, the respondents did not exhibit different response patterns in any of
the three versions.
Differences by Mode

The main differences detected by mode were due to self- or interviewer-administration
rather than question interpretation. That is, questions were left unanswered or were mismarked
on the self-administered form, which could not happen in the interviewer-administered mode.

4. Computer and Internet: Summary of Findings and Recommendations

Otherwise, there was no indication that the questions in one mode created more confusion, were
misinterpreted more often, or led to misreporting more than the other mode.
Differences by Language

The Spanish versions displayed the same issues as the English versions. No specific
translation issues were detected. The Spanish and English wording exhibited similar problems.
However, Spanish-speaking respondents generally had less knowledge on the topic, as more of
them did not have computer equipment in the household or internet access. Nonetheless, and
despite their difficulty with some probes, Spanish-speaking participants (12 out of 13 without
computer equipment) were able to answer the question about computer equipment easily and
correctly.
The question on internet access was equally non-problematic for Spanish and English
speakers. In terms of internet service, Spanish speakers were slightly more uncertain about the
type of service their household subscribed to than were English speakers.
No specific problems were found with the terminology in Spanish. The questions used
English terms rather than translation (e.g. “dial-up”, “DSL”), but this did not impair
understanding as these terms are used by Spanish speakers. None of the Spanish respondents
volunteered new terms for services that were not already mentioned in the questions, nor they
used such terms during the cognitive interview.
Differences by Version

The problems encountered in testing were common to all versions; as described above,
problems included the respondents’ lack of technical understanding about the equipment and
services in their households. While in the case of computer equipment and internet access this
did not cause serious difficulty or misreporting, in the case of internet service used by the
household it lead to incorrect responses.
Because of the complexity and length of the sequence of questions, alternate version
cards were not designed for the interviewer administered mode. Although the self administered
alternate version cards were designed and distributed, their use was limited, to a great degree
because the interviewers were unaccustomed to using cards for these questions in their previous
interviews that were not self-administered. Only 60% of the self-administered cases used the
cards for this module. When they were used, English language participants did not express a
strong liking for any of the versions. Spanish language respondents did express a preference for
Version 2 and Version 3, as they found them clearer than Version 1. Respondents liked the
include/exclude lists in Question 11, Version 3. Those with no internet subscription liked that
Version 2 explicitly offered such a choice.
None of the 3 versions appeared to lead to substantially higher misreporting or
comprehension difficulty. However, in Version 1, Question 9, response option “c” was
confusing to 4 respondents, all English speakers. These respondents were baffled by this
question, as they thought response options “a” and “b” had already covered all possible devices.
Version 3 stood out as working particularly well. First, Question 9 was a very simple
“Yes/No” question, and respondents understood it well, knew their household situations, and
25

4. Computer and Internet: Summary of Findings and Recommendations

could reply without hesitation. Question 10 asked about the same types of services as Versions 1
and 2, but respondents had far fewer problems in selecting an answer. Question 11 also worked
very well; respondents had no problems in providing an answer. The “Yes/No” structure of this
question and the examples of what to include and what to exclude were found to be very helpful
in answering. The include/exclude lists were cited by respondents across versions who selected
Version 3 as their preferred one. Although respondents could not necessarily explain what was
meant by “limited computing capability,” many were able to list devices that could appropriately
be included there (e.g., i-Pods, GPS devices, stove thermostats).
Appendix 10, Computer and Internet Module: Final Briefing Recommendations,
provides additional detail on the findings from this section.
4.2
Version Preference and Recommendations for the Computer and Internet
Module
We recommend keeping Versions 2 and 3 with no substantive wording changes. In the
self-administered formats, because we observed that some respondents did not follow skip
instructions, we recommend that the first option on the response lists is the answer that indicates
the respondent arrived to a given question by mistake (e.g., in Version 2, Question 9, move
“None” up to the top of the list; in Version 3, Question 10, add “No service” at the top of the
list).
To avoid the confusion about “wireless internet service,” we suggest modifying only this
response option. Respondents seemed to key in on the word “wireless.” However, most
respondents were familiar with wireless only as it relates to Wi-Fi and were consequently
confused by the instruction to “exclude in-house Wi-Fi.” We suggest replacing this response
category with either of the following response options: Mobile broadband plan for a computer or
cellular phone, or to be more specific, Mobile broadband plan for a computer or cellular phone
that allows access to the Internet outside of your home. The phrase “mobile broadband” is used
by the three major providers: AT&T, Verizon, and Sprint. In Spanish, these carriers use “banda
ancha móvil”, phrasing we recommend for the Spanish version.
The other problems in interpreting the questions about service or subscription were due to
lack of understanding of technology rather than the format or wording of the questions.
Nonetheless, because much of the terminology is technical and new to respondents, we
recommend “unbundling” the concepts as much as possible. Therefore, a “Yes/No” approach
such as the one employed in Version 3 may lead to the least amount of error.
While no issues with the multiple response options were detected during the cognitive
interviews, we recommend the field test to include clear specifications for the intervieweradministered versions of these questions to ensure that each category is read and a response is
provided. Questions 9a and 10a of Version 2 and Question 10 of Version 3 should be considered
for this clarification.
The recommended modified Versions 2 and 3 appear in Table 4-2.

4. Computer and Internet: Summary of Findings and Recommendations
Table 4-2. Proposed Alternate Wordings for the Computer and Internet Question
Version/
Language
Version 2
English

Self-administered
9.

Interviewer-administered

Which of the following devices
that could access the Internet are
currently owned or used in this
house, apartment, or mobile
home? Mark (X) one or more
boxes.

9a.

None
Desktop, laptop, netbook, or notebook
computer

10.

Which of the following devices
that could access the Internet are
currently owned or used in this
[house/apartment/mobile home] –
a desktop, laptop, netbook, or
notebook computer; a handheld
computer, smart mobile phone, or
other wireless device; some other
type of device, or none? Mark (X)
one or more boxes.

Handheld computer, smart mobile
phone, or other wireless device

Desktop, laptop, netbook, or notebook
computer – Go to 10a

Some other device: Specify type of
device __________________________

Handheld computer, smart mobile
phone, or other wireless device – Go to
10a

How do you or any member of
this household subscribe to the
Internet at this house, apartment,
or mobile home? Mark (X) one or
more boxes.
No subscription to an Internet Service
Provider

Some other device
None – Go to 10a
9b.

What is this other type of device?
____________________________

10a.

How do you or any member of
this household subscribe to the
Internet at this [house/
apartment/mobile home] – no
subscription to an Internet
Service Provider, dial-up service,
DSL service, cable-modem or
fiber-optic service, Mobile
broadband plan for a computer or
cellular phone, satellite service,
or some other type of Internet
service? Mark (X) one or more
boxes.

Dial-up service
DSL service
Cable-modem or fiber-optic service
Mobile broadband plan for a computer or
cellular phone
Satellite service
Some other service: Specify service
__________________________

No subscription to an Internet Service
Provider
Dial-up service
DSL service
Cable-modem or fiber-optic service
Mobile broadband plan for a computer or
cellular phone
Satellite service
Some other service
10b.

What is this other service?
____________________________

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language
Version 2
Spanish
(stateside
and Puerto
Rico)

Self-administered
9.

Interviewer-administered

¿Cuáles de los siguientes
equipos con los cuales pueden
obtener acceso a Internet tienen o
usan en esta casa, apartamento o
casa móvil? Marque (X) una o
más casillas

9a.

Ninguno
Computadora de escritorio, computadora
portátil, miniportátil (netbook) o
computadora personal móvil (notebook)
Computadora de mano, smartphone o
algún otro equipo inalámbrico

Computadora de escritorio, computadora
portátil, miniportátil (netbook) o
computadora personal móvil (notebook)

Algún otro equipo
Especifique el tipo de equipo
________________________
10. ¿Cómo se suscribe usted o algún otro
miembro de este hogar a Internet
en esta casa, apartamento o casa
móvil? Marque (X) una o más
casillas.
No tiene suscripción a un proveedor de
servicios de Internet

¿Cuáles de los siguientes
equipos con los cuales pueden
obtener acceso a Internet tienen o
usan en esta [casa, apartamento
o casa móvil]: una computadora
de escritorio, computadora
portátil, miniportátil (netbook) o
computadora personal móvil
(notebook); una computadora de
mano, smartphone o algún otro
equipo inalámbrico, algún otro
tipo de equipo o no usa ninguno?
Marque (X) una o más casillas.

Computadora de mano, smartphone o
algún otro equipo inalámbrico
Algún otro equipo
Ninguno
9b.

¿Cuál es este otro tipo de
equipo?_____________________

10a.

¿Cómo se suscribe usted o algún
otro miembro de este hogar a
Internet en esta [casa,
apartamento o casa móvil] : no
tiene suscripción a un proveedor
de servicios de Internet, servicio
de conexión Dial Up, servicio de
DSL, servicio de cable-modem o
de fibra óptica, servicio
inalámbrico de Internet,
incluyendo un plan de Internet
móvil pero excluya servicio Wi-Fi
en la casa, servicio por satélite, o
algún otro tipo de servicio de
Internet? Marque (X) una o más
casillas.

Servicio de conexión Dial Up
Servicio de DSL
Servicio de cable-modem o de fibra
óptica
Plan de banda ancha móvil (mobile
broadband plan) para computadora o
teléfono celular
Servicio por satélite
Algún otro servicio: Especifique el
servicio
___________________________

No tiene suscripción a un proveedor de
servicios de Internet
Servicio de conexión Dial Up
Servicio DSL
Servicio de cable-modem o de fibra
óptica
Plan de banda ancha móvil (mobile
broadband plan) para computadora o
teléfono celular
Servicio por satélite
Algún otro servicio
10b.

¿Cuál es este otro servicio?
____________________________

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language
Version 3
English

Self-administered

Interviewer-administered

Do you or any member of this
household subscribe to the
Internet at this house, apartment,
or mobile home?

9a.

Do you or any member of this
household subscribe to the
Internet at this [house/
apartment/mobile home]?

10.

What type of Internet service do
you or any member of this
household have at this house,
apartment, or mobile home? Mark
(X) one or more boxes.

10a.

What type of Internet service do
you or any member of this
household have at this
[house/apartment/mobile home]
Dial-up service

No service

DSL service

Dial-up service

Cable-modem or fiber-optic service

DSL service

Mobile broadband plan for a computer or
cellular phone

Cable-modem or fiber-optic service
Mobile broadband plan for a computer or
cellular phone

Satellite service
Some other type of Internet service?
Mark (X) one or more boxes.

Satellite service
Some other service – Specify service
____________________________
11.

Do you or any member of this
household currently own or use a
computer or related device at this
house, apartment, or mobile
home?
INCLUDE desktops, laptops,
netbooks, smart mobile phones,
hand-held computers, or other
types of computers.
EXCLUDE GPS devices, digital
music players, and devices with
only limited computing
capabilities, for example:
household appliances.

10b.

What is this service?
_____________________________

11.

Do you or any member of this
household currently own or use a
computer or related device at this
[house/apartment/mobile home]?
INCLUDE desktops, laptops,
netbooks, smart mobile phones,
hand-held computers, or other
types of computers.
EXCLUDE GPS devices, digital
music players, and devices with
only limited computing
capabilities, for example:
household appliances.

4. Computer and Internet: Summary of Findings and Recommendations

Version/
Language
Version 3
Spanish
(stateside
and Puerto
Rico)

Self-administered

Interviewer-administered

¿Se suscribe usted o algún otro
miembro de este hogar a Internet
en esta casa, apartamento o casa
móvil?

9a. ¿Se suscribe usted o algún otro
miembro de este hogar a Internet
en esta ?

10.

¿Qué tipo de servicio de Internet
tiene usted o algún otro miembro
de este hogar en esta casa,
apartamento o casa móvil?
Marque (X) una o más casillas.

10a.

Servicio de conexión Dial Up

Ningún servicio

Conexión DSL

Servicio de conexión Dial Up

Servicio de cable-modem o de fibra
óptica

Servicio de DSL
Servicio de cable-modem o de fibra
óptica

Plan de banda ancha móvil (mobile
broadband plan) para computadora o
teléfono cellular

Plan de banda ancha móvil (mobile
broadband plan) para computadora o
teléfono celular

Servicio de satélite
Algún otro servicio
Marque (X) una o más casillas.

Servicio por satélite
Algún otro servicio: Especifique el
servicio
____________________________
11.

¿Qué tipo de servicio de Internet
tiene usted o algún otro miembro
de este hogar en esta ?
INCLUYA computadoras de
escritorio, computadoras
portátiles, miniportátiles
(netbooks), smartphones,
computadoras de mano, u otros
tipos de computadora.
EXCLUYA Sistemas de
posicionamiento global (GPS),
reproductores digitales de música
y otros equipos con capacidad
limitada de computación, por
ejemplo, enseres eléctricos.

5. Food Stamps: Summary of Findings and Recommendations

Food Stamps: Summary of Findings and Recommendations

Two versions of a proposed new ACS question about household receipt of food stamps
were tested. Although both versions incorporated the new name of the Food Stamps program,
“Supplemental Nutrition Assistance Program (SNAP),” Version 1 placed greater emphasis on the
new name than did Version 2. Furthermore, Version 1 explicitly stated that SNAP and Food
Stamps are the same program. Table 5-1 presents the question wording of each Food Stamps
question version tested across modes and languages (English and Spanish).
A total of 87 participants (43 Spanish-speaking, 44 English-speaking) were probed on the
Food Stamps question. Participants were residents of Illinois (19), Maryland (15), District of
Columbia (14), North Carolina (12), Virginia (8), Colorado (6), and Florida (5). In addition, 8
participants were residents of Puerto Rico.
Table 5-1. ACS Questions Tested for Food Stamps Module
Version/
Language

Self-administered

Interviewer-administered

Version 1
English

15. IN THE PAST 12 MONTHS, did you or any
member of this household receive a government
benefit from the Supplemental Nutrition
Assistance Program (SNAP)—formerly known as
the Food Stamp Program—that can only be used
to buy food? Do not include WIC or the National
School Lunch Program.

15. IN THE PAST 12 MONTHS, did you or any
member of this household receive a government
benefit from the Supplemental Nutrition Assistance
Program (SNAP)—formerly known as the Food Stamp
Program—that can only be used to buy food? Do NOT
include WIC or the National School Lunch Program.

Version 1
Spanish
(stateside
and Puerto
Rico)

15. EN LOS ÚLTIMOS 12 MESES, ¿recibió usted
o algún miembro de este hogar algún beneficio
del gobierno por medio del Programa de
Asistencia Nutricional Suplementaria (SNAP)—
antes conocido como el Programa de Cupones de
Alimentos—que sólo se puede usar para comprar
alimentos? NO incluya WIC o el Programa
Nacional de Almuerzos Escolares.

15. EN LOS ÚLTIMOS 12 MESES, ¿recibió usted o
algún otro miembro de este hogar algún beneficio del
gobierno por medio del Programa de Asistencia
Nutricional Suplementario (SNAP)—antes conocido
como el Programa de Cupones de Alimentos—que
sólo se puede usar para comprar alimentos? NO
incluya WIC o el Programa Nacional de Almuerzos
Escolares.

IN THE PAST 12 MONTHS, did you or any
member of this household receive a government
benefit that can be used to buy food? Include
Food Stamps and Supplemental Nutrition
Assistance Program (SNAP). Do not include WIC
or the National School Lunch Program.

14. IN THE PAST 12 MONTHS, did you or any
member of this household receive a government
benefit that can only be used to buy food? Include
Food Stamps and Supplemental Nutrition Assistance
Program (SNAP). Do NOT include WIC or the National
School Lunch Program.

14. EN LOS ÚLTIMOS 12 MESES, ¿recibió usted
o algún miembro de este hogar algún beneficio
del gobierno que sólo se puede usar para
comprar alimentos? Incluya Cupones de
Alimentos y el Programa de Asistencia Nutricional
Suplementaria (SNAP). NO incluya WIC o el
Programa Nacional de Almuerzos Escolares.

14. EN LOS ÚLTIMOS 12 MESES, ¿recibió usted o
algún otro miembro de este hogar algún beneficio del
gobierno que sólo se puede usar para comprar
alimentos? Incluya Cupones de Alimentos y el
Programa de Asistencia Nutricional Suplementario
(SNAP). NO incluya WIC o el Programa Nacional de
Almuerzos Escolares.

Version 2
English

Version 2
Spanish
(stateside
and Puerto
Rico)

5. Food Stamps: Summary of Findings and Recommendations

5.1

Findings from the Food Stamps Module

Overview

Of the 87 participants probed on the food stamps question, 17 (10 English-speakers, 7
Spanish-speakers) experienced problems answering or interpreting the question. Twelve
participants answered the question incorrectly: 6 participants answered negatively (or did not
answer) when they should have answered positively, and 6 participants answered positively
when they should have answered negatively. For the most part, the observed problems seemed
independent of interview mode and form version. There were no language-specific problems.
The following examples illustrate the types of problems observed with the Food Stamps
question:
§

Food from food pantry. Three respondents incorrectly reported having received food
stamps on the basis of having received food from a food pantry or similar service. Two of
these respondents were Spanish speakers and the third (a recent African immigrant) did
not speak English well. At least one other English-speaking participant (who answered
the question correctly) believed that such services are government programs and thus are
relevant for the question. On the other hand, one Spanish-speaking participant was
incorrectly recruited as a food stamps recipient on the basis of aid from a food bank, yet
correctly excluded this aid and answered “no” to the target ACS question.

Other aid. Three respondents incorrectly reported having received food stamps on the
basis of other forms of aid. One answered “Yes” on the basis of SSI because she used this
money to buy food. Another based his “Yes” answer on the fact that his daughter
received WIC (he saw the instruction to exclude WIC but did not read it closely).
Actually, he explained that when he saw the reference to WIC, he assumed he was to
count it. A third respondent reported receiving food stamps on the basis of the school
lunch program (she neglected to read the instruction to exclude it).
One respondent answered “No” incorrectly because she knew her benefits mostly as
“TANF”, or Temporary Assistance to Needy Families (she received both food stamps
and TANF). In response to probing, she remembered that part of the benefit she received
on her electronic bank transfer (EBT) card could only be used for food. It is worth noting
that a few other respondents who received both food stamps and public assistance
referred to those benefits as “TANF” during the probing, even though they did realize the
difference between food stamps and welfare payments.
It should also be noted, however, that the 4 respondents who screened into the study as
having received food stamps on the basis of either WIC (3 persons) or the school lunch
program (1 person), correctly excluded such programs when responding to the target
question during the interview.

Reference period. Two persons incorrectly answered “No” due to issues with the
reference period. One interpreted “In the past 12 months” to mean “All 12 months” and
incorrectly answered she did not receive foods stamps because she had received them
only for the previous 2 months. The other respondent overlooked the phrase “In the past
12 months” and thought the question was asking whether she was currently receiving
food stamps. She answered negatively because she had stopped receiving food stamps 2
months before the interview.
32

5. Food Stamps: Summary of Findings and Recommendations

Social stigma. One respondent answered that she did not receive food stamps due to the
social stigma associated with them. Probing revealed that she had, in fact, received food
stamps and the interviewer noted that she seemed rather embarrassed about it.

Food Stamps and SNAP Terminology

There was very little recognition of the Supplemental Nutrition Assistance Program
(SNAP) among the participants, regardless of ethnicity or language spoken. Only about one-third
indicated that they had heard of the program, and many (even among those claiming to have
heard of it) assumed it was something other than food stamps. For example, some participants
assumed SNAP was a program meant specifically to improve nutrition:
§

“It’s just called food stamps, because I’ve never heard of SNAP.” A minute later: “I
mean, food stamps is just, you get an amount and you can buy food. If it’s not nutritious
or whatever, they don’t care. But the SNAP program sound like something that’s focused
on nutrition.”

“I consider them to be different. I know what food stamps is exactly—what it means to
me, and how I implement it. I’m familiar with the term, this SNAP thing—I have no idea
as a function of how it works. Supplemental Nutrition Assistance, I would assume is
some sort of giving specific foods which would then imply specific nutrition, like infant
formula, milk . . . food for pregnant women, whatever they might need. When it says
‘nutrition,’ I assume it means they’re targeting certain population groups or, and I don’t
know what it is, I’ve never received it, I just know the term.”

Version 1’s emphasis on the “SNAP” terminology almost led two English-speaking
respondents to misreport on the Food Stamps question. Both were responding to the intervieweradministered form of the question, and initially answered “No” upon hearing the reference to
“SNAP.” However, both changed their responses after hearing the reference to food stamps later
in the question. When asked to compare Versions 1 and 2 of the Food Stamp question, some
participants specifically pointed to Version 1’s emphasis on SNAP as a reason for preferring
Version 2. Some suggested that that it would be easy to answer Version 1 incorrectly. One food
stamps recipient stated:
“With that first one, I heard SNAP and I thought right away, that’s not me.”
A few participants noted that they preferred Version 1 because it made clear that SNAP
and Food Stamps were the same program.
Overall, the most commonly used term by English-speaking participants when referring
to food stamps was “food stamps.” This was true for both those who began receiving food
stamps within the past few months and those who had been receiving them for some time.
The Spanish-speaking participants used the terms “cupones de alimentos” or “estampillas
de comida” almost equally, though several were also familiar with the English terms. One
respondent said she was more familiar with the English terms than the Spanish terms and
suggested to include the English term “food stamps” in Spanish version of the question. One
Spanish-speaker who used the term “estampillas” noted that she was not certain it meant the
same as “cupones” (the term used in the question), but assumed so when answering the question.

5. Food Stamps: Summary of Findings and Recommendations

Another respondent did not recognize the terminology “cupones de alimentos”—she knew her
food stamps benefits only by the state’s name for the EBT card (Link card). It is significant that
virtually everyone receiving food stamps noted that they received this benefit on a card,
describing it as “a debit card,” or “a credit card.” Many referred to the state’s specific name for
the card (e.g., Link card). A number of participants (particularly those living in Puerto Rico)
indicated having heard of a program called “PAN,” which was an acronym for a former name of
the Food Stamps program.
Instructions Not to Include WIC or School Lunch Programs

The instructions not to include WIC or school lunch programs appeared to have been read
(or heard) and understood by the vast majority of participants. The instructions effectively kept
several people from misreporting. As one participant put it:
“They [the instructions] clarified it a little bit because I probably would have assumed WIC
was a part of it [the Food Stamps program].”
However, when probed, a small number of participants took issue with these instructions.
They did not see the logic in including the Food Stamps program but excluding others because
the programs are related. One respondent stated:
“It just ran through my mind why only food stamps and not WIC as well? WIC is a
government aid as well—help, for formula or anything like that for kids.”
Although these respondents understood and followed the instructions, their sentiments
are worth noting, as some respondents may be inclined to ignore such instructions.
Differences by Language

Very few (if any) of the observations from these interviews indicate that the processes of
comprehending and answering the Food Stamps question differed between the English-speaking
and Spanish-speaking respondents. One possible exception is the knowledge of government
benefit programs. As noted above, a small number of Spanish-speaking respondents assumed
that receiving food from a food pantry was relevant for the Food Stamps question. Probing of
these participants revealed that they were simply unfamiliar with the Food Stamps program. It
can be expected that recent immigrants to the United States and ineligible persons would have
relatively little knowledge of government benefit programs.
Differences by Version

Of the 43 persons (21 English-speakers, 22 Spanish-speakers) who were administered
Version 1 of the Food Stamps question, 37 (16 English-speakers, 19 Spanish-speakers) answered
correctly and did not have problems understanding what the question asked. Of the 44 persons
(23 English-speakers, 21 Spanish-speakers) who were administered Version 2 of the Food
Stamps question, 35 (18 English-speakers, 17 Spanish-speakers) answered correctly and did not
have problems understanding what the question asked. In this respect, the two versions appeared
quite comparable. But as noted above, many participants did not like Version 1’s emphasis on
the Supplemental Nutrition Assistance Program and two persons almost incorrectly answered
“No” to the Food Stamps question due Version 1’s emphasis on SNAP.

5. Food Stamps: Summary of Findings and Recommendations

5.2

Version Preference and Recommendations for the Food Stamps Module

The findings from the cognitive interviews do not strongly favor one version of the Food
Stamps question over the other. The problems we observed were about evenly distributed across
versions. In fact, when we asked participants which of the two versions they preferred, they were
about evenly split between Version 1 and Version 2. However, the problems and reactions of a
number of participants who did not recognize SNAP (including many who assumed it was
something other than food stamps) suggest that the question should not give emphasis to this
new program name at least that was the finding at the time of these cognitive interviews.
Nonetheless, given that the name change has already been rolled out, it will become more
recognizable among the recipients and the general population with time.
The recommendation for the version of the question to use for the field test is a modified
Version 2. While this version does refer to a card, it makes clear that SNAP and Food Stamps are
the same program. Since all states have converted the food stamps benefit to an EBT card, we
believe that it is important to include the term “government benefit card” as part of the question.
Another addition that we believe will help minimize misreporting is the addition of the
“assistance from food banks” in the exclusion statement at the end.
Table 5-2 shows the recommended version to be tested in the field test.
Table 5-2. Proposed Versions for Food Stamps Question
Language
Modified
Version 2

English
Modified
Version 2

Spanish

Self-administered and Interviewer-administered
In the past 12 months, did you or any member of this household receive a government
benefit card that can only be used to buy food? Include Food Stamps, now known as the
Supplemental Nutrition Assistance Program (SNAP). Do NOT include WIC, the National
School Lunch Program, or assistance from food banks
En los últimos 12 meses, ¿recibió usted o algún otro miembro de este hogar una tarjeta de
beneficios del gobierno que sólo se puede usar para comprar alimentos? Incluya beneficios
del programa de cupones de alimentos o estampillas de comida que ahora se llama
Programa de Asistencia Nutricional Suplementario (SNAP). NO incluya WIC, ni el Programa
Nacional de Almuerzos Escolares, ni ayuda de bancos de alimentos.

6. Parental Place of Birth: Summary of Findings and Recommendations

Parental Place of Birth: Summary of Findings and
Recommendations

Three versions of the Parental Place of Birth module were tested in both selfadministered and interviewer-administered modes. A total of 67 interviews were conducted using
the Parental Place of Birth module. Of the 67 interviews, 23 were completed using Version 1, 24
were completed using Version 2, and another 20 were completed using Version 3.
Interviews were conducted in both English and Spanish. Of the 67 interviews, 29 were
conducted in English and 38 were conducted in Spanish. Respondents were recruited to meet
four different characteristics. Of the 67 interviews, 24 (9 in English, 15 in Spanish) were with
respondents whose parents were born outside of the United States; 7 (6 in English, 1 in Spanish)
were with respondents whose parents were born in the United States; 23 (7 in English, 16 in
Spanish) were with respondents whose parents were born in a U.S. territory; and the remaining
13 (7 in English, 6 in Spanish) were with respondents who were adopted, step, or foster children.
See Table 6-1 for the question wording of each version tested. While the question
wording was the same for stateside and Puerto Rico Spanish, the instructions on the selfadministered version for Puerto Rico were different. The instruction to write the name of the
country listed Puerto Rico first before other countries.
Table 6-1. ACS Questions Tested for Parental Place of Birth
Version/
Language

Self-administered

Version 1 13.
English
14.
15.

Version 1
Spanish
(stateside
and
Puerto
Rico)

Interviewer-administered

In what country was this person’s
FATHER born?

13. In what country was your FATHER
born?

In what country was this person’s
MOTHER born?

14. In what country was your MOTHER
born?

What is this person’s ancestry or ethnic
origin? (For example: Italian, Jamaican,
African Am, Cambodian, Cape Verdean,
Norwegian, Dominican, French
Canadian, Haitian, Korean, Lebanese,
Polish, Nigerian, Mexican, Taiwanese,
Ukrainian and so on.)

15. What is your ancestry or ethnic origin?
(For example: Italian, Jamaican, AfricanAmerican, Cambodian, Cape Verdean,
Norwegian, Dominican, French
Canadian, Haitian, Korean, Lebanese,
Polish, Nigerian, Mexican, Taiwanese,
Ukrainian and so on.)

13. ¿En qué país nació el PADRE de esta
persona?

13.

En qué país nació [su PADRE / el
PADRE de PERSON’S NOMBRE]?

14. ¿En qué país nació la MADRE de esta
persona?

14.

En qué país nació [su MADRE /la
MADRE de PERSON’S NOMBRE]?

15. ¿Cuál es la ascendencia u origen étnico
de esta persona? (Por ejemplo: italiana,
jamaicana, africana americana,
camboyana, de Cabo Verde, noruega,
dominicana,franco-canadiense, haitiana,
coreana, libanesa, polaca, nigeriana,
mexicana, taiwanesa, ucraniana y así
por el estilo.)

15.

¿Cuál es la ascendencia u origen
étnico de usted? (Lea si es necesario:
Por ejemplo, italiano, jamaicano,
africano americano, camboyano, de
Cabo Verde, noruego, dominicano,
franco-canadiense, haitiano, coreano,
libanés, polaco, nigeriano, mexicano,
taiwanés, ucraniano y así por el
estilo.)

6. Parental Place of Birth: Summary of Findings and Recommendations

Version/
Language
Version 2
English

Self-administered

Interviewer-administered

13. Was this person’s FATHER born in or
outside the United States?
Born outside the United States – Print
name of foreign country, or Puerto Rico,
Guam, etc. __________________
14. Was this person’s MOTHER born in or
outside the United States?
Born outside the United States – Print
name of foreign country, or Puerto Rico,
Guam, etc. __________________

13a. Was your FATHER born in or outside
the United States?
13b. In what country was your FATHER
born?
14a. Was your MOTHER born in or outside
the United States?
14b. In what country was your MOTHER
born?
15.

15. What is this person’s ancestry or ethnic
origin? (For example: Italian, Jamaican,
African Am, Cambodian, Cape Verdean,
Norwegian, Dominican, French
Canadian, Haitian, Korean, Lebanese,
Polish, Nigerian, Mexican, Taiwanese,
Ukrainian and so on.)
Version 2
Spanish
(stateside
and
Puerto
Rico)

13.

What is your ancestry or ethnic
origin? (For example: Italian,
Jamaican, African-American,
Cambodian, Cape Verdean,
Norwegian, Dominican, French
Canadian, Haitian, Korean, Lebanese,
Polish, Nigerian, Mexican, Taiwanese,
Ukrainian and so on.)

¿Nació el PADRE de esta persona en
los Estados Unidos o fuera de los
Estados Unidos?

13a. ¿Nació su PADRE en los Estados
Unidos o fuera de los Estados Unidos?

¿Nació la MADRE de esta persona en
los Estados Unidos o fuera de los
Estados Unidos?

15.

13b. ¿En qué país nació su PADRE?
Nació fuera de los Estados Unidos –
14a. ¿Nació su MADRE en los Estados
Escriba en letra de molde el nombre del
Unidos o fuera de los Estados Unidos?
país extranjero o Puerto Rico, Guam, etc.
__________________
14b. ¿En qué país nació su MADRE
14.

Nació fuera de los Estados Unidos –
Escriba en letra de molde el nombre del
país extranjero o Puerto Rico, Guam, etc.
__________________
15.

¿Cuál es la ascendencia u origen
étnico de esta persona? (Por ejemplo:
italiana, jamaicana, africana americana,
camboyana, de Cabo Verde, noruega,
dominicana,franco-canadiense,
haitiana, coreana, libanesa, polaca,
nigeriana, mexicana, taiwanesa,
ucraniana y así por el estilo.)

6. Parental Place of Birth: Summary of Findings and Recommendations

Version/
Language
Version 3
English

Self-administered
13.

Interviewer-administered

Was this person’s FATHER born in the 13a. Was your FATHER born in the United
United States?
States?
No – Print name of foreign country, or
Puerto Rico, Guam, etc.
__________________

14.

Was this person’s MOTHER born in the
United States?
No – Print name of foreign country, or
Puerto Rico, Guam, etc.
__________________

15.

What is this person’s ancestry or
ethnic origin? (For example: Italian,
Jamaican, African Am, Cambodian,
Cape Verdean, Norwegian, Dominican,
French Canadian, Haitian, Korean,
Lebanese, Polish, Nigerian, Mexican,
Taiwanese, Ukrainian and so on.)

13b. In what country was your FATHER
born?
14a. Was your MOTHER born in the United
States?
14b. In what country was your MOTHER
born?
15.

What is your ancestry or ethnic origin?
(For example: Italian, Jamaican,
African-American, Cambodian, Cape
Verdean, Norwegian, Dominican,
French Canadian, Haitian, Korean,
Lebanese, Polish, Nigerian, Mexican,
Taiwanese, Ukrainian and so on.)

Version 3 13. ¿Nació el PADRE de esta persona en los 13a. ¿Nació su PADRE en los Estados
Estados Unidos?
Unidos?
Spanish
(stateside
No – Escriba en letra de molde el nombre del 13b. ¿En qué país nació su PADRE?
and
país extranjero o Puerto Rico, Guam, etc.
14a. ¿Nació [su MADRE / la MADRE de
Puerto
__________________
PERSON’S NOMBRE] en los Estados
Rico)
Unidos?
14. ¿Nació la MADRE de esta persona en los
Estados Unidos?
14b. ¿En qué país nació su MADRE?
No – Escriba en letra de molde el nombre del
15. ¿Cuál es la ascendencia u origen
país extranjero o Puerto Rico, Guam, etc.
étnico de usted? (Lea si es necesario:
__________________
Por ejemplo, italiano, jamaicano,
africano americano, camboyano, de
15. ¿Cuál es la ascendencia u origen étnico
Cabo Verde, noruego, dominicano,
de esta persona? (Por ejemplo: italiana,
franco-canadiense, haitiano, coreano,
jamaicana, africana americana,
libanés, polaco, nigeriano, mexicano,
camboyana, de Cabo Verde, noruega,
taiwanés, ucraniano y así por el estilo.)
dominicana,franco-canadiense, haitiana,
coreana, libanesa, polaca, nigeriana,
mexicana, taiwanesa, ucraniana y así por
el estilo.)

6.1

Findings from the Parental Place of Birth Module

Overview

Overall, respondents seemed to understand the Parental Place of Birth questions very
well and were able to answer them with very few issues. Of the 67 respondents, 61 were able to
answer the questions accurately and without difficulty. Of the six respondents (four in Spanish,
two in English) who had difficulty or seemed to answer the questions incorrectly, three
respondents answered “U.S.” instead of providing a U.S. territory. There was no indication that
this was related to the version of the questionnaire, as this occurred once in each of the three
versions. Two of these respondents were interviewed in Puerto Rico, and both were interviewerassisted cases. The interviewer-assisted interviews, in contrast to the self-administered
38

6. Parental Place of Birth: Summary of Findings and Recommendations

questionnaires, did not have specific instructions as to how to record U.S. territories, which may
have led to this error since the interviewers can only clarify if the respondent asks for assistance
before answering. The third case was an English language interview, and the respondent thought
that the territory was considered part of the U.S. However, once he re-read the self-administered
instructions, he changed his response to “outside the U.S.” and indicated that he was not paying
attention when he first answered.
Of the remaining three respondents who had difficulty or seemed to answer the Parental
Place of Birth question incorrectly, one respondent answered the Ethnic Origin question about
his father instead of himself. This was because he misinterpreted who the phrase “this person”
was referring to in the self-administered questionnaire. One respondent answered about her
adoptive parents because she was adopted at birth. She stated that she based her response on
“what the survey might want to know.” She did not consider that the question might be asking
about biological parents. The last of the six respondents who had difficulty indicated that she was
not 100% certain about her biological father’s place of birth but she was told by her adoptive
parents that he was from another country.
Nearly all respondents indicated that they knew where their parents were born or, when
answering for other household members, where those individuals’ parents were born. The few
instances of confusion mostly arose when the respondent simply did not have this information
(either because they were adopted or did not know the father of their biological child’s place of
birth). We would expect that situations with households with roommates or boarders might
exhibit more difficulty answering these questions, but there was only one case in the analysis that
mentioned a roommate situation and that person was able to answer the question about the
roommate’s parental place of birth and ethnicity.
Parents Born in U.S. Territory

One of the primary probes for this module was, “How do you think people who are from
U.S. territories, such as Puerto Rico or Guam, should answer this question? Should they put U.S.
or should they put something else?” Nearly all respondents said that this was an easy question to
answer, and indicated that they “just knew” where their parents were born. However, when
respondents were asked how they would report the place of birth had their parents been born in a
U.S. territory, responses were somewhat split. The majority responded that they would be
inclined to write in the name of the territory just for the sake of clarity, although some
respondents also indicated that they would consider the territory as part of the United States.
With this module, a particular concern was whether respondents whose parents were born
in U.S. territories would provide the name of the respective territories or simply say “U.S.”
Therefore, respondents were probed on this set of questions to determine how they thought
someone should answer if a parent was born in a territory. The majority of both English- and
Spanish-speaking respondents indicated that they would provide the name of the territory rather
than simply stating that their parents were born in the United States. Of the 29 English-speaking
respondents, 21 said they would provide the territory name. Of the 30 Spanish-speaking
respondents, 22 said they would provide the territory name. An additional eight Spanishspeaking respondents were not asked.

6. Parental Place of Birth: Summary of Findings and Recommendations

A total of 23 respondents actually had parents born in a U.S. territory (7 English, 16
Spanish interviews). All of the English-speaking respondents with parents born in a U.S. territory
ultimately gave a response of having parents born “outside the U.S.” and specified Puerto Rico.
Of the Spanish-speaking respondents, 14 out of 16 responded by specifying the name of the
territory. Two interviewer-administered respondents, whose parents were born in a U.S. territory,
answered that they were born in the United States; they considered Puerto Rico to be part of the
United States.
Additionally, the self-administered instrument’s instructions for this module included a
specific example as to how to handle a birthplace such as Puerto Rico and Guam. The
interviewer-administered versions did not contain this instruction and resulted in more confusion
as to whether or not a given territory should be specified. Only two respondents had problems
answering these questions in the self-administered versions. One had problems because she was
adopted at birth, and was unsure how to respond; the other respondent simply read the
instructions too quickly and answered in error. All other problems encountered in the selfadministered versions were issues with Question 15 regarding ethnic origin. Five respondents
reported issues with the interviewer-administered versions due to the fact that the question did
not clarify whether the U.S. territories should be included as part of the United States or
considered to be outside of the United States (with the other countries).
Among the interviews conducted in English, the majority of respondents for all
recruitment groups (i.e., parents born in the United States, parents born in a territory, parents
born outside the United States, and those who qualified for adopted/step/foster family) said they
would list the name of the territory. However, respondents whose parents were born outside of
the United States were the least likely to answer that way, with only 6 of 10 respondents
indicating that they would give the name of the territory. All six of the English-speaking
respondents with parents born in a U.S. territory ultimately responded as having parents born
“outside the U.S.” and specified Puerto Rico. However, two of the six expressed some confusion
when answering about their parents’ place of birth and initially answered that their parents were
born inside the United States.
Similarly, among the interviews conducted in Spanish, the majority of respondents for all
recruitment groups said they would list the name of the territory, or actually reported the territory
name. This was particularly true for respondents whose parents were born in a U.S. territory. Of
these respondents, 14 of 16 responded by specifying the name of the territory when asked for
their parent’s place of birth in the ACS question. The two Spanish-speakers who said that they
were born in the United States were both from Puerto Rico and indicated that Puerto Rico was
part of the United States. Because the intention is to have the respondents list U.S. territories as
being “outside the U.S.,” these two respondents were considered to have answered incorrectly.
English-speaking respondents who did not—or said they would not—provide the territory
name did so typically because they considered the territory to be part of the United States and
therefore considered an answer of “U.S.” to be accurate.
Issues Related Adopted, Foster, or Step Parents

Of the 13 respondents in this recruitment category, only one respondent answered for her
adoptive parents because she did not know her biological parents’ country of origin. Another 6
40

6. Parental Place of Birth: Summary of Findings and Recommendations

responded that they were thinking of their (or their stepchild’s, foster child’s, or adopted child’s)
biological parents. For the remaining 6 cases, it was not clear which parents were considered for
this question.
Ethnic Origin

Some additional confusion surrounded the idea of “ethnic origin.” The problems seem to
point to the use of the phrase “this person” when inquiring about ethnic origin. A different
wording was used in the interviewer administered versions, so this question did not present the
same problems in that mode. Additionally, the reference to “this person” appearing immediately
following questions about the parent’s place of birth caused some respondents to answer the
Ethnic Origin question about their parents’ heritage rather than their own.
When probed about who they were thinking about when they responded to this question,
respondents were divided between responding about themselves and responding about their
parents and family. Still a few other respondents indicated that they thought about their race or
the color of their skin when answering this question. Of the 31 individuals who completed the
self-administered versions, 6 respondents (4 English and 2 Spanish) had difficulty answering
Question 15; 24 respondents had no difficulty; and this probe was missing in the one remaining
interview. Overall, the terminology in the questions about ethnic origin was found to be easily
understood and straightforward. The answers and probes reveal this question was interpreted
equivalently by both Spanish-speakers and English-speakers.
Differences by Mode

Very few differences were noted between the modes (i.e., in-person intervieweradministered, phone interviewer-administered, and self-administered). Respondents across both
languages and across all eligibility qualifications seemed to be able to respond to the questions
with only a few problems, and these problems did not seem related to the interview mode, with
the exception that the self-administered version offered additional written instructions that
specified how to enter the information if respondents’ parents were born in Puerto Rico or Guam.
Respondents seemed to have fewer problems answering about parents born in a U.S. territory in
the self-administered version, due to the fact that there was an instruction indicating how to
record the responses in such situations.
Differences by Language

The only language-specific problems noted for the Spanish-speaking respondents were
with regard to the Ethnic Origin question (i.e., ACS Question 15), as noted previously, and the
phrase, “letra de molde,” or “print.” Respondents were asked the probe: “What did this mean to
Respondent: “print name of country, or Puerto Rico, Guam, etc.” / “Escriba en letra de molde el
nombre del país o Puerto Rico, Guam, etc.?” Although able to answer the question, some
respondents indicated they were unsure what the term meant for sure. When probed, some did
understand it to mean not to write in cursive (“letra pegada”). Others thought it mean to write in
all capital letters or to write in block letters. Because there was no evidence that respondents
were unable to answer the questions accurately because of the term “letra de molde” we do not
recommend a change. No other terminology was a specific problem in the parental place of birth
module in either English or Spanish

6. Parental Place of Birth: Summary of Findings and Recommendations

Appendix 12, Final Recommendations for Parental Place of Birth, provides additional
detail on these findings.
6.2
Version Preference and Recommendations for the Parental Place of Birth
Module
Fewer issues were noted with the Version 3 questionnaire than with the other versions.
However, many of the issues observed in Versions 1 and 2 were not specific to the wording in
those versions.
A majority of 23 respondents preferred Version 1; 19 respondents preferred Version 3,
and 18 respondents preferred Version 2. Another seven respondents noted that they had no
preference, claiming either that they did not see much difference between the versions or that
they would have no problem answering questions from any of the three, regardless of wording
and style differences. Although all three versions of the questions performed nearly equally for
the cases tested, with only two problem-cases reported for each (as previously documented in
Section 6.1), Version 3 seemed to be easier for respondents to answer accurately. Of the two
cases that were reported problems, one respondent said he was simply moving too quickly and
didn’t pay attention and the other respondent self-corrected before moving to the next question.
Respondents who said they preferred Version 3 felt that it would be easier to complete because
most respondents will have parents born in the United States. Other reasons cited for preferring
Version 3 were that is was more specific because it narrows down the choices, while the other
options were more complicated. Some respondents also favored the “Yes/No” design of the
question, as compared with the other versions.
A difference in preference was observed between English and Spanish speakers. The
majority of Spanish-speaking respondents preferred Version 1, while English speakers showed a
slight preference for Version 3. This is likely because most of the Spanish speakers we
interviewed had parents born in a foreign country, and these respondents find it is easier to
simply write in the name of the country, as opposed to having to check a box and then also write
in the name of the country. In contrast, English speakers prefer to quickly check a box if their
parents were born in the United States and to skip writing in the name of the country.
Because problems were observed in all versions of the questions, we recommend testing
revised questions for the Parental Place of Birth questions. The recommended versions include a
re-organization of the order of the questions, beginning by first asking about ethnic origin.
Additionally, the proposed test version gives the respondent three specific response options
instead of two. By including “in a U.S. territory” as a response option, the respondent would no
longer have to make the decision regarding whether to include that as part of the U.S. or not.
Table 6-2 shows the recommended versions to be tested in the field test, including an
unmodified Version 3 and an alternate wording that addresses the two major points of confusion
in the cognitive interviews.

6. Parental Place of Birth: Summary of Findings and Recommendations
Table 6-2. Proposed Versions for Parental Place of Birth and Ethnicity Questions
Language
Version 3
English

Self-administered

Interviewer-administered

13. Was this person’s FATHER born in the
United States?
No – Print name of foreign country, or
Puerto Rico, Guam, etc.
__________________
14. Was this person’s MOTHER born in
theUnited States?
No – Print name of foreign country, or
Puerto Rico, Guam, etc.
__________________

13a. Was your FATHER born in the United
States?
13b. In what country was your FATHER born?
14a. Was your MOTHER born in the United
States?
14b. In what country was your MOTHER born?
15.

What is your ancestry or ethnic origin?
(For example: Italian, Jamaican, AfricanAmerican, Cambodian, Cape Verdean,
Norwegian, Dominican, French Canadian,
Haitian, Korean, Lebanese, Polish,
Nigerian, Mexican, Taiwanese, Ukrainian
and so on.)

13a.

¿Nació su PADRE en los Estados Unidos?

13b.

¿En qué país nació su PADRE?

14a.

¿Nació [su MADRE / la MADRE de
PERSON’S NOMBRE] en los Estados
Unidos?

14b.

¿En qué país nació su MADRE?

15.

¿Cuál es la ascendencia u origen étnico
de usted? (Lea si es necesario: Por
ejemplo, italiano, jamaicano, africano
americano, camboyano, de Cabo Verde,
noruego, dominicano, franco-canadiense,
haitiano, coreano, libanés, polaco,
nigeriano, mexicano, taiwanés, ucraniano
y así por el estilo.)

15. What is this person’s ancestry or ethnic
origin? (For example: Italian, Jamaican,
African Am, Cambodian, Cape Verdean,
Norwegian, Dominican, French Canadian,
Haitian, Korean, Lebanese, Polish,
Nigerian, Mexican, Taiwanese, Ukrainian
and so on.)
Version 3
Spanish

13. ¿Nació el PADRE de esta persona en los
Estados Unidos?
No – Escriba en letra de molde el nombre
del país extranjero o Puerto Rico, Guam,
etc. __________________
14.

¿Nació la MADRE de esta persona en
los Estados Unidos?
No – Escriba en letra de molde el nombre
del país extranjero o Puerto Rico, Guam,
etc. __________________

6. Parental Place of Birth: Summary of Findings and Recommendations

Language
English
Wording
Alternate
Test
Version

Self-administered

Interviewer-administered

13.

What is this person’s ancestry or ethnic
origin?

13. What is (your / [NAME]’s) ancestry or ethnic
origin?

14.

Was this person’s father born in the
United States, in a U.S. territory, or
outside the United States? Mark (x) one
box.

14a. Was (your / [NAME]’s) father born in the
United States, in a U.S. territory, or outside
the United States?
In the United States – Go to Question 15a

In the United States

In a U.S. territory

In a U.S. territory – Print name of territory
(e.g., Puerto Rico, Guam): ____________
Outside the United States – Print name of
foreign country: __________________
15. Was this person’s mother born in the
United States, in a U.S. territory, or
outside the United States? Mark (x) one
box.

14b. In what (country/U.S. territory) was (your /
[NAME]’s) father born?
15a. Was (your / [NAME]’s) mother born in the
United States, in a U.S. territory, or outside
the United States?
In the United States – Go to Question 16

In the United States

In a U.S. territory

In a U.S. territory – Print name of territory
(e.g., Puerto Rico, Guam): ____________

Outside the United States

Outside the United States – Print name of
foreign country: __________________
Spanish
Wording
Alternate
Test
Version

Outside the United States

15b. In what (country/U.S. territory) was (your /
[NAME]’s) mother born?

13. ¿Cuál es la ascendencia u origen étnico
de esta persona?

13.

14. ¿Nació el padre de esta persona en los
Estados Unidos, en un territorio de los
EE.UU., o fuera de los Estados Unidos?
Marque (X) UNA casilla.

14a. ¿Nació (su/el) padre (de NAME) en los
Estados Unidos, en un territorio de los
EE.UU., o fuera de los Estados Unidos?

En los Estados Unidos

¿Cuál es (su/la) ascendencia u origen
étnico (de NAME)?

En los Estados Unidos – Go to 14
En un territorio de los EE.UU.

En un territorio de los EE.UU. – Escriba en
letra de molde el nombre del territorio (Por
ejemplo, Puerto Rico, Guam): __________
Fuera de los Estados Unidos – Escriba en
letra de molde el nombre del país
extranjero: _____________________
15. ¿Nació la madre de esta persona en los
Estados Unidos, en un territorio de los
EE.UU., o fuera de los Estados Unidos?
Marque (X) UNA casilla.

Fuera de los Estados Unidos
14b. ¿En qué (país/territorio de los EE.UU) nació
(su/el) padre (de NAME)?
15a. ¿Nació (su/la) madre (de NAME) en los
Estados Unidos, en un territorio de los
EE.UU., o fuera de los Estados Unidos?
En los Estados Unidos – Go to Question 14
En un territorio de los EE.UU.
Fuera de los Estados Unidos

En los Estados Unidos
En un territorio de los EE.UU. – Escriba en
letra de molde el nombre del territorio (Por
ejemplo, Puerto Rico, Guam):
___________________
Fuera de los Estados Unidos – Escriba en
letra de molde el nombre del país
extranjero: _______________

15b. ¿En qué (país/territorio de los EE.UU) nació
(su/la) madre (de NAME)?

7. Veteran Identification: Summary of Findings and Recommendations

Veteran Identification: Summary of Findings and
Recommendations

A total of 47 interviews were conducted using the Veterans Identification module. Three
versions of the Veterans Identification module were tested in both self-administered and
interviewer-administered modes. However, the interviewer-administered Version 1 and Version
2 questions were identical; they differed only in the self-administered questions. Of the 47
interviews conducted, 15 were conducted with Version 1, 16 with Version 2, and 16 with
Version 3.
Interviews were conducted in both English and Spanish. A total of 31 of 47 interviews
were conducted in English. Respondents were recruited to meet five different characteristics. Of
the 31 English respondents, 4 were current military personnel, 6 had been in the Reserves, 6 had
been in the National Guard, 7 were military veterans, and the remaining 8 respondents had a
household member who had been in the military.
A total of 16 of 47 interviews were conducted in Spanish. Because the ability to read and
write in English is required for U.S. military service, Spanish-speaking respondents were
recruited for different categories than were the English-speaking respondents. Ten respondents
with a household member in the military were recruited; two of these respondents had actually
served in the military themselves while living in Puerto Rico. The remaining six Spanishspeaking respondents were not in the military and did not have a household member in the
military.
See Table 7-1 for the question wording of each version tested.
Table 7-1. ACS Questions Tested for Veteran Identification
Version/
Language
Version 1
English

Self-administered

Interviewer-administered

28. Has this person ever served on ACTIVE
DUTY in the U.S. Armed Forces,
Reserves, or National Guard? Active
duty includes federal activation of the
Reserves or National Guard for service
in Iraq, Afghanistan, or elsewhere but
does NOT include Reserve or National
Guard training. Mark (X) ONE box.

28a.

(Has / Have you) ever
served on ACTIVE DUTY in the U.S.
Armed Forces, Reserves, or National
Guard? Active duty includes federal
activation of the Reserves or
National Guard for service in Iraq,
Afghanistan, or elsewhere but DOES
NOT include Reserve or National
Guard training.

28b.

(Are you / is ) now on
ACTIVE DUTY?

28c.

(Has / Have you) ever been
in the military Reserves or National
Guard?

Never served in the military
Now on active duty
On active duty in the past, but not now
Training for the Reserves or National Guard
only

7. Veteran Identification: Summary of Findings and Recommendations

Version/
Language
Version 1
Spanish
(stateside
and Puerto
Rico)

Self-administered

Interviewer-administered

28. ¿Ha estado esta persona alguna vez en el
SERVICIO MILITAR ACTIVO en las
Fuerzas Armadas, la Reserva Militar o la
Guardia Nacional de los Estados Unidos?
El servicio activo incluye activación
federal de la Reserva Militar o la Guardia
Nacional para servicio en Irak, Afganistán
o en otro lugar, pero NO incluye
entrenamiento para la Reserva Militar o
Guardia Nacional. Marque (X) UNA casilla.
Nunca estuvo en el servicio militar

28a.

28b.

¿Está usted ahora en SERVICIO
ACTIVO?

28c.

¿Ha estado usted alguna vez en la
Reserva Militar o la Guardia
Nacional?

En servicio activo ahora
En servicio activo en el pasado, pero no
ahora

¿Ha estado usted alguna vez en el
SERVICIO MILITAR ACTIVO en las
Fuerzas Armadas, la Reserva Militar o
la Guardia Nacional de los Estados
Unidos? El servicio activo incluye
activación federal de la Reserva Militar
o la Guardia Nacional para servicio en
Irak, Afganistán o en otro lugar, pero
NO incluye entrenamiento para la
Reserva Militar o Guardia Nacional.

Entrenamiento para la Reserva Militar o la
Guardia Nacional solamente
Version 2
English

28. What is this person’s ACTIVE DUTY
military status in the U.S. Armed Forces,
Reserves, or National Guard? Active duty
includes federal activation of the
Reserves or National Guard for service in
Iraq, Afghanistan, or elsewhere but does
NOT include Reserve or National Guard
training. Mark (X) ONE box.

For the interviewer-administered mode, Version 1
and Version 2 were identical.

Never served in the military
Now on active duty
On active duty in the past, but not now
Training for the Reserves or National Guard
only
Version 2
Spanish
(stateside
and Puerto
Rico)

28. ¿Cuál es el estatus del SERVICIO
MILITAR ACTIVO de esta persona en las
Fuerzas Armadas, la Reserva Militar o la
Guardia Nacional de los Estados Unidos?
El servicio activo incluye activación
federal de la Reserva Militar o la Guardia
Nacional para servicio en Irak, Afganistán
o en otro lugar, pero NO incluye
entrenamiento para la Reserva Militar o
Guardia Nacional. Marque (X) UNA casilla.
Nunca estuvo en el servicio militar
En servicio activo ahora
En servicio activo en el pasado, pero no
ahora
Entrenamiento para la Reserva Militar o la
Guardia Nacional solamente

For the interviewer-administered mode, Version 1
and Version 2 were identical.

7. Veteran Identification: Summary of Findings and Recommendations

Version/
Language
Version 3
English

Self-administered

Interviewer-administered

28. Has this person ever served on active
duty in the U.S. Armed Forces, military
Reserves, or National Guard? Active duty
does NOT include training for the
Reserves or National Guard, but DOES
include activation, for example, for
service in Iraq, Afghanistan, or elsewhere.

28a. (Has / Have you) ever served on
active duty in the U.S. Armed Forces,
military Reserves, or National Guard?
Do not include training for the
Reserves or National Guard, but do
include activation, for example, for
service in Iraq, Afghanistan, or
elsewhere.

No, never served in the military
Yes, on active duty in the past, but not now
No, training for the Reserves or National
Guard only

28b.

(Are you / is ) currently on
active duty?

28c.

(Have you / has ) ever been in
the U.S. military Reserves or the
National Guard?

28a.

¿Ha estado usted alguna vez en el
servicio militar activo en las Fuerzas
Armadas, la Reserva Militar o la
Guardia Nacional de los Estados
Unidos? No incluya entrenamiento
para la Reserva Militar o Guardia
Nacional, pero sí incluya activación
federal para servicio en Irak,
Afganistán o en otro lugar.

28b.

¿Está usted ahora en servicio militar
activo?

Yes, now on active duty
Version 3
Spanish
(stateside
and Puerto
Rico)

28. ¿Ha estado esta persona alguna vez en el
servicio militar activo en las Fuerzas
Armadas, la Reserva militar o la Guardia
Nacional de los Estados Unidos? El
servicio activo NO incluye entrenamiento
para la Reserva Militar o la Guardia
Nacional, pero SÍ incluye activación, por
ejemplo, para servicio en Irak, Afganistán
o en otro lugar.
Sí, en servicio activo ahora
Sí, en servicio activo en el pasado, pero no
ahora
No, entrenamiento para la Reserva Militar o
la Guardia Nacional solamente

28c. ¿Ha estado usted alguna vez en la
Reserva Militar o la Guardia Nacional
de los Estados Unidos?

No, nunca estuvo en el servicio militar

7.1

Findings from the Veteran Identification Module

Overview

The purpose of this question is to determine a respondent’s veteran status. A key
distinction for determining veteran status for individuals who are in the Reserves or National
Guard is determining whether that person had been activated or whether they received training
only. Consequently, Versions 1 and 2 of the questions included the following clarification:
Active duty includes federal activation of the Reserves or National Guard for
service in Iraq, Afghanistan, or elsewhere but DOES NOT include Reserve or
National Guard training.
Version 3 included a similar clarification:
Do not include training for the Reserves or National Guard, but do include
activation, for example, for service in Iraq, Afghanistan, or elsewhere.

7. Veteran Identification: Summary of Findings and Recommendations

Of the 47 respondents, 32 understood the question as intended and answered accurately.
Sixteen respondents demonstrated some type of confusion or misunderstanding with the
question. Of these 16 respondents, only 2 answered the question inaccurately for their situations.
Both respondents answered that they were on active duty when they had only been in training for
the National Guard. The remaining 14 had some type of difficulty with the question but were
able to answer accurately for their situations. Most of the errors and confusion that occurred was
due to the clarification phrase provided in all versions of the question. The specific problems
with this question are discussed in more detail below.
Active Duty

Respondents were probed on their understanding of “active duty” or “servicio militar
activo” as used in the survey question. Out of the 47 respondents, 13 (8 English and 5 Spanish)
indicated during probing that training was active duty, which contradicts the clarification
statement. Despite the confusion, most answered accurately because they had actually served on
active duty. Had they only participated in training they may have considered this active duty and
may have answered the question inaccurately. Five of the recruited respondents had been in the
National Guard, six had been in the Reserves and one respondent had a son who was in the
Reserves. Of these 12 respondents, six had been activated at one point and provided an answer to
the ACS question that confirmed that. The other six had not been activated yet. Four out of the
six answered accurately by indicating that they had been in training only or had not served in
active duty military. Two respondents, one who was in the Reserves and one whose son was in
the Reserves, both answered that this person had been activated. During probing, it was revealed
that these two people had not been activated and had only received training for the Reserves, and
thus answered this question incorrectly.
“Federal Activation” and “Activation”

The clarification phrase used in this question also contained the phrase “federal
activation” (activación federal) in Versions 1 and 2 and “activation” (activación) in Version 3.
Respondents who were not in the Reserves or National Guard were asked if they were familiar
with the term and how they understood it. Of the 29 respondents who received this probe, many
were not familiar with the term, but only 5 could not understand what it meant as used in the
question. None of these 5 respondents were in the military themselves. Three were Spanishspeaking respondents, one of which said that it referred to helping other countries by building
houses, bridges, or highways—going to another country to help rather than to make war.
Respondents who were in the Reserves or the National Guard were asked about the
difference between “state” and “federal” activation and their understanding of the term in
general. Of the 5 National Guard participants, 4 understood the difference. One participant, did
not understand the term “activation” in general; however, he had difficulty with the entire
cognitive interview process. Of the 6 Reservists, all understood the term “activation” but only 3
knew the difference between state and federal activation. The other three were unsure of the
difference, although they knew what “federal activation” meant; however, because “state
activation” applies only to the National Guard and not to the Reserves, this lack of distinction
seemed unproblematic.

7. Veteran Identification: Summary of Findings and Recommendations

Although the respondents unfamiliar with the term were evenly split among the three
versions of the instrument, the responses to the probe for Version 3 were more precise and
confident than the responses for the other two versions.
In addition, participants who understood the term “activation” frequently used other
terms to describe activation, such as “called up,” “called to serve,” “deployed,” “deployment,”
“mobilize” or “mobilization.”
Service in “Iraq, Afghanistan, or Elsewhere”

Another problem with the clarification instruction was that 4 out of the 47 respondents
focused on the locations, “Iraq, Afghanistan, or elsewhere,” that are listed in the question. All
four respondents were English-speakers. These respondents thought that the question was asking
just about those specific locations and did not realize that they were just examples. All four
respondents who reported such problems were using Versions 1 and 2. Version 3 includes the
phrase “for example” before listing the locations, which may have helped to abate the problem in
that version.
“Reserves” versus “Military Reserves”

The Census Bureau was also concerned about the phrase “military Reserves/La Reserva
Militar” versus “Reserves/La Reserva” and wanted to know which phrase was more clearly
understood and preferred by respondents. The English-speaking respondents were fairly evenly
split on their preferences. Both the Reserves and the National Guard respondents had a slight
preference for “Reserves,” whereas the remaining English-speaking respondents preferred
“military Reserves.” When asked why they preferred “Reserves,” those respondents said that
they thought people would understand it, and that “military Reserves” was redundant. When
asked why they preferred “military Reserves,” those respondents reported that there are other
types of reserves, such as police reserves, and therefore “military Reserves” would be more
clear.
Spanish-speaking respondents emphatically preferred “La Reserva Militar.” Thirteen
respondents preferred “La Reserva Militar” compared with one who preferred “La Reservas.”
Respondents indicated that the phrase “la Reserva” has several different meanings in Spanish,
including “reservations.”
Differences by Mode

The use of the clarification instruction was more problematic in the self-administered
version of the instrument because the clarification was provided in italics. Five of the 22
respondents who received the self-administered survey said they did not read it. Reading the
clarification changed two respondents’ understanding of the question. Additional respondents did
read the clarification but did not fully understand it, primarily because they considered training
to be active duty.
The interviews conducted with the self-administered surveys elicited an additional
problem related to the issue of training and active duty. The self-administered mode listed four
response options, which varied slightly in Version 3: Never served in the military; Now on active
duty; On active duty in the past, but not now; Training for the Reserves or National Guard only.

7. Veteran Identification: Summary of Findings and Recommendations

The interviewer-administered versions were phrased as “yes” or “no” questions. As a result,
respondents in the self-administered mode noted some confusion between the category “Training
for the Reserves or National Guard” compared with either of the active duty categories, “Now on
active duty” or “On active duty in the past, but not now.” Although this problem was more
evident in the self-administered version, it would most likely be just as problematic in the
interviewer-administered version. Respondents who received training only were unsure how to
answer. It was unclear to them whether they should indicate that they were on active duty or not.
Differences by Language

Overall, there were fewer issues with the Spanish-speaking respondents because the
questions did not apply to the majority of them directly. Only 2 of the 16 Spanish-speaking
respondents had actually been in the military themselves; therefore, they were often not as
knowledgeable about the question subtleties that were confusing for some of the English
respondents. Only one respondent misunderstood the question completely and thought that it was
asking about the military helping other countries in need (as opposed to war or other military
operations). Spanish respondents did not appear to have any difficulties with the translation of
the question with the exception of the use of “La Reservas” as noted above.
Differences by Version

Overall, there were not any issues that could be attributed with certainty to the version of
the question. In particular, Versions 1 and 2 were very similar in wording and only differed in
the self-administered version.
Although two respondents answered incorrectly in Version 3, and none answered
incorrectly in Versions 1 or 2, the potential to answer incorrectly existed in all versions. A
respondent who received Version 1 and a respondent who received Version 2 misunderstood
“active duty” to include their training, but both of these respondents had also been activated.
Consequently, their answers, that they had been on active duty in the past, were correct. If they
had not been activated, they likely would have answered erroneously in the way that the
respondents in Version 3 did. In all three versions, respondents expressed confusion over how to
treat training and whether it should be considered “active duty” for this question. This was
particularly problematic for respondents who are in or were in the Reserves or National Guard.
When asked which version they preferred, 22 of 47 respondents chose Version 3. This
choice was fairly consistent across mode and language, with the exception of self-administered
Spanish, for which 3 out of 6 respondents chose Version 1.
When asked why they preferred the version they selected, most said that it was simpler,
clearer, or easier to understand. For Version 3, there were 5 respondents who specifically said
that they preferred the way the “include / do not include” information was presented. In this
version, the “do not include” information was before the “include” information; the sequence is
opposite in Versions 1 and 2.
Two respondents preferred Version 2 in self-administered mode because this version
asked, “What is this person’s active duty status,” instead of “Has this person ever served on
active duty?” However, two other respondents particularly disliked Version 2 for the same

7. Veteran Identification: Summary of Findings and Recommendations

reason. The additional reasons that respondents gave for preferring Version 1 or 2 were
inconsistent and not necessarily specific to these versions.
Appendix 13, Final Recommendations for Veteran Identification, provides additional
detail on the findings from this section.
7.2
Version Preference and Recommendations for the Veteran Identification
Module
We recommend using a revised Veterans Identification question in the field test because
Version 3, the control, was preferred over either Versions 1 and 2. In addition, it was preferred
for reasons that were pertinent and convincing. Few differences were observed between Versions
1 and 2 and those versions presented issues similar to those emerging from Version 3. Specific
recommendations are as follows:
§

Respondents noted that the question was confusing because it says that “active duty”
does not include training, yet respondents noted that training is active duty. In addition,
the question says to exclude people who only had training, but only being in training is a
response option. We recommend excluding the clarification phrase because it does not
appear to help. If you must include the clarification phrase, we recommend that instead of
clarifying “active duty,” we tell respondents specifically what to exclude (e.g., “Do NOT
include active duty for training”) before indicating what to include.

Some respondents focused on the locations “Iraq, Afghanistan, or elsewhere” and were
not thinking of the United States or other locations overseas. We recommend changing
the location to refer to “the United States or overseas.”

Although most respondents seemed to understand the term “activation,” they frequently
used other terms, including “called up,” “mobilization,” and “deployment.” We
recommend using these additional terms to ensure that the question is understood by as
many respondents as possible.

Several respondents did not read the clarification for active duty because they felt
confident that they knew what “active duty” meant. To increase the likelihood that the
instruction will be read, we recommend that it not be italicized but that it instead use the
same typeface as the rest of the question.

Particularly in Spanish, respondents preferred the term “military Reserves” to
“Reserves.” We recommend using the phrase “la Reserva Militar” in all instances for the
Spanish question. In Version 3 of the English question, the first reference is to “military
Reserves,” and the remaining references are to “Reserves.” We recommend preserving
this approach because it clarifies the term “Reserves” upon introduction but then shortens
it for the rest of the question.

We recommend changing the response options for the self-administered questionnaire in
the following ways:
§

Change the option, “no, training for the Reserves or National Guard only,” to “only
active duty for training (ADT) for the National Guard or Reserves.” Placing the “only
active duty for training (ADT) for the National Guard or Reserves” option before either

7. Veteran Identification: Summary of Findings and Recommendations

of the “active duty” options also may encourage more respondents to appropriately select
the “active duty training” response. Please note that the phrase “active duty for training”
and the acronym “ADT” are English-specific phrases without an equivalent in Spanish.
The phrase can be translated, but the acronym is not meaningful in Spanish and thus was
not included.
§

Change the sequence of response options by placing the two “no” responses together,
followed by the two “yes” responses, but remove the “yes” and “no” because they may
encourage respondents to skip reading some of the options.

Table 7-2 shows the recommended alternate wording that can be tested against the
control (Version 3).
Table 7-2. Proposed Alternate Wordings for the Veteran-Identification Question
Language

Self-administered

Interviewer-administered

English
Wording

28. Has this person ever served on active
duty in the U.S. Armed Forces, military
Reserves, or National Guard? Do NOT
include active duty for training (ADT)
for the Reserves or National Guard,
but DO include activation,
mobilization, or deployment for
service in the U.S. or overseas. Mark
(X) ONE box.

28a.(Has / Have you) ever served on
active duty in the U.S. Armed Forces,
military Reserves, or National Guard? Do
not include active duty training for the
Reserves or National Guard, but do
include activation, mobilization, or
deployment for service in the United
States or overseas.

Never served in the military
Only active duty for training (ADT) for
the Reserves or National Guard

28b.

Are you currently on active duty?

28c.Have you ever been in the U.S. military
Reserves or the National Guard?

On active duty in the past, but not now
Now on active duty
Spanish
Wording

28. ¿Ha estado esta persona alguna vez
en el servicio militar activo en las
Fuerzas Armadas, la Reserva Militar o
la Guardia Nacional de los Estados
Unidos? NO incluya servicio activo
para entrenamiento (ADT, por sus
siglas en inglés) para la Reserva
Militar o la Guardia Nacional, pero
INCLUYA activación, movilización o
despliegue para servicio en los EE.UU.
o en el extranjero. Marque (X) UNA
casilla.

28a.

¿Ha estado ( / you) served
on active duty, even if it was just for part
of the period. / Did ( / you) serve
on active duty during:]

Version 3
Spanish
(stateside
and PR)

September 2001 or later

August 1990 to August 2001 (including
Persian Gulf War)

September 1980 to July 1990

May 1975 to August 1980

Vietnam Era (August 1964 to April 1975)

March 1961 to July 1964

February 1955 to July 1961

Korean War (July 1950 to January 1955)

January 1947 to June 1950

World War II (December 1941 to December
1946)

November 1941 or earlier

29. ¿Cuándo estuvo esta persona en servicio
activo en las Fuerzas Armadas de los
Estados Unidos? Marque (X) una casilla
por CADA período durante el cual esta
persona estuvo en servicio militar,
aunque fuera sólo por parte del período.
Septiembre del 2001 ó después
Agosto del 1990 a agosto del 2001
(incluyendo la Guerra del Golfo Pérsico)
Septiembre del 1980 a julio del 1990

29. [Usando la Tarjeta A, por favor, dígame
cada período en el cual usted estuvo en
servicio activo, aunque sólo fuese parte
del período. / ¿Estuvo usted en servicio
militar activo durante:]
Septiembre del 2001 ó después
Agosto del 1990 a agosto del 2001
(incluyendo la Guerra del Golfo Pérsico)
Septiembre del 1980 a julio del 1990
Mayo del 1975 a agosto del 1980

Mayo del 1975 a agosto del 1980
Época de Vietnam (agosto del 1964 a abril
del 1975)
Marzo del 1961 a julio del 1964

Época de Vietnam (agosto del 1964 a abril
del 1975)
Marzo del 1961 a julio del 1964
Febrero del 1955 a febrero del 1961

La Guerra de Corea (julio del 1950 a enero
del 1955)

Segunda Guerra Mundial (diciembre del
1941 a diciembre del 1946)

Enero del 1947 a junio del 1950

Febrero del 1955 a febrero del 1961

Segunda Guerra Mundial (diciembre del
1941 a diciembre del 1946)

Enero del 1947 a junio del 1950

Noviembre del 1941 ó antes

8. Veteran Period of Service: Summary of Findings and Recommendations

8.1

Findings from the Veterans Period of Service Module

Of the 47 respondents, 38 understood the question as intended and answered accurately. Nine
respondents demonstrated some type of confusion or misunderstanding with the question. Of these,
five answered incorrectly because they included times when they were in training only; two answered
incorrectly because they could not accurately recall the dates; and two answered incorrectly because
they did not review the answer choices thoroughly enough before answering.
Overall, respondents reacted very favorably toward the response categories. One respondent
especially liked that the response categories included “nonwar” periods. She contrasted these answer
options with the typical options she sees that cover only war periods: “Again because it’s kind of hard,
because my time period, when I was in, no one really pays much attention to that time period. They
only mark times where actually the country’s at war.” Respondents also liked that the categories
showed dates and labels.
Nonetheless, two respondents initially answered incorrectly because they focused on the “era”
rather than on the dates. During probing, they realized that years for another category fit as well. In
addition, most respondents indicated that recalling when they or a household member served was easy;
however, two respondents did have difficulty recalling the specific dates of service. For one respondent
this was because he got out of the service over 40 years ago. The other respondent was answering for
her spouse and just could not recall whether he got out of the service before or after “September 2001,”
the more recent response category.
Although the response categories were not problematic for the remaining respondents, two
respondents found that the presentation of the dates and time periods was somewhat confusing and
suggested that they be better organized, for example, by listing the dates first and then the eras in
parentheses. Currently, for some options, the date is first; for other options, the era is first.
A consistent problem for the Veteran Identification question (Section 7) was whether training
was considered active duty; this was also an issue for the Veteran Period of Service question. Five
respondents indicated that they did not exclude time spent in training only. The first three participants
were in the Reserves or National Guard (or answering about someone who was) and included the time
the person was in the Reserves or National Guard, even when the person was not on active duty.
Several other respondents indicated during probing that they were thinking about their time in training
when they answered this question, but it was during a time period when they were also active.
Overall, there were not any issues that could be attributed with certainty to the version of the
question. Versions 1 and 2 of this question were identical. Version 3 was very similar to Versions 1
and 2, except that Version 3 contained 11 timer periods in the response options instead of 9. Few
problems were identified with this question across all versions, and those problems that did occur (e.g.,
including time spent while in training) were observed in all versions of the question.
Similarly no issues were observed with regard to language. As mentioned previously, because
the U.S. military service requires the ability to speak and read English, many of the Spanish-speaking
respondents did not have to answer this question because they were not in the service themselves.
However, all respondents were shown this question and none of the Spanish-speaking respondents
exhibited in difficulties in understanding the question as it was written.
Appendix 14, Final Recommendations for Veteran Period of Service, provides additional detail
on these findings.
56

8. Veteran Period of Service: Summary of Findings and Recommendations

8.2
Version Preference and Recommendations for the Veterans Period of Service
Module
Although most respondents had no difficulty with any version of this question, like the Veteran
Identification question, it is unclear as to how to treat time spent in training. For regular military
respondents, it was unclear whether basic training or other periods of training should be included. This
issue is not a major one, however, because basic training or other training periods tend to be brief. It
was a more significant issue for Reserves and National Guard respondents because they did serve on
active duty. While none of the respondents voiced confusion over whether they should include training
or not, respondents were inconsistent in whether they included or excluded training. It was unclear
whether their active duty training should be included as “active duty.” Currently, the question does not
instruct the respondent to exclude any training. If it is important that they exclude time spent in
training, we suggest adding an instruction to exclude training for the Reserves or National Guard. The
results of the field test can be used to determine whether fewer response options are chosen for the
question with the instruction than for the control, Version 3.
Although the majority of respondents reacted very favorably toward the response options, two
respondents suggested that the response options be open-ended. Two other respondents found the
categories somewhat confusing because the dates and categories were blended. For example, some
options have the date first, and others have the era first. In addition, two respondents initially failed to
select all of the periods that applied because they focused on the “era” and did not review the dates.
Consequently, we also recommend that all response options be listed with the dates first and the era in
parentheses (Table 8-2).
Table 8-2. Proposed Alternate Wordings for the Veteran-Identification Question
Language

Self-administered

Interviewer-administered

English
Wording

29. When did this person serve on active duty
in the U.S. Armed Forces? Do not include
time spent in training for the Reserves or
National Guard. Mark (X) a box for EACH
period in which this person served, even if
he or she served just for part of the period.

29. [Using Card A, please tell me each period in
which ( / you) served on active duty,
even if it was just for part of the period. / Did
( / you) serve on active duty during
any of the following periods?]

September 2001 or later
August 1990 to August 2001 (including
Persian Gulf War)

Do not include time spent in training for the
military Reserves or National Guard.
September 2001 or later

May 1975 to July 1990

August 1990 to August 2001 (including Persian
Gulf War)

August 1964 to April 1975 (Vietnam Era)

May 1975 to July 1990

February 1955 to July 1964

August 1964 to April 1975 (Vietnam Era)

July 1950 to January 1955 (Korean War)

February 1955 to July 1964

January 1947 to June 1950

July 1950 to January 1955 (Korean War)

December 1941 to December 1946 (World
War II)

January 1947 to June 1950

November 1941 or earlier

December 1941 to December 1946 (World War
II)
November 1941 or earlier

8. Veteran Period of Service: Summary of Findings and Recommendations
Language

Self-administered

Interviewer-administered

Spanish
Wording

29. ¿Cuándo estuvo esta persona en servicio
militar activo en las Fuerzas Armadas de
los Estados Unidos? No incluya el tiempo
que haya pasadoen entrenamiento para la
Reserva Militar o la Guradia Nacional.
Marque (X) una casilla para CADA periodo
durante el cual esta persona estuvo en
servicio militar, aunque fuera sólo por
parte del periodo.

29. [Usando la Tarjeta A, dígame en cuál periodo
estuvo (/usted) en servicio activo,
aunque fuera sólo por parte del período./¿Ha
estado (/usted) en servicio militar
activo durante alguno de estos periodos?]
No incluya el tiempo que haya pasado en
entrenamiento para la Reserva Militar o la
Guardia Nacional

Septiembre del 2001 ó después

Agosto de 1990 a agosto del 2001
(incluyendo la Guerra del Golfo Pérsico)

Agosto de 1990 a agosto del 2001 (incluyendo
la Guerra del Golfo Pérsico)

Mayo de 1975 a julio de 1990

Agosto de 1964 a abril de 1975 (Época de
Vietnam)

Febrero de 1955 a julio de 1964

Julio de 1950 a enero de 1955 (Guerra de
Corea)

Enero de 1947 a junio de 1950

Diciembre de 1941 a diciembre de 1946
(Segunda Guerra Mundial)

Noviembre de 1941 ó antes

9. Wages and Salary: Summary of Findings and Recommendations

Wages and Salary: Summary of Findings and Recommendations

Two versions of the Wages and Salary module were tested in interviewer-administered
mode only. A total of 31 interviews were conducted using the Wages and Salary module. Of the
31 interviews, 14 interviews were completed with Version 1, and 17 interviews were completed
using Version 2. Interviews were conducted in both English and Spanish. Of the 31 interviews,
13 were conducted in English and 18 were conducted in Spanish.
Respondents were recruited to meet four different characteristics. Of the 31 interviews, 3
English and 3 Spanish interviews were with respondents whose earned income includes at least
10% in tips; 4 English and 2 Spanish interviews were with respondents whose earned income
includes at least 10% in bonuses or commissions; 5 English and 6 Spanish interviews were with
respondents with multiple jobs but no self-employment; 3 English and 4 Spanish interviews were
with respondents with multiple jobs with self-employment; and the remaining 2 English and 5
Spanish respondents had none of the above characteristics.
Table 9-1 shows the question wording of each version tested.
Table 9-1. ACS Questions Tested for Wages and Salary
Version/
Language

Selfadministered

Version 1
English

Selfadministered
Version 1 was
not tested

Interviewer-administered
The next few questions are about income DURING THE PAST 12
MONTHS
49a1a. Did you receive any wages or salary?
Yes
No à Skip to Q49a2a
49a1b. How much did you receive from all jobs before taxes
and other deductions?
49 a2a. Did you receive any additional tips, bonuses, or
commissions DURING THE PAST 12 MONTHS?
Yes
No à Skip to Current Q47b
49a2b. How much did you receive from all jobs before taxes
and other deductions?

9. Wages and Salary: Summary of Findings and Recommendations

Version/
Language

Selfadministered

Version 1
Spanish
(stateside and
Puerto Rico)

Selfadministered
Version 1 was
not tested

Interviewer-administered
Las siguientes preguntas son sobre ingreso DURANTE LOS
ÚLTIMOS 12 MESES
49a1a. ¿Recibió usted algún jornal, sueldo o salario?
Yes
No à Skip to Q49a2a
49a1b. ¿Cuánto recibió usted de todos los empleos antes de
impuestos y otras deducciones?
49a2a. ¿Recibió usted alguna propina, bonificación o
comisión adicional DURANTE LOS ÚLTIMOS 12
MESES?
Yes
No à Skip to Current Q47b
49a2b. ¿Cuánto recibió usted de todos los empleos antes de
impuestos y otras deducciones?

Version 2
English

Selfadministered
Version 2 was
not tested

The next few questions are about income DURING THE PAST 12
MONTHS
49a1. Did you receive any wages, salary, tips, bonuses, or
commissions?
Yes
No à Skip to Current Q49b
49a2.

Version 2
Spanish
(stateside and
Puerto Rico)

Selfadministered
Version 2 was
not tested

How much did you receive before taxes and other
deductions?

Las siguientes preguntas son sobre ingreso DURANTE LOS
ÚLTIMOS 12 MESES
49a1. ¿Recibió usted algún jornal, sueldo o salario,
propina, bonificación o comisión?
Yes
No à Skip to Current Q49b
49a2.

9.1

¿Cuánto recibió usted antes de impuestos y otras
deducciones?

Findings from the Wages and Salary Module

Overview

The purpose of this sequence of questions is to elicit earned income, whether this income
is coming from wages, salary, tips, bonuses, or commissions. In Version 1, this information is
elicited by first asking about whether the person has received any wages or salary income; if the
answer is “yes,” the follow-up question asks how much. This is followed by a question on tips,
bonuses, or commissions; if the answer is “yes,” the follow-up question asks for the amount
received. In contrast, Version 2 asks about wages, salary, bonuses, tips, or commissions in a
single question. For those who report having received any such earnings, the follow-up question
asks for the amount received.

9. Wages and Salary: Summary of Findings and Recommendations
Single versus Multiple Jobs and Self-employment Income

Respondents who had only one source of income from a regular paycheck did not appear
to have any difficulty deciding what to include, regardless of version. However, some
respondents had a broader interpretation and included bonus, tip, or commission income under
wages and salary.
Across versions, there was a general tendency among respondents to report what they
considered substantial or important, and there was also a tendency among respondents not to
report smaller amounts or earned income from nonregular jobs. These respondents did not
necessarily think of themselves as self-employed when doing small or odd jobs. Only 5 of the 11
respondents screened and recruited as having multiple jobs (without self-employment) in the past
year verified through their answers that they had actually held more than one job at the same
time. Among those five respondents, three included the wages and salary from all of their jobs;
the remaining two did not. One of the respondents reported income from her main job only
because, as she explained, the series of questions preceding question 49a1 focused on last week’s
job, and that form of questioning had made her focus only on her main job. The other respondent
also reported only the wages from his “basic” employment. Of the seven respondents screened as
having multiple jobs and also engaging in self-employment, six confirmed self-employment and
a job. Of these six respondents, five reported their job income, including the self-employment
income, collectively under “wages and salary.” This is problematic because there is a separate
self-employment question later, and there can be double reporting of the self-employment
earnings.
The “wages and salary” question precedes the question about self-employment. As a
result, at least three respondents who were self-employed reported their self-employment income
in this section. All three respondents also reported their income under the self-employment
questions in the ACS survey, effectively double-counting their income.
Reference Period

All but a handful of respondents felt it would be much easier to report their earned
income if interviewed soon after tax-return preparation. This suggests that respondents were
thinking of—and possibly reporting about—the prior calendar year rather than the past 12
months, which was confirmed in the probes as we indicate below. Only a few indicated that they
calculated their answers by adding or multiplying monthly or hourly amounts. In addition,
almost all respondents indicated that it would be easier to report gross income before deductions.
This was particularly true of respondents who received a regular paycheck. As one respondent
commented, the withholdings change from year to year and make it difficult to recall the exact
amount of the paychecks, making gross income easier to report. However, a few respondents did
indicate that it would be easier to report pretax earnings.
For both versions of these questions, respondents were not thinking of the correct
reference period (i.e., the past 12 months) when they provided their responses. Of the 14
respondents in Version 1, only 5 (1 in English and 4 in Spanish) seemed to use the correct
reference period. In two other English-language cases, it is not clear whether the respondents
were relying on their tax returns or not. A third case refused to answer the wages and salary
question. Of the 17 respondents in Version 2, only 2 (1 in English and 1 in Spanish) clearly kept

9. Wages and Salary: Summary of Findings and Recommendations

the reference period in mind. In three other cases, it is not clear whether the respondents were
relying on their tax returns or not. When respondents answered incorrectly, they tended to either
report for only part of the reference period, report for the 2008 calendar year instead of the past
12 months, or use their current salary information to create an estimate for the past 12 months.
Differences by Language

The questions tested in this module were very simply worded in English. There were no
terminology issues with English respondents, and the same was the case with Spanish speakers.
Terms such as “salary,” “wages,” “bonus,” “tips,” “commissions,” “taxes,” and “deductions” are
clear concepts that respondents did not exhibit problems with, whether or not they received them.
The same was true in Spanish for the translated terms (jornal, sueldo salario, bonos, propinas,
comisiones, impuestos, deducciones).
Five of six respondents who had different types of problems that made them misreport
the amounts earned or report them under the wrong category of earnings were Spanish speakers.
Nonetheless, the probing did not reveal any reasons why the Spanish versions of the questions
could have created more difficulties than the English.
Spanish-speaking respondents expressed a stronger preference for one version of the
questions (Version 2) than English speakers. However, as discussed below, we believe other
demographic characteristics may have been responsible for this preference, rather than a
language version or translation issue.
Differences by Version

While the English-language respondents divided almost evenly in their preference for one
version over the other, about two-thirds of the Spanish-language respondents preferred Version
2. Because the Spanish-language respondents typically have lower education levels than the
English-language respondents, it is possible that their preference is based on the shorter sequence
of text to listen to and process. While each group felt that the version it preferred was simpler
and clearer, those who preferred Version 2 particularly liked the fact that it consisted of a shorter
sequence of questions. Some felt that the early mention of bonuses, commissions, or tips might
have helped them remember to report these types of income. Those who preferred Version 1
particularly liked the fact that the sequence of questions asked about different types of earnings
separately. The respondents who received bonuses also had a preference for Version 1.
Of the 14 respondents in Version 1, there were 8 (5 in Spanish and 3 in English) who
seemed to understand the questions as intended and to answer them correctly. Five respondents
(1 in Spanish, 4 in English) answered the questions inaccurately, and the remaining respondent
refused to answer. Those with salary or wages only were more likely to understand the question.
While no particular demographic patterns seemed to be associated with the respondents who had
problems, the problems all occurred with respondents who had more complicated earnings than
just a salary or regular wages. The problems detected included difficulty recalling earnings,
reporting commissions or bonuses under wages and salary, double reporting of earnings, and
earnings entirely left out.
One respondent was confused by the fact that the follow-up questions to both the wages
and salary and the bonus/tips/commission questions were worded identically. Another
62

9. Wages and Salary: Summary of Findings and Recommendations

respondent, who had no wages or salary to report, was confused by the word “additional” in the
bonus/tips/commission question.
Of the 17 respondents in Version 2, there were 11 (5 in English and 6 in Spanish) who
seemed to understand the questions as intended and to answer them correctly. No particular
demographic characteristics were observed in this group. The group did not include more of any
specific type of income recipient. Six respondents (5 in Spanish and 1 in English) had different
types of problems that made them misreport the amounts earned or report them under the wrong
category of earnings. Although almost all were Spanish speakers, it is unclear why the Spanish
version of the question could have created more difficulties than the English. No other particular
demographic patterns seemed to be associated with these problems. The types of problems
detected were similar to those reported for Version 1. In addition, the combination of tips,
bonuses, and commissions with wages and salary made some respondents selectively hear only
some of these types of income. For instance, two respondents recalled hearing only the question
asked about tips.
Appendix 15, Final Recommendations for Wages and Salary, provides additional detail
on the findings from this section.
9.2
Version Preference and Recommendations for the Wages and Salary
Module
Because the preference for one version over the other was not so strong, and the reasons
provided were not persuasive enough, we must base our recommendations on factors other than
respondent preference. On the basis of this analysis, there is no clear evidence that one version
worked better than the other. In both Version 1 and Version 2, just over half of the respondents
answered without error and interpreted the questions as intended. Nonetheless, we are inclined to
recommend using Version 1 with some changes because separating wages and salary from the
other types of compensation (e.g., tips, bonuses, commissions) avoids some confusion. In
Version 2, three respondents lost track of all of the types of earnings that the question specifies,
and they did not hear the earnings that they needed to report.
The following changes are recommended:
§

Put the term “additional” in parentheses in Question 49a2a.

Add an indication that an annual figure is expected in Questions 49a1b and 49a2b;
several respondents wondered if they should provide a weekly, monthly, or annual figure.

Because a large number of respondents across versions were not reporting totals for the
past 12 months but rather for the last calendar year, we recommend reconsidering the
period of reference used for these questions. If ACS can consider a change in reporting
period, we strongly recommend asking about the prior calendar year, as a large number of
respondents already answered these questions with that time frame in mind and indicated
it would be easier to do so. If ACS cannot consider such a change in reference period, the
questions need to stress the desired period, as in the questions below.

9. Wages and Salary: Summary of Findings and Recommendations

Respondents with self-employment income tended to report it under the wages and salary
question and frequently under the self-employment question as well. To avoid this, we
recommend moving the self-employment question before the wages and salary question.

Because Question 49a1b uses the term “wages and salary,” Question 49a1b should use
that phrase as well. Similarly, the terms “tips, commissions, and bonuses” should be used
in 49a2b to differentiate it from 49a1b.

Table 9-2 shows the recommended alternate wording that can be tested against the
control (i.e., Version 3).
Table 9-2. Proposed Alternate Wording for the Wages and Salary Question
Spanish version
Las siguientes preguntas son sobre ingreso
DURANTE LOS ÚLTIMOS 12 MESES . . .
49a1a. ¿Recibió usted algún jornal, sueldo o
salario en los últimos 12 meses?

English version
The next few questions are about income DURING
THE PAST 12 MONTHS. . .
49a1a. Did you receive any wages or salary
during the past 12 months?

Sí

Yes

No à Skip to Q49a2a

49a1b. ¿Cuánto recibió usted en total de sus
jornales, sueldos o salarios de todos
los empleos en los últimos 12 meses
antes de impuestos y otras
deducciones?
49a2a. ¿Recibió usted alguna propina,
bonificación o comisión (adicional)
DURANTE LOS ÚLTIMOS 12 MESES?

49a1b. How much did you receive in total for
all wages and salary from all jobs in the
past 12 months before taxes and other
deductions?
49 a2a. Did you receive any (additional) tips,
bonuses, or commissions DURING THE
PAST 12 MONTHS?
Yes

Sí
No à Skip to Current Q47b
49a2b. ¿Cuánto recibió usted en total de todas
las propinas, bonos o comisiones de
todos los empleos en los últimos 12
meses antes de impuestos y otras
deducciones?

No à Skip to Current Q47b
49a2b. How much did you receive in total for
all tips, bonuses, or commissions from
all jobs in the past 12 months before
taxes and other deductions?

10. Interest and Dividends: Summary of Findings and Recommendations

10.

Interest and Dividends: Summary of Findings and
Recommendations

Two versions of the Interest and Dividends module were tested in the intervieweradministered mode only. A total of 33 interviews were conducted using this module. Of the 33
interviews conducted, 16 were conducted with Version 1, and 17 were conducted with Version 2.
Interviews were also conducted in both English and Spanish. A total of 17 of 33
interviews were conducted in English, and 16 were conducted in Spanish. Respondents were
recruited to meet three different characteristics. Of the 33 interviews, 8 in English and 4 in
Spanish were conducted with respondents who received interest and/or dividends; 5 in English
and 5 in Spanish were conducted with respondents who received rental income or income from
royalties, estates or trusts; and 4 in English and 7 in Spanish were conducted with respondents
who did not receive any type of property income.
See Table 10-1 for the question wording of each version tested.
Table 10-1. ACS Questions Tested for Property Income
Version/
Language

Selfadministered

Version 1
English

Selfadministered
Version 1 was
not tested

Interviewer-administered
Q49c1a. [The next few questions are about income DURING THE PAST 12
MONTHS. . .]
Did [/you] receive any interest or dividends [DURING THE
PAST 12 MONTHS]? Report even small amounts credited to an
account.
Yes
No à Skip to Q49c2a
Q49c1b. What was the amount received? ________
Q49c2a. Did [/you] receive any net rental income, royalty income,
or income from estates and trusts [DURING THE PAST 12
MONTHS]?
Yes
No à Skip to Q49d1
Q49c2b. What was the amount received? ________

10. Interest and Dividends: Summary of Findings and Recommendations

Version/
Language

Selfadministered

Interviewer-administered

Version 1
Spanish
(stateside
and Puerto
Rico)

Selfadministered
Version 1 was
not tested

Q49c1a. [Las próximas preguntas tienen que ver con el ingreso DURANTE
LOS ÚLTIMOS 12 MESES. . .]
¿Recibió [/usted] algún interés o dividendo [DURANTE
LOS ÚLTIMOS 12 MESES]? Informe cantidades acreditadas a una
cuenta aunque sean cantidades pequeñas.
Si
No à Pase a la pregunta Q49c2a
Q49c1b. ¿Cuál fue la cantidad recibida? ________
Q49c2a. ¿Recibió [/usted] algún ingreso neto de rentas, ingreso
por derecho de autor o ingreso de herencias o fideicomisos
[DURANTE LOS ÚLTIMOS 12 MESES]?
Si

No à Pase a la pregunta Q49d1
Q49c2b. ¿Cuál fue la cantidad recibida? ________
Version 2
English

Selfadministered
Version 2 was
not tested

Q49c1a. [The next few questions are about income DURING THE PAST 12
MONTHS. . .]
Did [/you] receive any interest or dividends [DURING THE
PAST 12 MONTHS]? Report even small amounts credited to a
checking or savings account.
Yes
No à Skip to Q49c2a
Q49c1b. What was the amount received? ________
Q49c2a. Did [/you] receive any net rental income, royalty income,
or income from estates and trusts [DURING THE PAST 12
MONTHS]?
Yes
No à Skip to Q49d1
Q49c2b. What was the amount received? ________

10. Interest and Dividends: Summary of Findings and Recommendations

Version/
Language

Selfadministered

Interviewer-administered

Version 2
Spanish
(stateside
and Puerto
Rico)

Selfadministered
Version 2 was
not tested

Q49c1a. [Las próximas preguntas tienen que ver con el ingreso DURANTE
LOS ÚLTIMOS 12 MESES. . .]
¿Recibió [/usted] algún interés o dividendo [DURANTE
LOS ÚLTIMOS 12 MESES]? Informe cantidades acreditadas a una
cuenta de cheques o de ahorro aunque sean cantidades
pequeñas.
Si
No à Pase a la pregunta Q49c2a
Q49c1b. ¿Cuál fue la cantidad recibida? ________
Q49c2a. ¿Recibió [/usted] algún ingreso neto de rentas, ingreso
por derecho de autor o ingreso de herencias o fideicomisos
[DURANTE LOS ÚLTIMOS 12 MESES]?
Si
No à Pase a la pregunta Q49d1

Q49c2b. ¿Cuál fue la cantidad recibida? ________
Version 3
English

Selfadministered
Version 3 was
not tested

Interviewer-administered question Version 3
Question series #49c was not included in Version 3

Version 3
Spanish

Selfadministered
Version 3 was
not tested

Spanish IA question text Version 3
Question series #49c was not included in Version 3

10.1

Findings from the Interest and Dividends Module

Overview

The majority of respondents could answer, without difficulty, the questions regarding
whether the respondent received any interest or dividends (Question 49c1a) and the amount
received (Question 49c1b). The Spanish-speaking respondents expressed more difficulty with
these questions than did the English-speaking respondents. However, the primary difficulty that
Spanish-speakers had was in knowing whether a small amount of interest was worth reporting
for these questions. Based on the cognitive interview data, there were no differences in
respondents’ understanding of the questions or their ability to report accurately based on the
mode of interview when comparing the telephone and face-to-face interviews. (No selfadministered version of these questions was tested in the cognitive interviews.)
Education and income seemed to be correlated with respondents’ understanding of the
property income questions. A higher percentage of respondents with college degrees were
included in the Property Income module due to the screening process required to recruit people
who were eligible. Those people with property income tended to have higher education levels
than some of the other respondents. However, there were some respondents with less than a high
school education who were included in the Interest and Dividends analysis, and those people
tended to have more difficulty with these questions regardless of the language spoken.

10. Interest and Dividends: Summary of Findings and Recommendations
Reporting Interest and Dividends Received (Question 49c1a)

The primary goal of the cognitive interviews for this module was to determine if the
respondents understood that they should report any amount of income from interest or dividends,
no matter how small the amount may have been. The two versions of the ACS questions both
included clarification statements to ensure that even small amounts of income were included.
The text in both versions was very similar, but Version 2 had an additional point of clarification
added. The Version 2 text read, “Report even small amounts credited to a checking or savings
account.” Where the Version 1 text only referred to “an account,” the Version 2 text added the
words “checking or savings account.”
Based on a review of the comments and explanations among the 14 respondents who
reported having received interest or dividends, it was clear that those who received such income
understood and reported accurately. Confusion was primarily expressed by respondents who did
not receive income from property income. Six respondents, who did not report having interest or
dividends income, did not understand the concept of reporting small amounts of interest. The
respondents thought it had to do with reporting small amounts of money in the bank, having
small saving accounts, if one uses direct deposit; and one respondent thought that it might also
include small bonuses received. They did not demonstrate a working understanding of interestbearing accounts and the interest accrued. These respondents did not mention dividends at all
when they explained the meaning of the question during the probing part of the interview.
Twenty-one of 33 respondents were able to understand the question regarding income
from interest and dividends (Question 49c1a), and they seemed to answer correctly. Only 6 of 33
respondents outwardly exhibited any difficulty in answering the question; however, 12
respondents (8 in Spanish and 4 in English) reported confusion at some point during the probes
or they provided responses that indicated they did not completely understand the question.
Despite apparent confusion for some, only six respondents (three in Spanish and three in
English) actually answered the questions incorrectly. Although the other respondents stated that
they did not understand the terminology, they reported no income from interest or dividends,
which was correct.
Of the six respondents who exhibited difficulty answering Question 49c1a, five were
Spanish-speaking respondents, while only one was an English-speaking respondent. The
difficulties noted, however, were not related to language issues. Three of the respondents
expressed that it would be difficult to report an amount so small. The one English-speaking
respondent had no difficulty for himself but could not answer for another household member.
The other two Spanish-speaking respondents were not familiar with the terminology, “interés o
dividendo” (“interest or dividends”) in the question. Through probing, it was determined that
these respondents did not have a clear understanding of the terms. One thought that “dividendo”
(“dividend”) had to do with a disability payment. Another thought the question had to do with
reporting one’s salary.
Reporting Amount of Interest and Dividends Received (Question 49c1b)

Three of 33 respondents exhibited difficulty answering Question 49c1b, which asks about
the amount of funding received: 1 of 16 Spanish-speaking respondents and 2 of 17 Englishspeaking respondents. For 7 of 33 respondents, Question 49c1b was not applicable or had no

10. Interest and Dividends: Summary of Findings and Recommendations

response. For these seven respondents, six reported that they received no interest or dividends
and one indicated that he did but the amount was so little that he did not count it. Of the seven
with no response to Question 49c1b, four expressed difficulty with the previous question
(49c1a), which asked if the respondent had received any interest or dividends.
Reference Period, “The Past 12 Months”

Numerous respondents who reported having interest and dividends acknowledged that
they provided their response to the question based on the previous calendar year (e.g. January
2008 to December 2008) rather than the past 12 months. Four respondents reported this way
because they had recently completed their 2008 taxes. These respondents indicated that it would
be more difficult to report this information if it were not “tax time.” However, other respondents
were not clear that the 12-month reference period was intended to be a rolling time period. They
thought that it meant the previous calendar year. Because of this confusion by respondents, the
recommendation to revise the reference period to include the specific dates for the computerassisted interviewer-administered versions is documented in the final recommendations for this
module.
Reporting Income from Rent, Royalties, Estates, or Trusts (Question 49c2a)

Most of the respondents could answer the questions regarding whether they had received
any net rental income, royalty income, or income from estate and trusts (Question 49c2a) and
what the amount of income was (Question 49c2b). Overall, both versions tested were well
understood by the respondents, including those who actually had this type of income to report as
well as those who did not. The only problematic term in this question was “net” rental income
(“ingreso neto de rentas”). This term was misunderstood by both English- and Spanish-speaking
respondents. (Additional detail about the terminology issues is documented in the subsequent
section.)
Most respondents had no difficulty articulating the types of income that would be
included in Question 49c2a. Among those respondents who reported having received rent,
royalties, estate income, or trusts, their comments were directed more to their own personal
situations, with a focus on the type of income they receive. Other respondents who did not report
receiving income in Question 49c2 provided adequate explanations of the intent of the question
with only a few exceptions
Two of 33 respondents exhibited difficulty answering Question 49c2a; both respondents
were English-speakers. None of the 16 Spanish-speaking respondents showed difficulty. There
were 2 additional respondents among the 33 who had no recorded response to the interviewerobserved question about the difficulty in reporting income from rental properties, royalties,
estates, or trusts, however neither of these respondents reported having any such income to
report.
Two of 33 respondents exhibited difficulty answering Question 49c2b; both were
English-speaking respondents. Fourteen of 33 respondents correctly answered “Not Applicable”
for Question 49c2b because they reported having no income in their answers to Question 49c2a.
One case was missing a response to the interviewer-observed question about the level of
difficulty, but this respondent did not report having income at Question 49c2a either.

10. Interest and Dividends: Summary of Findings and Recommendations
Difficulty with Terminology

One of the probes in the interview asked specifically if respondents understood the term
“net income” from rental properties (“ingreso neto de rentas”). This proved to be quite
problematic for many respondents. A variety of both correct and incorrect responses were
provided by both English-speaking and Spanish-speaking respondents. Most of the respondents
were confident in their understanding (regardless of whether they were correct or not), but a few
were uncertain and changed their minds as they contemplated the intended meaning:
§

Eight respondents thought that “net income” from rent was clear and would be
understood by all.

Four respondents thought that “net income” from rent would not be clearly understood.

Six respondents thought that “net income” from rent meant “all,” “total,” or “total
income” (“ingreso total”).

Nine respondents thought that “net income” from rent was income after expenses or after
taxes. (Some, but not all of these respondents were also included in the count of
respondents who reported they thought the term “net” was clear.)

Two respondents thought that “net income” from rent meant “true value.”

Two respondents thought that “net income” from rent meant “gross” (“en bruto”) or
before expenses.

Based on the cognitive interview data for this question about “net rental income,” one of
the versions recommended for the field test includes an additional clarifying statement to explain
what is intended for the term “net” in this question.
Differences by Language

For Question 49c2a, which asks about income,from trusts, two Spanish-speaking
respondents indicated that the word “fideicomiso” (trusts) might be difficult for some people to
understand. One respondent said that this word was used more in Mexico, though she did
understand it and she was not from Mexico. The term as it is written in the question is correct,
and it is not a regional term, but a legal term. Most respondents seemed to understand enough of
what it meant to be able to understand if they did or did not receive income from trusts
(“fideicomiso”). Although these two terms (“trusts and “fideicomiso”) were not specifically
probed as part of this cognitive interview, we believe that the English-speaking respondents
would have similar difficulty defining a trust in accurate legal terms, but the English word itself
is a simple word that does not attract attention in the question whereas the term in Spanish is a
multi-syllable word that stands out in the sentence more and was noticed by two of the
respondents as being difficult to understand.
Appendix 16, Interest and Dividends Module: Final Briefing Recommendations, has
additional detail on these findings.

10. Interest and Dividends: Summary of Findings and Recommendations

10.2 Version Preference and Recommendations from the Interest and Dividends
Module
The differences between Version 1 and Version 2 for the Property Income questions were
relatively minor. The difference was only found in Questions 49c1a, which asks about receiving
income from interest or dividends. For Version 1, the clarification statement after the question
was, “Report even small amounts credited to an account.” The Version 2 text read, “Report even
small amounts credited to a checking or savings account.”
When asked which version they preferred for the Interest and Dividends questions, 16 of
33 respondents preferred Version 2, whereas 11 respondents preferred Version 1. Six
respondents had no preference. Of the 16 respondents who received Version 1 of the ACS
instrument, the same number of respondents preferred Version 1 as preferred Version 2 (n=7 for
each). However, the 17 respondents who received Version 2 of the ACS were more than twice as
likely to prefer Version 2 (9 of 17 preferred Version 2, whereas 4 of 17 preferred Version 1).
Also, of the 17 who received Version 2 of the ACS, more respondents had no preference for
either version (4 of 17), whereas of the respondents who received Version 1 of the ACS, only 2
of 16 had no preference.
When asked to explain their preference for one version over the other, about half (n=6) of
the 11 respondents who selected Version 1 preferred it because it is more general. The
respondents did not like that Version 2 specifically mentioned “checking/savings” because they
felt it would narrow the focus too much and could cause other respondents to interpret the
question in a more limited way (i.e., to include only checking and savings accounts).
Even though more respondents preferred Version 2 because it might help people know to
include even small amounts of interest from checking or savings accounts, several respondents
noted that they thought only interest from checking or savings accounts should be included. They
did not understand that the reference to including even small amounts from checking and savings
was just an example. These respondents’ answers indicated that they thought the reference to
reporting “even small amounts credited to a checking or savings account” meant that the
question was intended to limit the reported amount only to the interest earned from bank
accounts. They did not understand that the examples included were only to illustrate the types of
small amounts of income that might be included. Similarly, respondents who preferred Version 1
indicated that they thought other people might misinterpret the wording in Version 2 to mean
only report income from savings or checking accounts.
Based on specific explanations for preferences, it was clear that some respondents felt
that the addition of “checking or savings” would help reduce underreporting in the amounts of
interest and dividends. However, there were also respondents who felt that the addition of the
terms “checking and savings” only seemed to help clarify but would actually cause some
respondents to underreport income from interest and dividends because they would report only
income from checking and savings and not from stocks, bonds, or business accounts that might
provide income. This observation was based both on the actual types of income reported for each
version as well as the comments from the probing questions. Both Spanish-speaking and
English-speaking respondents had similar observations and understandings of the intentions of
the questions.

10. Interest and Dividends: Summary of Findings and Recommendations
Specific recommendations for Revised Question 49c1a

Based on the observations documented in this report, RTI recommends using a modified
Version 2 for the field test. The (modified version of the alternate wording) should make it clear
that interest earned from checking and savings accounts is just an example of the types of interest
to be included. For the interviewer-administered versions of these questions, which are
computerized, we suggest filling the months whenever the phrase “the past 12 months” is used.
For example: “The next few questions are about income DURING THE PAST 12 MONTHS,
which is from to .”
Specific recommendations for Revised Question 49c2a

As noted in the specific observations for question 49c2a, there was some confusion about
the term “net rental income.” RTI recommends clarifying this term and testing respondents’
understanding. From the probed responses, many respondents think of “net” only in terms of
salary and taxes. They are not accustomed to thinking of “net rental income” and had difficulty
understanding and explaining what it meant.
Table 10-2 shows the recommended versions to be tested in the field test.
Table 10-2. Proposed Question Wordings for the Interest and Dividends Questions
Language

Interviewer-administered

English
Version
to be
Tested

Q49c1a. [The next few questions are about income DURING THE PAST 12 MONTHS, which is
from to . .]
Did [/you] receive any interest or dividends [DURING THE PAST 12
MONTHS]? Report even small amounts. For example, report any interest or
dividends credited to a checking or savings account as well as any other income
from interest or dividends
Yes
No à Skip to Q49c2a
Q49c1b. What was the amount received? ________
Q49c2a. Did [/you] receive any net rental income, royalty income, or income from
estates and trusts [DURING THE PAST 12 MONTHS]? Net rental income is the
amount earned after expenses.
Yes
No à Skip to Q49d1
Q49c2b. What was the amount received? ________

10. Interest and Dividends: Summary of Findings and Recommendations

Language

Interviewer-administered

Spanish
Version
to be
Tested

Q49c1a. [Las próximas preguntas tienen que ver con el ingreso DURANTE LOS ÚLTIMOS 12
MESES, es decir desde hasta . . .]
¿Recibió [/usted] algún interés o dividendo [DURANTE LOS ÚLTIMOS 12
MESES]? Mencione cantidades, aunque sean pequeñas, por ejemplo intereses o
dividendos que le hayan acreditado a una cuenta de cheques o de ahorros así
como cualquier otro ingreso por intereses o dividendos
Si
No à Pase a la pregunta Q49c2a
Q49c1b. ¿Cuál fue la cantidad recibida? ________
Q49c2a. ¿Recibió [/usted] algún ingreso neto de rentas, ingreso por derecho de
autor o ingreso de herencias o fideicomisos [DURANTE LOS ÚLTIMOS 12 MESES]?
Ingreso neto de rentas es la ganancia que queda después de descontar los gastos.
Si
No à Pase a la pregunta Q49d1
Q49c2b. ¿Cuál fue la cantidad recibida? ________

11. Cash Public Assistance: Summary of Findings and Recommendations

11.

Cash Public Assistance: Summary of Findings and
Recommendations

We tested two versions of proposed new ACS questions about public assistance as a
source of income. Both versions included phrasing to encourage respondents to report public
assistance as an income source even if they had received it only once during the 12-month
reference period. One version did this by stating “even if for only one month;” the other version
stated “even if for only one payment.” Within the self-administered mode, a further variation
concerned the ordering of the two key phrases within the question: Version 1 stated “…even if
for only one month, for this person or any children in this household,” while Version 2 stated “…
for this person or any children in this household, even if only one payment.” Table 11-1 presents
the question wording of each Public Assistance question version tested across modes and
languages (English and Spanish).
Table 11-1. ACS Questions Tested for Public Assistance
Language
Version 1
English

Self-administered
49.

49f1.

Interviewer-administered

Mark (X) the “Yes” box for each type
of income this person received, and
give your best estimate of the total
amount during the PAST 12
MONTHS…

Q49c1a. [The next few questions are about income
DURING THE PAST 12 MONTHS….]
49f1.

Any welfare payments or cash
assistance from the state or local
welfare office, even if for only one
month, for this person or any
children in this household. Do not
include benefits from food, energy,
or rental assistance programs.

Did [/you] receive any welfare
payments or cash assistance from the
state or local welfare office, for
[/yourself] or any children in this
household during the past 12 months?
Include all assistance, even if for only one
month. Do NOT include benefits from food,
energy, or rental assistance programs.
[IF YES] 49f2. What was the amount?
___________

[IF YES] 49f2. TOTAL AMOUNT for
past 12 months:
Version 1
Spanish

49f. Algún pago de bienestar público o
asistencia en dinero en efectivo de la
oficina estatal o local de bienestar,
aunque sólo sea por un mes, para
esta persona o cualquier niño de este
hogar. no incluya beneficios de
programas de asistencia para
alimentos, energía o alquiler.

Q49c1a. [Las próximas preguntas tienen que ver
con el ingreso DURANTE LOS ÚLTIMOS 12
MESES….]
49f1.

[IF YES] CANTIDAD TOTAL para los
últimos 12 meses:

¿Recibió usted algún pago de bienestar
público o asistencia en dinero en efectivo
de la oficina estatal o local de bienestar,
para usted mismo o cualquier niño de este
hogar DURANTE LOS ÚLTIMOS 12
MESES. Incluya toda la asistencia
recibida, aun si fuera sólo por un mes. NO
incluya beneficios de programas de
asistencia para alimentos, energía o
alquiler.
[IF YES] 49f2. ¿Cuál fue la
cantidad?________

Version 2
English

49.

Mark (X) the “Yes” box for each type
of income this person received, and
give your best estimate of the total
amount during the PAST 12

Q49c1a. [The next few questions are about income
DURING THE PAST 12 MONTHS….]
49f1.

Did [/you] receive any welfare
payments or cash assistance from the

11. Cash Public Assistance: Summary of Findings and Recommendations

Language

Self-administered

Interviewer-administered

MONTHS…
49f1.

state or local welfare office, for
[/yourself] or any children in this
household DURING THE PAST 12
MONTHS? Include all assistance, even if
for only one payment. Do NOT include
benefits from food, energy, or rental
assistance programs.

Any welfare payments or cash
assistance from the state or local
welfare office for this person or any
children in this household, even if
only one payment. Do not include
benefits from food, energy, or rental
assistance programs.

[IF YES] 49f2. What was the amount?
___________

[IF YES] 49f2. TOTAL AMOUNT for
past 12 months:
Version 2
Spanish

Spanish SA question text Version 2
49f.
Algún pago de bienestar público o
asistencia en dinero en efectivo de la
oficina estatal o local de bienestar
para esta persona o cualquier niño
de este hogar, aunque sólo sea un
pago. No incluya beneficios de
programas de asistencia para
alimentos, energía o alquiler.
[IF YES] CANTIDAD TOTAL para los
últimos 12 meses:

Spanish IA question text Version 2
Q49c1a. [Las próximas preguntas tienen que ver
con el ingreso DURANTE LOS ÚLTIMOS 12
MESES….
49f1.

¿Recibió usted algún pago de bienestar
público o asistencia en dinero en efectivo
de la oficina estatal o local de bienestar,
para usted mismo o cualquier niño de este
hogar DURANTE LOS ÚLTIMOS 12
MESES. Incluya toda la asistencia
recibida, aun si fuera sólo por un pago. NO
incluya beneficios de programas de
asistencia para alimentos, energía o
alquiler.
[IF YES] 49f2. ¿Cuál fue la
cantidad?________

Forty-eight participants (24 English, 24 Spanish) were recruited specifically for the
public assistance module on the basis of having received public assistance. However, during the
interviews, we learned that slightly more than half (28 respondents) had not received the type of
public assistance of interest (i.e., welfare payments). The key screener item that identified
persons on public assistance during the recruiting was, “Are you currently receiving state or local
public assistance or welfare?” Many people answered “Yes” to this question during the screening
on the basis of having received benefits from other government programs, such as Supplemental
Security Income (SSI); Women, Infants, and Children (WIC); Medicaid; unemployment
compensation; Food Stamps; and, in one case, a child tax credit. In two cases, it was unclear
what happened because the participants indicated no one in the household had received any
government assistance of any kind in the past year. On a more encouraging note, the vast
majority of these participants (25) answered “No” correctly in response to the target question in
the cognitive interview. The 3 persons who incorrectly answered “Yes” are discussed in more
detail below.
In addition to the 48 respondents specifically recruited to test the public assistance
question, we probed 27 people who screened into the study as having food stamps. Thus, the
public assistance question was tested with a total of 75 respondents.

11. Cash Public Assistance: Summary of Findings and Recommendations

11.1

Findings from the Cash Public Assistance Module

Overview

Of the 75 respondents probed on the key public assistance question, 17 (11 Englishspeaking, 6 Spanish-speaking) experienced noteworthy problems in answering or interpreting it.
In our judgment, 12 to 13 respondents appeared to have answered the question incorrectly: 7
respondents answered positively when they should have answered negatively; and 2 respondents
(possibly 3) answered negatively when they should have answered positively. Finally, 3
respondents neglected to answer the question. The problems observed seemed largely
independent of interview mode, form version, or language of administration. The following is a
detailed description of the major problems we observed:
§

Other benefits. Four respondents answered “Yes’ incorrectly on the basis of other
unrelated benefits. One did so on the basis of unemployment compensation; another on
the basis of SSI payments (unfortunately, the previous SSI item was skipped for this
person due to time constraints, so this may have contributed to the misreporting). One
reported receiving public assistance because her son was receiving Medicaid; another
reported receiving public assistance because he received food stamps.
A couple of respondents (who answered correctly) believed that child support may be
relevant to the question. One person, after hearing the question, asked whether it included
unemployment compensation; she specifically pointed to the phrase, “Include all
assistance” and noted it made her think she should perhaps report it as public assistance.
One person answered correctly but wondered if her SSI payments were relevant to the
question (even though she had reported them in the previous question specifically asking
for SSI). Another person correctly answered “No,” but later noted that she was unsure
whether the tax credit for her children was relevant to the question. Finally, one
respondent (who correctly excluded WIC when answering) suggested that the question
should be reworded to include the “correct” examples (e.g., unemployment, retirement).
This person noted, “It’s confusing because ‘public assistance’ includes a lot of different
things, like help with the rent, with food, unemployment.”

Subjective interpretation. One respondent received public assistance, but incorrectly
reported that the other two household members received it as well, since the benefit was
“for the entire family.” Another person (mentioned previously as reporting public
assistance based on her son’s Medicaid) reported that both she and her husband received
public assistance. She noted that the question asked whether each person received
assistance for a child, and, as she saw it, they both did. We should note that virtually all
of the study participants who reported receiving public assistance had received it
themselves. In only 1 or 2 cases, the report was based on someone else in the household.
At least two people incorrectly reported that they had not received public assistance. One
mentioned in the probing that she had received “TANF” (Temporary Assistance to Needy
Families) up until recently. She did not realize that TANF was the type of public
assistance we were asking about and seemed to think the question was asking strictly
about payments in cash. Similarly, another person failed to report receiving public
assistance; probing revealed that she had received TANF during the year, but last

11. Cash Public Assistance: Summary of Findings and Recommendations

received what she considered “public assistance” several years before. We are unsure
about a third person who screened in as receiving public assistance, but answered “No” to
the target question. During probing, she said that she had been on WorkFirst (a state
name for welfare), but it had been a bit over a year since she had been on the program.
Thus, her negative response might have been correct; however, we are not sure that the
respondent recognized WorkFirst as public assistance. Finally, one person hesitated
before answering “yes” (correctly), noting that she received TANF and was not sure if
TANF was relevant to the question.
Key Terminology

“Any kind of cash from government [Would] include cash given: not cash in hand, but
cash in the form of check or put on a card which is given to individuals who only receive
welfare and that’s [the card] how they access the funds, not a cash handout.”

“Welfare checks. Any kind of aid that’s cash—not medical, food. Nothing else.”

Reporting the Amount of Public Assistance Received

Several problems were observed with respect to reporting the amount of public assistance
among the 22 cases where public assistance was received either by the respondent or another
household member, six people reported a monthly amount rather than the full amount received
over the previous 12 months. Four of these cases were in the interviewer-administered mode, and
in at least two of them, the persons explicitly stated that they were reporting a monthly figure.
The reason why the question was misinterpreted is not entirely clear, but we suspect that
respondents simply found the mathematical calculations involved in reporting an annual amount
to be too daunting. One participant explained that reporting an annual amount was too difficult
because her benefit was adjusted every so often based on the number of hours worked at her job.
At least three people included items that they should have excluded from the reported
amount. For example, one respondent had a roommate who received both food stamps and public
assistance. Probing revealed that she had reported the amount her roommate received for both
benefits combined—she said she considers the benefits to be essentially the same thing and was
unable to report the public assistance amount by itself. Another person combined her child
support with the public assistance amount. Yet another person combined his public assistance
with the social security death benefit that his stepson received.
Other problems we encountered included a respondent in the self-administered mode
leaving a zero off the number she entered on the form, thus reporting only one-tenth of the public

11. Cash Public Assistance: Summary of Findings and Recommendations

assistance amount she had received, and a respondent whose family received public assistance
for only a few months trying to forecast the amount for a 12-month period.
Differences by Language

We observed no notable differences between English- and Spanish-speaking respondents
with respect to interpretation or difficulties in answering the target public assistance question.
Problems and underlying causes were similar between the two language versions.
Differences by Version

Of the 39 respondents (19 English-speakers, 20 Spanish-speakers) probed on Version 1
of the public assistance question, 31 (13 English-speakers, 18 Spanish-speakers) answered
correctly and appeared to have no problems understanding what the question was asking. Almost
every respondent experiencing a problem with Version 1 was discussed in an example above.
There were 2 respondents who simply needed to read (or hear) the question more than once, as
they found the question rather long. We observed no problems related to the specific instructions
“even if for only 1 month” in Version 1.
Of the 36 persons (18 English-speakers, 18 Spanish-speakers) probed on Version 2 of the
public assistance question, 31 (15 English-speakers, 16 Spanish-speakers) seemed to have
answered correctly, with no problems understanding what the question was asking. All
respondents who answered incorrectly were discussed in the examples above. We observed no
problems related to the specific instructions “even if for only one payment” in this version.
When asked to compare the two versions of the question and state a preference,
respondents were evenly split. In fact, many did not seem to notice the true difference between
the two versions, and said they did not have a preference. One person suggested that combining
“past 12 months” with “just one month” (Version 1) might be confusing. A more important
finding is that a small number of respondents indicated they preferred the wording of Version 2,
where the question specified “even if for only one payment” because it acknowledged that not all
benefits were paid on a monthly basis
The additional variation within the self-administered mode (ordering of the phrases “even
if for only one month/payment” and “for this person or any children in this household”) was also
a negligible factor in the findings of this study. Respondents in this mode were almost evenly
split as to which version of the question was better, and only one person pointed to the order of
the phrasing as a reason for her preference. This participant noted that Version 1 was “detaching
the payments from the people,” and she found that “disruptive.” But many did not articulate a
clear reason for their preference, and when they did, they usually pointed out the “month” versus
“payment” difference.
11.2

Version Preference and Recommendations for Cash Public Assistance

The findings of this study do not clearly point to one question version being better than
the other. The problems and difficulties we observed were almost evenly distributed across the
two versions and unrelated to the wording variations of interest. Our main recommendation is to
place more emphasis on the instructions not to include benefits from other programs, and use

11. Cash Public Assistance: Summary of Findings and Recommendations

“month” rather than “payment.” Therefore, our recommendation is a modified version of the
question.
Table 11-2 shows the recommended version to be tested in the field test.
Table 11-2. Proposed Versions for Cash Public Assistance Question
Language
Modified
Version

English

Self-administered

Interviewer-administered

49f1. Any welfare payments or cash
49f1. DURING THE PAST 12 MONTHS,
assistance from the state or local welfare
did [/you] receive any
office, even if for only one month, for this
welfare payments or cash
person or any children in this
assistance from the state or local
household? Do not include benefits from
welfare office for
any other type of assistance, such as
[/yourself] or any children
SSI, food, energy, or rental assistance
in this household, even if for only
programs.
one month? Do NOT include
benefits from any other type of
assistance, such as SSI, food,
energy, or rental assistance
programs.

Modified
Version

49f1. Algún pago de bienestar público del
49f1. DURANTE LOS ÚLTIMOS 12 MESES,
programa o asistencia en dinero en
¿recibió [/usted] algún pago de
efectivo de la oficina estatal o local de
bienestar público o asistencia en dinero en
Spanish
bienestar, aunque sólo sea por un mes,
efectivo de la oficina estatal o local de
para esta persona o cualquier niño de
bienestar, para [/usted mismo] o
este hogar. NO incluya beneficios de
cualquier niño de este hogar, aunque sea
ningún otro tipo de programas de
sólo por un mes? NO incluya beneficios de
asistencia, como por ejemplo SSI, ayuda
ningún otro tipo de programas de
para alimentos, energía o para pagar el
asistencia, como por ejemplo SSI, ayuda
alquiler.
para alimentos, energía o para pagar el
alquiler.

Because a few respondents did not realize that “TANF” was considered welfare, we
suggest creating a show card with the name of the TANF program in each state (similar to the
card created for food stamps) so that interviewers can determine whether the program the
respondent mentions is correct.

12. Conclusions and Recommendations for Future Research

12.

Conclusions, Lessons Learned, and Recommendations for
Future Research

Overall, the primary objectives of the Cognitive Testing of the American Community
Survey Content Test Items pretest, which was to evaluate eight question topics as part of the
ACS survey, were met. Through the cognitive interviewing and analysis of 115 English
interviews and 105 Spanish interviews, final recommendations for proposed questions to be
tested in the field test in 2010 were identified. Any issues that were identified were documented
in draft and final recommendations briefing reports. The cognitive interview respondents
identified both questionnaire wording that worked well and some problematic wording of some
questions. The Lead Researchers and analysts from RTI, RSS, and Westat proposed
recommendations for possible revisions based on the feedback from the interviews. Ultimately,
the recommendations are believed to help increase comprehension and consistency of
understanding for Spanish- and English-speaking respondents for the Field Test of the American
Community Survey.
12.1

Lessons Learned

The scope of the ACS Content Test task order was quite ambitious given the number of
interviews, the multiple versions of the instruments and the schedule. The Census Bureau
understood this from the outset and encouraged multiple research contractors to collaborate on
the effort in order to meet the project goals on schedule. The collaboration both resolved issues,
making it possible to complete the work on schedule, and created other challenges that had to be
overcome in an expeditious manner. The significant lessons learned from this effort are
documented below:
•

Collaboration among multiple contractors and across locations – Overall, the
subcontracting worked well and ensured that the overall schedule deadlines were met.
With the number of cognitive interviews that were required in a relatively short time
period, it was necessary to divide the work among different research organizations. An
example of one aspect of the collaboration that worked well was the development of the
detailed recruiting spreadsheets that were shared online and were updated weekly initially
then daily as the recruitment targets were met. Using these tools, the three subcontractors
were able to manage the overall goals reasonably well. There were a few target cells that
were over-recruited due to multiple interviewers working simultaneously but this was a
known risk given the compressed schedule. This might have been an issue even with
multiple interviewers working for a single organization. A recommendation for future
research would be to allow additional time in the schedule to better ensure that
procedures and protocols can be implemented to maximize communication across sites.

•

Recruitment of highly specified respondent characteristics – From the initial contract
discussions, all parties were aware that a number of the specific types of respondents
would be difficult to identify and recruit. As documented in Chapter 3, Recruiting,
there were some recruitment targets that fell short of the goals, but nearly all of the

12. Conclusions and Recommendations for Future Research

targets had some respondents. (The exceptions were both for monolingual Spanishspeakers for the two target groups, municipal Wi Fi users and parents born in the US.)
Although there was a large overall number of interviews to be completed, many of the
individual cells of recruitment characteristics by instrument version and mode of
interview made the task of assigning a version and mode a challenge. While the initial
plan was to recruit heavily at the beginning and then make selections, the actual schedule
of activities did not allow for this due to the delays in finalizing the versions of the
instruments and protocols.
•

Recruitment of Spanish-speaking respondents – With the recent political climate and
policies directed toward immigrant populations, we have observed significant resistance
to recruitment efforts over the past years. One exception to this is the recruitment in
locations such as Chicago, which is considered to be a sanctuary city, and Puerto Rico. In
order to reduce the impact of such resistance, additional time is required for the
recruitment of Spanish-speaking respondents and more personal recruitment efforts are
required.

•

Planning for cognitive interviews and protocol development - The estimated time for
completing the ACS questions before beginning the cognitive interview probing section
was 38 minutes. This was based on the administration of the ACS by interviewers using
a computerized instrument rather than paper and pencil, as was used for the cognitive
interviews. We found during the cognitive interviews that the time for the administration
for the ACS itself was significantly longer. This was due in part to the paper and pencil
administration but also due to the nature of the cognitive interview itself. The fact that
the respondents knew their task was to identify problems with the questions and possibly
spent more time thinking about their responses meant that many times they did spend
more time with the initial ACS questions. In some cases, this led the interviewer to have
to move more quickly through the probing sections of the interview. As a result the
quality of some of the interviews suffered in that the richness of the probes was not as
high a quality as it could have been without a time constraint. A recommendation for
future research would be to limit the scope of the ACS questionnaire administration to the
questions that are relevant to the probed questions.

•

Ensuring adequate time for instrument development – While the initial schedule seemed
to allow adequate time for the protocol development, we underestimated the complexity
of the instruments that would be included for this research. The development of the
initial six interviews in English, then the translation and development of the six stateside
Spanish instruments did not allow sufficient time in the schedule for adequate testing of
the instruments before training. As a result, the instruments were revised following the
initial day of training and each contractor then conducted a retraining locally several
weeks later. If future research could allow more time between the project kick-off
meeting and the initial training, it would allow more time for adequate testing of the
instruments. Alternately, less complex instruments would have made the initial schedule
more feasible.

•

Training – While the training model that was used for this project was based on other
similar research, the time allowed for the practice interviews was not sufficient due to the
81

12. Conclusions and Recommendations for Future Research

overall length of the cognitive interview. As mentioned previously, additional time in the
schedule would allow for more sufficient testing of the instruments before training. Quite
a significant amount of time in training was spent answering questions about the use of
the paper and pencil versions of the ACS, which led to less time for actual practice for the
interviewers.
•

Security clearance – The process for getting new staff cleared through the eQip systems
varied quite a bit from individual to individual but the overall impact on the staffing and
interviewing plans were consistent for the contractors who had new staff assigned. Our
recommendation for future research is to assume a worst-case scenario for the security
clearances so that there is less likelihood of a negative impact on the schedule.

•

Quality of Spanish translation of the tested questions – One positive finding from this
experience was that the feedback from cognitive interview respondents was much more
positive about the quality of the Spanish translations. There were very few instances of
translations problems with the questions tested. We believe that this is partly due to the
fact that even the “new” questions tested were based on existing versions of questions
that had vocabulary that has been well translated and tested prior to this research effort.

12.2

Recommendations for Future Research
The following are possible topics for future research related to the ACS:

As the Census Bureau expands the number of languages for which it produces translated
materials, there is a continuing need to review the translations to ensure that they meet
the standards and guidelines for cultural appropriateness as well as accuracy.

Additional research is needed that targets a better understanding as to which ACS
materials are most useful for these linguistically isolated populations.

Research is needed into effective communication protocols used with translation
contractors to better understand how to facilitate two-way communication with Census
Bureau methodologists and translators to improve the cognitive equivalence of the
intended messages. There is also a need to better understand how constraints of matching
English in both sentence structure and format can impact the cultural sensitivity of the
messages.

An investigation into how these non-English speakers are motivated to complete the ACS
forms and how they actually complete the forms is needed. For example, do they
complete the forms themselves or do they have an English-speaking family member or
friend complete the forms for them?

An investigation is needed that will provide insight on reasons for response/non-response
to the ACS.

File Type	application/pdf
File Title	Microsoft Word - Final_Report_Aug11.doc
File Modified	0000-00-00
File Created	0000-00-00