Download:
pdf |
pdfDRAFT
VOTING AND CIVIC ENGAGEMENT (VCE)
SUPPLEMENT TO CPS:
COGNITIVE TESTING (ROUND 1) SUMMARY REPORT
Jennifer M. Rothgeb (SRD)
Aniekan A. Okon (DSD)
Gianna S. Dusch (DSD)
November 6, 2007
Contact Information:
Jennifer M. Rothgeb
Statistical Research Division
U.S. Census Bureau
301-763-4968
[email protected]
DRAFT
VOTING AND CIVIC ENGAGEMENT (VCE) SUPPLEMENT TO CPS
COGNITIVE TESTING (ROUND 1) SUMMARY REPORT
EXECUTIVE SUMMARY -will be included in final report. Refer to Section IV-General
Information below to determine which sections of the report might require the most discussion at
the meeting.
I.
BACKGROUND - will be included in final report
II.
OBJECTIVES - will be included in final report
III. METHODOLOGY - will be included in final report
A. General- will be included in final report
B. Protocol for Cognitive Interviewing - will be included in final report
C. Administrative Information and Participant Demographics
Fifteen laboratory participants were recruited for the first round of cognitive testing of the
proposed Voting and Civic Engagement (VCE) supplement to the Current Population Survey
(CPS.) They were recruited through newspaper ads, postings on craigslist.com, our database of
potential laboratory participants and word-of-mouth. The interviews were conducted by
experienced cognitive interviewers (one person from the Statistical Reseach Division (SRD) and
two people from the Demographic Surveys Division (DSD) in late September and early October.
Because the CPS is conducted by telephone and in person, four of the interviews were conducted
by telephone and eleven were in-person interviews. Three of the telephone interviews included
persons in very rural areas in Colorado, central Pennsylvania and central Virginia. The in-person
interviews were conducted with persons in the Baltimore-Washington metropolitan area. The
interviews took approximately 75 - 90 minutes. All interviews were audio taped (with
permission of the participant.) Participants were provided $40.
Two versions of the VCE questionnaire were used. Eight participants were administered Version
A and 7 were administered Version B.
Page 2
Provided below in Table 1 is summary demographic information about the 15 participants.
Table 2 provides information on the household size for each participant.
Table 1. Demographic Characteristics of VCE Cognitive Interview Participants
Gender
Age
Race
Education
Male
4
15-24
0
White
12
< high school
1
Female
11
25-34
3
Black
3
H.S. grad
2
35-44
3
Some college
5
45-54
5
B.A.
6
55-64
3
M.A.
1
65+
1
TOTAL
15
15
15
15
Table 2. Household Size of VCE Cognitive Interview Participants
Household Size
VCE Participants
1 person
4
2 people
7
3 people
2
4 people
1
Unknown
1
TOTAL
15
In considering the results discussed in the next section, the usual caveat applies regarding
cognitive interviews. Participants were not randomly selected and the results are not
generalizable to the larger population. It should also be noted that the participants included in
this research were older and had a higher educational attainment level than the general
population.
Page 3
IV. RESULTS AND RECOMMENDATIONS
General Information
In this section, there is a subsection for each question. Within each subsection, the tested
question wording is provided. The response distribution is provided alongside the specific
response option within the question wording section. Following the question wording, the results
are described, followed by a recommendation for round two, if deemed necessary. A justification
for the recommendations is provided And finally there is space allowed for the final decision for
each item, once decisions are made about round two.
There were several items that worked well, as evidenced by participants’ paraphrases and
responses to targeted probes. Some detailed discussion is provided in the section related to the
specific item. No revisions to these items for round two testing is recommended. These items
include the following questions:
A (local elections: questionnaire item 1)
B (family and friends political discussions: questionnaire item 2)
C (means of obtaining news and information: questionnaire item 3a-3e)
D (political action: questionnaire item 4a-4e)
F (served as officer or on committee: questionnaire item 6)
G (attendance at meeting of group or organization: questionnaire item 7)
H (eating dinner with household members: questionnaire item 8)
J (talk with any of neighbors in 10 or 20 closest households: questionnaire item 10)
K (favors with neighbors: questionnaire item 11)
Items for which revisions for round two testing are recommended and a couple of items for
which recommendations are not yet formulated include the following:
E (participation in groups and organizations: questionnaire item 5a-5f)
I (communicating with family and friends by Email or Internet: questionnaire item 9)
L (close friends question series: questionnaire item 12a-12e)
M (social trust: questionnaire item13 )
N (social trust: questionnaire item 13R)
O (knowledge about Federal laws: questionnaire item 14)
Page 4
Detailed Results
A. Local Elections
1. Question Wording
In any election, some people are not able to vote for various reasons. Thinking about
elections for LOCAL government officials over the past two years, have you voted in all
of them, most of them, some of them, or none of them?
9
9
9
9
9
9
All of them
Most of them
Some of them
None of them
DK
Refused
6
2
3
4
2. Discussion of Results
From paraphrases and targeted probes it was clear that participants understood this question
and were able to answer the question fairly easily. Their interpretation of the phrase
“elections for local government officials” was consistent with the intended definition. The
types of local electoral offices that participants mentioned, when probed, included sheriffs,
judges, county and city elections, local supervisor, school board, mayor, governor, anyone
below state level - below congressional level, councilwoman, superintendent of education,
county/city - smaller than state, county executives, local judges.
Of those for whom we probed about the timing of local elections compared to national
elections, eight of the eleven participants said their local elections are held at the same time
as general elections.
Information from the probe asking about how many local elections had taken place in the area
during the past two years probably isn’t particularly useful because the probe omitted the
critical word “local”. Participant’s responses to the probe included some discussion of
general elections as well as local. Responses ranged from one to six elections during the past
two years. In response to the probe about how many of them (local elections) they actually
voted in, eight of 9 participants were consistent in the original response they gave to the
survey question. One participant reported “some of them” to survey question, but reported “I
don’t know, maybe none” to the probe question.
In response to the probe inquiring whether a reference period of three years rather than two
years would produce any additional election participation, nearly all said “no” or that they
didn’t thinks so. Three of the 13 participants receiving the probe mentioned issues related to
memory and recall for that long of a time period. Interestingly a three year reference period
from when cognitive interviews were conducted would have included the 2004 general
election. Given that many participants mentioned that their local elections are held the same
Page 5
as the general election, additional election participation was expected.
3. Recommendations
Leave question as worded.
4. Justification for Recommendations
Question worked well as designed.
5. Final Decision
To be determined
B. Family and Friends Political Discussion
1. Question Wording
PRE-SUP
Lead-in - The next set of questions are about people’s involvement and communication
within their communities.
In a TYPICAL MONTH, when communicating with family or friends, how often are
politics discussed -- basically every day, a few times a week, a few times a month, once a
month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
4
2
5
3
1
2. Discussion of Results
From paraphrases and targeted probes it was clear that participants understood this question
and were able to answer the question fairly easily. Their interpretation of the terms
“communicating” and “politics” were overwhelmingly consistent with the intended
definitions.
“Communicating” was interpreted accurately as including discussions face-to-face, by phone,
and through Email. When probed about what people they were thinking of when answering
the question, eleven of the fifteen said friends and family, two said friends and two said
family. When probed whether their response would be different had question been specific to
Page 6
family, nearly all participants said their response to survey question would have been the
same. One participant said response would have been less often and another said it would
have been more often. When probed whether response would be different had question been
specific to friends, eight participants said their response to the survey question would have
been less often.
3. Recommendations
Leave question as worded.
4. Justification for Recommendations
Question worked well as designed.
5. Final Decision
To be determined
C. Means of Obtaining News and Information
1. Question Wording
I am going to read some ways that people get news and information. Please tell me how
often you do each of the following in a TYPICAL MONTH:
(a) Read a newspaper in print or on the Internet - basically every day, a few times a
week, a few times a month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day 8
A few times a week 3
A few times a month 1
Once a month
3
Not at all
DK
Refused
Results: Participants were clear that question was asking whether they read a newspaper
in print or on the internet. Their paraphrases were consistent with intent of question. In
response to targeted probe about inclusion of print newspaper and newspaper website,
participants said it was clear the question was asking about both. (One participant said it
was clear, but thought the question should be split into two questions.) From information
obtained after part “e” of this question, it was obvious that participants knew that this
question (part “a”) and the part “e” were different questions and there was not duplicate
reporting occurring between the two items.
Page 7
(b) Read news magazines such as Newsweek or Time, in print or on the Internet basically every day, a few times a week, a few times a month, once a month or
not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
1
5
1
2
5
Results: Participants’ paraphrases and responses to targeted probes indicated they had a
clear understanding of the question and no difficulty responding.
c) Watch the news on television or get news from television internet sites - basically
every day, a few times a week, a few times a month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
8
2
2
2
Results: Participants’ paraphrases and responses to targeted probes indicated they had a
clear understanding of the question and no difficulty responding.
(d) Listen to the news on the radio or get news from radio internet sites - basically
every day, a few times a week, a few times a month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
8
3
2
2
Results: Participants’ paraphrases and responses to targeted probes indicated they had a
clear understanding of the question and no difficulty responding.
Page 8
(e) Obtain news from any other Internet sources that we have not previously asked
about such as blogs, chat rooms, independent news services, etc. - basically every
day, a few times a week, a few times a month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day
2
A few times a week
1
A few times a month
1
Once a month
Not at all
11
DK
Refused
Results: Participants’ paraphrases and responses to targeted probes indicate they had a
clear understanding of the question and no difficulty responding. Participants had a
different interpretation of part “e” and part “a” and there did not seem to be duplicate
reporting between these two items.
2. Discussion of Results (Entire series a - e)
The question series worked fine. The paraphrases and response to targeted probes indicate
no problems with any of the five items (“a”- “e”) in the series. We probed whether there
were any other means through which people get the news and no one reported anything
other than “talking to people.” We probed about the participants main media source for
civic and political information. In descending order of frequency of main media source
was television (7), newspapers (4), Internet (3) and radio (1).
3. Recommendations
Make no revision to question series.
4. Justification for Recommendations
No problems detected.
5. Final Decision
To be determined
Page 9
D. Political Action
1. Question Wording
I am going to read a list of things some people have done to express their views. Please
tell me whether or not you have done any of the following in the last 12 months, that is
between September 2006 and now:
(a) Contacted or visited a public official - at any level of
government - to express your opinion?
Yes 5
No 10
(b) Attended a meeting where political issues are discussed?
Yes 6
No 9
c) Bought or boycotted a certain product or service because of
the social or political values of the company that provides it?
Yes 5
No 10
(d) Taken part in a march, rally, protest or demonstration?
Yes 1
No 14
(e) Showed support for a particular political candidate or party
by distributing campaign materials, fundraising, making a
donation or in some other way?
Yes 5
No 10
2. Discussion of Results
Participants’ paraphrases and responses to targeted probes indicate they understood the intent
of the question and had no difficulty providing a response. Their interpretations of key phrases
in the questions were understood as intended. Key phrases that were probed include: “public
official,” “contacted or visited,” “bought or boycotted a certain product or services,” “social
and political values,” “march, rally, protest or demonstration,”and “particular political
candidate or party.”
In response to the probe about “social and political values,” it was clear that a few participants
were thinking of the terms out of context from the survey question. Their interpretation of the
phrase included “your personal, social or moral values that pertain to you, whether it is religion
or animal rights,” “to me the first thing I thought about was abortion,” “social values would be
the way we live or react to each other or react to issues that are positive to our particular way of
thinking or will benefit society as a whole, and political values is about doing the best for the
good of society as a whole.”
3. Recommendations
Make no revisions to question series.
4. Justification for Recommendations
No evidence problems with question series.
Page 10
5. Final Decision
To be determined
E. Participation in Groups and Organizations
1. Question Wording (Version A)
Now I would like to ask about groups or organizations in which people sometimes
participate. I will read a list of types of organizations. Please tell me whether or not you
have participated in any of these groups during the last 12 months, that is between
September 2006 and now:
(a) A hobby, sports, arts, music or other leisure activity group?
Yes 4
No 4
(b) A service, social welfare, or fraternal organization?
Yes 2
No 5 Dk 1
c) A youth, parent, or school-support organization?
Yes 3
No 4
(d) A labor union or professional, trade, farm or business
association?
Yes 3
No 5
(e) A neighborhood, ethnic or political organization?
Yes 3
No 4
(f) A church, synagogue, mosque or other religious institution
or organizations?
Yes 4
No 3
2. Discussion of Results
Both versions of this item had some issues which need to be addressed prior to Phase II testing.
Some of the words identifying groups in various categories presented problems for some
participants.
Category “a” (A hobby, sports, arts, music or other leisure activity group?)
Most participants thought this question was asking about community involvement. Two
participants were confused by the question. One thought the question was asking “do you hang
out and do sports. I really don’t know.” The participant asked for the question to be reread and
then said “it’s asking me what kind of hobby you have”. Upon further probing, he said “it’s
kind of confusing to me”. Another responded “no” and when asked to paraphrase the question,
responded “to be honest with you, I really do not know. I guess the first thing that threw me off
was hobby, sports. Yes, I go to sports, but in an actual group am... I am just not familiar with
what that would mean... and that would be my answer.” Another participant said he did not
know what a “leisure activity group” is.
Page 11
Category “b” (A service, social welfare, or fraternal organization?)
A few participants mentioned that they did not know what a “fraternal” organization was. One
asked if it was like a “fraternities or sororities.” One reported that if “fraternal” had not been
included in the grouping in the question, his response might have been different. focused on
“welfare” and “fraternal”.
The words “service” and “social welfare” led one participant to think that category was
communicating public assistance welfare as opposed to a social service volunteer group.
Another also repeated the word “welfare” and asked “is the question did you give to them.”
The participant said that “social welfare” makes it sound like do you participate in the welfare
of your neighborhoods, like you don’t want no drug dealers or people robbing. It’s more like a
neighborhood watch.” Another said this question was a “little more complicated”. One
participant said his only thought was firefighters and police associations my wife and I made a
donation to them.” His response to the question was “no.”
Category “c”(A youth, parent, or school-support organization?)
Participants seemed to have a fairly clear understanding of that category and didn’t experience
problems responding. One participant did respond “yes” and said he is a coach for a
recreational soccer league and that he was not sure whether to report that here or under
category A. He explained that he thought category “a” had to do with him and things he did for
himself and that category “c” had to do with youth and doing things for the youth. So he
thought his coaching activity should be reported under category “c.”
Category “d” (A labor union or professional, trade, farm or business association?)
This category was generally understood by participants.
Category “e”(A neighborhood, ethnic or political organization?)
This category was generally understood.
Category “f”(A church, synagogue, mosque or other religious institution or organizations?)
Several participants mentioned in their paraphrases that they thought this question was asking
if they going to church (in addition to other church-related ) activities. In the hypotheticals we
administered asking if a person attends church but does not participate in any other church
activities, 4 of 6 participants said attending church should be counted as participating and
should be included as a “yes” response to category “f.” This is not what the sponsor intended.
The wording of this category should be revised prior to Phase II testing.
Page 12
Hypotheticals Administered Retrospectively.
To determine whether participants would think involvement with specific groups would count
as “participating” and to determine what category they thought would be appropriate for
various activities, descriptions of involvement with groups were presented to participants at
the end of the interview. Provided below is the name of the group or description of the activity
along with the information obtained from participants.
Book club - Five of seven participants thought being in a book club should count and be
classified in category “a”. Two participants reported that it should not be counted with one
person elaborating by saying “because books, you discuss things, but I don’t know if any
changes are made because of those books.”
Rotary Club or Lions Club - Four of seven participants said they thought involvement with a
civic organization should count and be classified in “b,” while one participant thought it should
be classified in “a.” Two reported “yes,”but did not say which category they thought was most
appropriate.
Homeless shelter - All six participants asked about this reported that involvement in activities
with homeless shelters should count and should be included in category “b.”
Mothers Against Drunk Driving - All seven participants asked about involvement with this
group reported “yes,” it should be considered as participating. However there were different
opinions of which category it should be reported. Four different categories were selected
which included “b,” “c,” “e,” and “f.”
Neighborhood Watch Group or Neighborhood Advisory Committee - All seven participants
asked about involvement with such a group reported that it should count as participating and be
included in category “e.”
Going to religious services, but not participating in any other activities in the religious
organization - Four of the six participants thought this should be counted as participating and
be included in response to category “f.” Two reported that it should not be counted as
participating.
Page 13
Question Wording (Version B)
Now I would like to ask about groups or organizations in which people sometimes
participate. I will read a list of types of organizations. Please tell me whether or not you
have participated in any of these groups during the last 12 months, that is between
September 2006 and now:
(a) A sport, hobby, arts, cultural or other leisure activity group?
Yes 4
No 3
(b) A child welfare, social service, or public assistance organization? Yes 3
No 3
c) An educational, fraternal, alumni, or parent/teacher association? Yes 4
No 2
(d) A labor union or business, professional, trade, farm or association? Yes
No 8
(e) An ethnic association, political organization, or advocacy group? Yes 2
No 4
(f) A church, synagogue, mosque or other religious institution
Yes 3
No 7
or organizations?
Discussion of Results
Category “a” (A sport, hobby, arts, cultural or other leisure activity group?)
Several participants responding to this version mentioned the concept of “formalized groups”
when paraphrasing the question. One who gathers with friends regularly either for “trivia night”
reported that it wasn’t a “formalized” group, but it’s a group to her and she was trying to decide
whether that counted. She again stated that the question sounds like it’s asking if she’s in a
leagues of some kinds. She stated that the terms “hobby” and “leisure” are both extremely
general. She stated that she didn’t feel qualified to say “yes”, but that she did anyway.
Another participant started to interpret “sport” and “hobby” very generally saying he doesn’t
participate, but goes down to the part and watches the local schools play sports. He said “So if
that’s participation, then I’ve done that.” Upon further probing, the participant said “meaning was
I involved in it and did I go in and ask how did I help? Can I assist you with some of the kids and
get involved. That’s what you mean, but I haven’t done that.”
Categories “b,” “c,” “d,” “e” and “f”
No problems indicated through the responses or paraphrases.
Hypotheticals Administered Retrospectively.
To determine whether participants would think involvement with specific groups would count as
“participating” and to determine what category they thought would be appropriate for various
activities, descriptions of involvement with groups were presented to participants at the end of
the interview.
Book club - Four of five participants thought being in a book club should count and be classified
in category “a”. One reported that it should count but did not specify which category to put it in.
Page 14
Rotary Club or Lions Club - Four of five participants said they thought involvement with a civic
organization should count. Classification varied. One person thought category “b” was
appropriate, one thought category “c” was appropriate, while two thought category “e” would be
appropriate. One responded “don’t know.”
Homeless shelter - All four participants asked about this reported that involvement in activities
with homeless shelters should count, but they were split as to which category it should be
reported. Two thought it should be category “b” and two thought category “c.”
Mothers Against Drunk Driving - All five participants asked about involvement with this group
reported “yes,” it should be considered as participating. All five thought it should be reported in
category “e.”
Neighborhood Watch Group or Neighborhood Advisory Committee - All five participants asked
about involvement with such a group reported that it should count as participating and be included
in category “e.”
Going to religious services, but not participating in any other activities in the religious
organization - Four of the five participants thought this should be counted as participating. One
thought category “f” was the appropriate place to report such activity. Two other persons
identified category “f,” but were a bit tentative about their choice. One other person reported it
should count, but did not identify which category such activity should be reported.
3. Recommendations
****Unfortunately I put evaluation of this item off until the end. Therefore in the interest of time,
I am sending the report to you without recommendations for this item. I will develop
recommendations prior to the November 14 meeting and send them out to everybody.****
4. Justification for Recommendations
5. Final Decision
To be determined
Page 15
F. Served as Officer or on Committee
1. Question Wording
In the last 12 months, between September 2006 and now, have you been an officer or served
on a committee of any group or organization?
9
9
9
9
Yes
No
DK
Refused
2
13
2. Discussion of Results
Responses to paraphrases and targeted probes indicate that participants understood the question as
intended and had no difficulty providing a response. Phrases for which we probed included “been
an officer or served on a committee” and “group or organization.”
3. Recommendations
Make no revisions to question.
4. Justification for Recommendations
No problems detected during testing.
5. Final Decision
To be determined
G. Attendance at Meeting of Group or Organization
1. Question Wording
In the last 12 months, between September 2006 and now, have you attended a meeting of
any group or organization?
9
9
9
9
Yes
No
DK
Refused
8
7
Page 16
2. Discussion of Results
From paraphrases and targeted probes, participants seemed to understand the question, key
phrases and have no difficulty providing a response. Targeted probes were administered for
“attended a meeting,” and “of any group or organization.”
Information obtained from participants about the use of phrase “groups or organizations” versus
“clubs or organization” indicate that the word “club” was interpreted more narrowly for those few
participants who thought there was a difference between the word “clubs” and “groups.” Most
participants did not think there was a difference between the two terms.
The types of group or organizations participants attended included neighborhood advisory groups,
horseback riding clubs, social spiritual groups, political groups, alumni organizations, school
groups, soccer meetings, church meetings, mentoring committees, church ministries, and
alcoholic anonymous meetings. Persons participating in unstructured, informal groups (e.g.,farm
women getting together when someone dies or gets sick to make sure food is provided) accurately
responded “no” to the survey question. Persons whose only participation in a group is with
Sunday church services accurately responded “no” to the survey question.
Of the eight persons who answered “yes” to the survey questions, we asked if issues related to the
main purpose of the group had been discussed at the meetings they reported attending and they all
responded “yes.”
When probing to determine whether participants thought “social gatherings” of groups should be
included in response to the original survey question, 5 persons thought they should and 7 persons
thought they should not. (Note that it is not clear that participants understood what was being
asked in the probe about social gatherings. In Phase II, the probe will be structure differently so it
is clear what’s being asked.)
3. Recommendations
No revisions recommended. But we recommend revising the probe about social gatherings so it’s
more clear what is being asked.
4. Justification for Recommendations
No problems detected with question
5. Final Decision
To be determined
Page 17
H. Eating Dinner with Household Members
1. Question Wording
In a TYPICAL MONTH, how often do you eat dinner with any of the other members of
your household - basically every day, a few times a week, a few times a month, once a
month or not all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
8
2
1
2. Discussion of Results
There were 4 single person households that were not asked this question. Therefore information
about this question is based on responses from 11 participants. The question was well understood
by participants as evidenced by their paraphrases and responses to targeted probes.
In response to the probe about “what persons were you thinking of when answering this
question,” one participant indicated that in addition to her husband, she was also thinking of her
daughter and son (who do not live in the household.). This participant lived in a very rural area on
200 acres outside a small town. She said it was hard not to include them because in her mind they
are members of her household even though they have their own places to live.
3. Recommendations
No revisions recommended.
4. Justification for Recommendations
No problems detected during testing.
5. Final Decision
To be determined
Page 18
I. Communicate by Email or Internet
1. Question Wording
In a TYPICAL MONTH, how often, if at all, do you communicate with family or friends by
Email or on the Internet - basically every day, a few times a week, a few times a month, once
a month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
4
4
1
1
4
2. Discussion of Results
For the most part, participants understood this question. There were a couple of participants who
initially responded based on communication only with their family. During the probing,
participants became aware that the question was inquiring about communication with friends as
well as family. These participants said the response would be more frequent and changed their
initial response.
In response to probes, participants indicated that they did think communication on MySpace or
Facebook should be taken into account when answering the survey question which is consistent
with the sponsors’ intent. The majority of participants accurately did not think text messaging
should be considered. The four participants who responded that text messaging should be
considered did not indicate they were thinking of text messaging when formulating their response
to the survey question or in their paraphrase of the question.
3. Recommendations
Revised question wording for Phase II so the current phrase “family or friends” reads “friends or
family.”
4. Justification for Recommendations
A few participants only heard “family” and did not consider the communication they have with
friends by Email or through the Internet until the probing questions were asked. The
communication with friends was more frequent and changed which response category the
participant chose. For Phase II cognitive testing, we recommend reversing the order of “family or
friends” to “friends or family” and include probes to determine if participants are including both
groups when responding to the question.
Page 19
5. Final Decision
To be determined
J. Talk with Any of Neighbors in 10 or 20 closest households
1. Question Wording
In a TYPICAL MONTH, how often do you talk with any of the neighbors in the 10 or 20
households that live nearest to you - basically every day, a few times a week, a few times a
month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
4
4
6
1
2. Discussion of Results
Participants understood the question as indicated by their paraphrases and responses to probes.
There were numerous probes asked both concurrently and retrospectively to ensure that
participants understood the question intent, specific terms and phrases, and to determine how
participants were defining the “10 or 20 households that live nearest to them.” They correctly
understood that “talking” involved more than a casual ‘hello.” Additionally, participants
correctly answered based on how often they speak with any of the neighbors in the 10 or 20
households that live closest to them. One participant in a very rural area said the had to think of
a 20 miles radius to get the ten closest households. When probes inquiring about how many of
their neighbors they typically talk to, participants responses ranged from none to eight.
Five of the participants thought the question was asking only about “face-to-face” talking and did
not include phone conversations. Two of these people don’t ever talk to the neighbors on the
phone and the participants acknowledged that affected their response to the probe question.
Only one participant indicated that the people she was thinking of included family members.
This was the person in the very rural area who had to think of a 20 mile radius to get ten closest
households.
After review of all the responses to the numerous probes, there is evidence to suggest that most
participants did not consider the specifics of “10 or 20 households that lives nearest to them”
when responding to this question. Rather, participants were responding with regards to how
often they talk to their neighbors who live “in my building,”, “ in my community,” “households
Page 20
closer than 10-20, fewer,” etc. The range of proximity of how close the neighbors live to them
ranged from “across the hall” to “within several acres” to “within a mile.” Only one participant
indicated that the frequency depends on the time of year and that in the spring and summer there
is more interaction because people are outside more.
3. Recommendations
Despite the suspected lack of adherence the criteria of “10 or 20 households that live closest to
you,” participants seemed to be responding in a manner consistent with that desired by the sponsor.
No revisions are recommended for this question.
4. Justification for Recommendations
When all information is evaluated, there is not any evidence that revisions are warranted.
5. Final Decision
To be determined
K. Favors with Neighbors
1. Question Wording
In a TYPICAL MONTH how often do you and your neighbors do favors for each other? By
favors we mean such things as watching each other’s children, helping with shopping, house
sitting, lending garden or house tools, and other small acts of kindness - basically every day,
a few times a week, a few times a month, once a month or not at all?
9
9
9
9
9
9
9
Basically every day
A few times a week
A few times a month
Once a month
Not at all
DK
Refused
2
1
6
5
1
2. Discussion of Results
Participants had a clear understanding of the question as evidenced by their paraphrases and
responses to targeted probes. They understood terms of “neighbors,” “favors,” and “small acts of
kindness,” as intended. All participants understood that favors do not require face-to-face
interaction.
Page 21
3. Recommendations
No revisions recommended
4. Justification for Recommendations
No evidence of any problems during testing.
5. Final Decision
To be determined
L. Close Friends
1. Question Wording
a. About how many CLOSE FRIENDS do you have, if any? These are people you feel at
ease with, can talk to about private matters, or call on for help. Would you say that you
have no close friends, one or two, three to five, six to ten, or more than ten?
9 No close friends (Skip to 13)
9 One or two
9 Three to five
8
9 Six to ten
5
9 More than ten
2
9 DK
9 Refused
b. How many of these close friends have a college degree?
____
c. How many did not graduate from high school?
____
d. How many are of a different race than you?
____
Page 22
e. How many are from a different social or economic background than you?
2. Discussion of Results
Question “a” (number of close friends)
Question “a” was generally understood by all participants as evidenced by paraphrases and
response to probes. The terms “close friends” and “private matters” were understood by
participants in a manner consistent with that intended by the sponsor.
Three of the ten participants probed as to whether the question is intended to include family
members who are considered as close responded “no.” Four other participants said they did not
include family members in their count, even though they do have some they consider close
friends. The sponsor indicated prior to testing that close friends may include family members. In
Phase II, we may need to consider an inclusionary phrase for the question, given that most
participants with family members who are close friends did not include them in their count.
There were two participants who gave some indication that they were surprised by the question
about close friends, but those comments were provided only in response to probes and/or request
for paraphrases. One said “Um...I guess my first question was why are they asking this question.
It seemed like a strange question for the Census Bureau to be asking. But I think it was pretty
straightforward.” Another was: “It’s a strange question to me actually for some reason. Just how
many close friends you have and then tells you how to define a close friend.” Upon probing, the
participant said the reason she said it was a strange was “because of the context of where we’d
been in the survey, and I’m trying to think of what its purpose is as a question. I guess because if
you’re going to talk about politics or be engaged, it depends how big your social group is. But I
wasn’t expecting it.”
The period of time which participants were thinking about when answering the question varied
widely. Several participants reported they were thinking about their entire life. Some reported
“now,” “recent, present” Other time periods included “within the last 12 months,” and “last
couple of years.”
The number of close friends reported by participants fit within three of the 5 response categories
and were skewed toward the higher end of the response scale. (Nobody reported less than three to
five close friends.) This may be a function of the scale used. Participants may have been
widening their definition of close friends to fit with the scale. For example, since we offer “six to
ten” and “more than ten” participants may have counted some people as close friends that they
otherwise would not have had the top of our scale been four or more. We may want to test
different response categories in Phase II to determine if respondents’ answers are affected by the
response scale.
Question “b” (college degree)
Participants understood the question. All but one participant reported that an Associate of Arts
degree should count as a “college degree”for the purpose of responding to this question which fits
Page 23
with the definition provided by the sponsor.
Although question “b” asks “how many...”, several participants did not provide a number, but
responded by saying “most,” “all,” “majority,” “half,” or “50%”. This may be due to context
effects, since the prior question only requires the participant to provide a “range” as their answer
for number of close friends. For the subsequent questions, participants, may not have expected to
be required to give an exact number. Consideration may need to be given to restructuring the
question and have a response scale with percentages rather than asking for an exact number of
friends with a particular characteristic.
Question “c” (did not graduate from high school)
Question “c” was generally understood by participants. Several participants responded by saying,
“No, they all graduated from high school,” or “they all did.” If the question remains in its current
form, interviewer instructions will need to alert interviewers about the potential for these types of
responses. Verification of the response by the interviewer may be necessary to ensure the
interviewer should enter “0" indicating that the respondent has no friends who did not graduate
from high school.
Question “d” (different race)
Question “d” was understood by participants. No comments were provided by participants about
this question except for one participant who expressed that she was surprised that her answer was
zero and that it is somewhat embarrassing since you like to think you have diversity among your
friends.”
Ten of the 15 participants reported “none” or “zero” to this question. The other five responses
were “ 1, 1, 2, 3, 10% (of the 6-10 close friends).” No one expressed any sensitivity or
offensiveness about this question.
Question “e” (different social or economic background”
Participants’ responses to this question were varied, both in response to the survey question and
in response to the probe inquiring about the phrase “from a different social or economic
background than you.” The majority included a reference to income or finances when
responding.
One participant focused on “social” background and was thinking about the sexual orientation of
his close friends and that it was different than his. Another participant reported that he had grown
up on a farm and about three of his close friends were like him and the other two weren’t. For
him “growing up on farm” was the “dividing line.” One participant could not answer, responding
“I guess I can tell the difference between a really rich person and a really poor person... but
everything else in-between really blurs... so I can’t really answer that question.” Three
participants mentioned race or ethnicity when responding to the probe.
Page 24
Question difficulty
When retrospective probes were administered about whether any of the questions in the survey
were difficult to answer, three of the fifteen participants mentioned the “friends” questions. But the
responses weren’t necessary negative. One participant said “The most difficult were the ones
about my friendships and their backgrounds and things like that. Difficulty was just in terms of
how specific I needed to be to get what you needed.” Another said “Some of them are
thought provoking.. I think the friends section....how many people you consider your best friends.
And if they’re educated, that’s not a requirement for my friendship so I had to think about who had
what. And the ethnic one.” Another said “Friends’ question took a little more time, the one who
you can tell anything to because some people almost fit that category and I had to think through
that question.”
Question sensitivity or offensive
There was no outward indication of question sensitivity or offensiveness other than what is noted
above, with one exception. During retrospective probing, when asked if any of the question were
sensitive or offensive, one participant said “yes” and responded “That one I was upset about was if
my friends are of a different ethnic background. No other ones.”
When asked if other people might find the questions sensitive or offensive, one participant said
“maybe the one about the different races of your friends. Maybe people would have the same
reaction I did which was surprise that they couldn’t come up with any. I don’t know if it would be
sensitive as much as embarrassing or something. You expect diversity in your friends is something
some people prize.”
3. Recommendations
Question “a”. For round two, use a different response scale for the number of “close friends” with
the highest category being four or more. Suggested response scale: None, one, two, three, four or
more.
Question “b” through “d”. For round two, use a response scale with percentages rather than asking
for an exact number of friends with a specific characteristic. Suggested response scale: 0%, 10%,
25 %, 50%, 75%, 90%, 100%. (Relevant literature will be reviewed to determine if there is
information about response scales using percentages as options.)
Question “e”. Alternative wording needs to be developed for round two testing, but that has not
yet been done.
4. Justification for Recommendations
Question “a.” Responses to the current Question “a” were skewed toward the higher end of the
response scale. We do not know if that was due to desirability effects given the structure of the
response scale. Therefore, we propose testing a response scale with a smaller number of close
Page 25
friends as the highest category during round two..
Question “b - d”. Responses to the current Question “b” through “d” were frequently not
answered in a format which fit with the format requested in the question. (E.g. Percentages
reported rather than exact number of close friends with the characteristic of interest.)
Question “e”. There was not clear understanding of this question. Consideration need to be
given to developing an alternative question wording for this item for round two testing.
5. Final Decision
To be determined.
Page 26
M. Social Trust (Single Question - Version A)
1. Question Wording
We would like to ask a question about how you view other people generally, that is, those
people other than your immediate family and close friends. Would you say most people can
be trusted or that you cannot be too careful in dealing with people?
9
9
Q
9
9
Most people can be trusted
You cannot be too careful in dealing with people
It depends
DK
Refused
N. Social Trust (Two-part question - Version B)
1. Question Wording
We would like to ask a question about how you view other people generally, that is, those
people other than your immediate family and close friends.
(a) How much do you agree with the following statement: “Most people can be trusted.” Do
you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree or
strongly disagree with that statement?
9
9
9
9
9
Q
Q
Strongly agree
Somewhat agree
Neither agree nor disagree
Somewhat disagree
Strongly disagree
DK
Refused
(b) How much do you agree with the following statement: “You cannot be too careful in
dealing with people.” Do you strongly agree, somewhat agree, neither agree nor
disagree, somewhat disagree or strongly disagree with that statement?
9
9
9
9
9
9
9
Strongly agree
Somewhat agree
Neither agree nor disagree
Somewhat disagree
Strongly disagree
DK
Refused
Page 27
2. Results
Two versions ( Versions A and B) of the “social trust” question were administered to all
participants. The Version A “social trust” question contained two extreme positions and
participants were requested to choose between the two. There was an “it depends” response option
which was not read to the participant and was only used if the person could not choose between the
two stated positions and responded that “it depends.” The Version B “social trust” question
contained two items. These items were derived from the single item Version A question. The first
position offered to participants was “most people can be trusted.” Participants were then provided
with an five-point agree/disagree response scale. The second item of the two-part question
provided the second position which was “You cannot be too careful in dealing with people.”
Again, a five-point agree/disagree response scale was provided to participants. The two-part item
was developed to allow participants to choose categories other than just the two extreme positions
offered by the single item question.
Eight of the 15 participants received the Version A “social trust” question (single question) during
the “core” survey and were administered the Version B questions (two-part question) at the end of
the survey when retrospective probes and questions were included. Seven of the 15 participants
received the Version B “social trust” two-part question in the “core” survey and received the single
item Version A “social trust” question in the retrospective section. Summary results for both
versions of the “social trust” question are provided below in Tables 3 and 4.
Table 3 displays responses to the single question version of the “social trust” question. Table 3
provides data for all 15 participants and shows the distribution of responses by when the question
was administered (core survey versus retrospectively.) The distribution is virtually the same with
the exception of the one case for which an “It depends” response was volunteered by the
participant.
The majority (10 of 15) of participants indicated that “most people can be trusted” when the single
question option was used and participants had to choose between two extremes of “most people
can be trusted” or “You cannot be too careful in dealing with people.”
Table 3. Social Trust Question (Single Question - Version A)
Social Trust Question
(Single question)
Core
Questionnaire
Retrospective
Question
Most Can Be Trusted
5
5
10
Cannot Be Too Careful
2
2
4
It Depends (volunteered
response)
1
0
1
TOTAL
8
7
15
Page 28
Total
Table 4 displays responses to the two-part version of the “social trust” question. Table 4 provides
data for 14 participants and shows the distribution of responses by when the question was
administered (core versus retrospectively.) It is interesting that all 7 participants receiving the twopart item in the core questionnaire responded “somewhat agree” for part “a” of the two-part item.
Their responses to part “b” varied. Those participants who received the two-part question in the
retrospective section of the questionnaire had more varied responses to both part “a” and part “b.”
The lack of variability in the part “a” responses is interesting and could be due to social
desirability. It might be worthwhile in round two testing to reverse the order of the two-part item,
presenting the statement “you cannot be too careful in dealing with people” first, and the “most
people can be trusted” statement second and continue to use the five point agree/disagree scale.
Consideration might also be given to reversing the order of the scale so “the disagree options are
presented before the agree options.)Note when looking at the “total” column in Table 4 below, that
the majority of participants agree with both items in the two-part question.
Table 4. Social Trust Questions (Two-part question - Version B)
Social Trust
(Two-part question)
Core
Questionnaire
Retrospective
Question
Total
2
2
3
10
Somewhat Disagree
1
1
Strongly Disagree
1
1
7
7
14
Strongly Agree
4
2
6
Somewhat Agree
1
1
2
1
1
2
4
1
1
7
14
a. Most Can Be Trusted
Strongly Agree
Somewhat Agree
7
Neither Agree nor
Disagree
TOTAL
b. Cannot Be Too
Careful
Neither Agree nor
Disagree
Somewhat Disagree
2
Strongly Disagree
TOTAL
7
Page 29
Tables 5 and 6 below, track by Case ID, the responses a participant gave to the core “social trust”
item(s) they received and the responses they provided to the retrospective item(s). This is probably
most valuable to CNS as they decide how (and if) they might analytically use the data from the
two-part question and what consistency exists between responses to the two versions of the “social
trust” question. The cases are ordered somewhat according to consistency of responses.
Table 5. Social Trust Question - Responses to Core and Retrospective Question(s) by Case
Case
ID
Core Question
(Version A)
Single Question
Response
Retrospective Question
Retrospective Question
Part A - Most People Can
Be Trusted
Part B - Cannot be too careful
in dealing with people
A5
Can be trusted
Strongly agree
Strongly disagree
A1
Can be trusted
Strongly agree
Somewhat disagree
A4
Can be trusted
Somewhat agree
Somewhat disagree
A6
Can be trusted
Somewhat agree
Neither agree nor disagree
A2
Can be trusted
(missing)
(missing)
A9
Cannot be too careful
Strongly disagree
Strongly agree
A8
Cannot be too careful
Somewhat disagree
Strongly agree
A3
It depends
Somewhat agree
Somewhat agree
Page 30
Table 6. Trust Question - Responses to Core and Retrospective Question(s) by Case
Case
ID
Core Question
Version B
Core Question
Version B
Retrospective Question
Part A - Most People
Can Be Trusted
Part B - Cannot be too
careful in dealing with
people
Single Question
Response
B2
Somewhat agree
Somewhat disagree
Can be trusted
B7
Somewhat agree
Somewhat disagree
Can be trusted
B5
Somewhat agree
Strongly disagree
Can be trusted
B6
Somewhat agree
Strongly disagree
Can be trusted
B3
Somewhat agree
Somewhat agree
Can be trusted
B1
Somewhat agree
Strongly agree
Cannot be too careful
B4
Somewhat agree
Strongly agree
Cannot be too careful
3. Recommendations
It might be worthwhile in round two testing to reverse the order of the two-part item, presenting the
statement “you cannot be too careful in dealing with people” first, and the “most people can be
trusted” statement second and continue to use the five point agree/disagree scale. Consideration might
also be given to reversing the order of the scale so “the disagree options are presented before the agree
options.
For the single response item, it could also be interesting to see if response distributions would be
different by switching the order of the two statements.
4. Justification for Recommendations
The majority of responses for both the single item and the two-part item leaned heavily in the “most
people can be trusted” category rather than the “cannot be too careful in dealing with people.” It is
unknown whether there is a social desirability effect or order effect. Switching the order of the two
statements as well as the order of the agree/disagree scale in the two-part item may provide some
useful data.
5. Final Decision
To be determined.
Page 31
O. Knowledge about Federal Laws
1. Question Wording
The last two questions are about federal laws. If you don’t know the answers, just tell me and
we’ll go on.
(a) What individual or group of individuals has the responsibility to make the final
decision on whether a law is constitutional or not - is it the Congress, the Supreme
Court, or the President of the United States?
9
9
9
9
9
Congress
Supreme Court
President of the United States
DK
Refused
3
8
2
2
(b) Do you happen to know how much of a majority is required for the U.S. Senate and
House to override a presidential veto? Is it 51 percent, 60 percent, 67 percent or 80
percent?
9
9
9
9
9
9
51 percent
60 percent
67 percent
80 percent
DK
Refused
2
1
8
1
3
2. Discussion of Results
Participants understood the question being asked as indicated by their paraphrases and responses to
probes. However, only eight of the fifteen participants had the correct answer. And several of
those with the correct answer expressed uncertainly in their response by using qualifiers such as “I
think” or by indicating that they guessed at their answer when they were asked how they arrived at
their answer. Three participants volunteered that they did not know the answer.
During discussion of the item, a few participants expressed a bit of embarrassment that they might
not know the correct answer.
3. Recommendations
There is not enough data to determine whether or not the questions are problematic. They are
“test-like” questions which may cause some discomfort. The questions should be tested further in
round two
Page 32
4. Justification for Recommendations
Some participants expressed embarrassment at their lack of knowledge needed to answer the
questions correctly. To determine if this might present problems in the field, the questions should
be included in round two. Perhaps some additional probes should be added to gain more insight
into what participants think about these questions.
5. Final Decision
To be determined.
Page 33
P. Retrospective Probes About the Survey Questions
Once the survey questions were completed, there were additional retrospective questions administered
to participants in order to gauge their overall opinion of the survey questions, get a sense of whether
there were any issues related to question difficulty, sensitivity, and appropriateness of questions for a
Census Bureau survey. We also inquired about participants’ thoughts on proxy reporting for other
household members. An entire series of questions was administered to determine participants’
understanding of the reference period “typical month” which was used in six of the survey questions.
1. Overall Opinion about the Survey Questions
The 15 participants were asked the overall opinion about the survey questions. Responses were
overwhelmingly positive. Responses included the following:
“They really made me think. I was really thinking hard about what you were asking me.”
“This is... I’ve done one before. Their question was a little more mind boggling and confusing. I think
it was pretty cut and dry.”
“Seemed fine.”
“Uh.. As I said, there were some points of embarrassment that this is information (the last two
questions) that I should know from going to school. I think people might be embarrassed and just go
look things up.”
“Those were nice questions. Those were questions that should be everyday questions to people.
Basically about governments, schools and organizations, which everyone should get involved
basically.”
“They were great.”
“It was easy.”
“The were pertinent and directly related to the topic.”
“I liked the survey. It was interesting. I thought it covered a variety of topics. I thought the questions
were very clear overall. I didn’t...it was more struggling with my own retention if I didn’t have it in
front of me rather than trying to interpret the question.”
“I thought they were very interesting. It made me kind of think about how there’s maybe more to
explore out there than I have explored.”
“Interesting... I appreciate the opportunity to help you guys do this so that you can narrow the
questions down... so that there can be a precise as then can be for the wide group of people that would
be answering these questions.”
Page 34
“I thought they were fine. I thought some of them were pretty clear cut.”
“I think they were excellent questions and I enjoyed the survey myself. I am not sure everybody else
will.. But it depends on who you are.”
“Very interesting. First of all it makes me think about what and what I don’t’ know about politics.”
“I thought it as pretty simple. I do not think there was anything complex.”
2. Sensitivity Issues
Only one of the fifteen participants responded that they did find the questions sensitive or
offensive. When probed about which questions were offensive, the participant responded “The one I
was upset about was if my friends are of a different ethnic background...no other ones.”
When asked if other people might find the questions sensitive or offensive, the following responses
were provided by 4 of the 13 participants:
“Um, I mean maybe the ones about the different races of your friends. Maybe people would have the
same reaction I did which was surprise that they couldn’t come up with any. I don’t know if it would
be sensitive as much as embarrassing or something. You expect diversity in your friends is something
some people prize. “
“Uh, Maybe when people are overly religious, they may have questions about why you’re asking how
often they go to church or something like that. But I think that’s a very small percentage. “
“Delineating between church meetings and church attendance might be sensitive for some people. But
everything else no.”
“Not offensive. Sensitive maybe”
3. Difficulty Issues
Four of the fifteen participants indicated that they found some of the questions difficult to
answer. When probed about which questions were difficult, the questions about federal laws and
friendships were mentioned..
4. Proxy Issues
Of the 10 participants administered the retrospective question inquiring if they would be able to
answer the survey questions about everyone that lives in their household, eight of the ten said
they would. One of these participants did state that the “opinion questions” would be guess work if he
responded for other household members.
One participant who stated that he would not be able to answer the survey questions about everyone in
Page 35
the household indicated that the questions which may be problematic were the questions about
“politics, the social clubs and the topic on trust.”
When asked if they thought one household respondent would be able to answer the questions about
everyone who lived there, four reported yes and four reported no.
5. Willingness to Respond
Of the ten participants asked if they thought other people in their household would be willing to
answer the survey questions, three of the ten indicated other people probably wouldn’t be willing,
but the reasons were not specific to this survey. Reasons given had more to do with other people in
the household not interested in participating in general.
6. Participation in Census Bureau Surveys
All fifteen participants responded “no” when asked if there was anything about the questions
that might cause you not to want to participate in Census Bureau surveys.
7. Appropriateness for a Census Bureau Survey
When asked if they thought the questions were appropriate for a Census Bureau survey all
responded “yes.”
8. Reference Period Issues
a. Typical Month
Six of the survey questions used a reference period of “typical month.” During retrospective
probing, participants were asked how they interpreted that time frame. Several of the participants
didn’t understand the probe and just reported what a typical month is (e.g. From the 1st to the 30th
or 31st.) Some participants thought of “last month”. One who mentioned last month said that had
last month been atypical in some way, she would have provided different answers. Another
participant thought about the questions about neighbors and stated that he was thinking of months
when you see your neighbors so he was reporting for months like May and June. Another said she
thought about the last year and divided it up.
Participants were asked if the reference period wording had been “During a TYPICAL MONTH in
the past year,” if that would produce a different understanding. Four participants stated it would
change their answers, three said it would not change their answers and one said he/she didn’t
know, stating it would be situational. However when they explained their answers it was again
clear that several participants did not understand what was being asked as they responded “any
month during that timeframe. 30 days,” “ I kind of think 30 days.” Clearly the probes attempting
to get at participants’ understanding of the reference periods need to be improved!
Page 36
b. Number of Days Corresponding to Response Options for Typical Month Questions.
To determine participants’ interpretation of the response options used in the six questions which
asked about activities in a typical month, participants were asked to state the number of days they
think of when they heard the various response options.
The response options used in the questions about “typical month” activities included the following:
Basically every day
A few times a week
A few times a month
Once a month
Not at all
Participants seemed to have a relatively common understanding of the response categories.
Basically every day - This response option was interpreted as low as five days a week to a high of
seven days a week.
A few times a week - Interpretations of this response option ranged from a low of two days a week
to a high of “4 or 5" days a week.
A few times a month - Interpretations ranged from two - six times per month. One participant was
outside this range with a response of “10 - 12, a couple days each week.”
Once a month - Interpretations were exact and consistent - once a month.
Not at all - Interpretations were exact and consistent - none.
While the probes administered to obtain information about the reference period “typical month” were
not designed well enough for participants to articulate their interpretations, the extensive discussions
about the survey items and participants’ responses to them provide no information to lead us to think
the “typical month” reference period was problematic.
Page 37
| File Type | application/pdf |
| Author | owner |
| File Modified | 2007-11-06 |
| File Created | 2007-11-06 |