Cognitive Testing Report
2013 NCCT OMB submission Attachment D cognitive testing report.docx
2013 National Census Contact Test
Cognitive Testing Report
OMB: 0607-0972
Test 15 Cognitive Testing Report DRAFT
Test 15: Alternative Contact Test Cognitive Testing Report
Michelle
Smirnova & Paul Scanlon
Introduction
Test 15 seeks to test the validity of an alternate contact frame, which will be used in later tests during the 2020 Census Research and Testing phase. Alternative contact methodologies that are sought to be researched are phone calls to cell phone and landlines, text messages to cellphones, and emails. Test 15 will be conducted over the phone using a Computer Assisted Telephone Instrument (CATI) with 10,000 respondents between January 7th 2013 and February 1st 2013.
The Test 15 questionnaire takes the approach of requesting email addresses and phone numbers from the respondents; which would later be matched to the Alternate Contact Frame. Because of potential sensitivities, it was determined that the production questionnaire would not attempt to verify the information on the Alternate Contact Frame with respondents.
This study sought to verify whether or not the questionnaire developed for Test 15 would be successful in eliciting phone numbers and email addresses to validate against the Alternate Contact Frame. This report outlines the results and analysis of the in-person cognitive interviews that were conducted in August of 2012 in the Washington DC metropolitan area.
Email and phone call strategies were also directly tested in this study in the recruitment efforts. A list containing phone numbers and email addresses associated with sampled housing units was used by recruiters to contact potential respondents and request their participation in this study. Our sample of 500 housing units proved to be generally unsuccessful in yielding a workable sample for cognitive interviews. Although there are a number of factors that may have contributed to the non-response of this sample, over half of the email addresses and phone numbers were inaccurate, which may give us some insight into the quality of the particular list used in this particular study. The outcome of this strategy is discussed in greater detail in the results section of this report.
Cognitive Interviewing
Cognitive
interviewing is a qualitative methodology that offers the ability to
understand the cognitive process behind answers to survey questions.
There are a variety of cognitive interviewing techniques—some
concurrent to the survey administration and others that are
retrospective, and ask questions about the process at the very end.
The Test 15 employed a combination of concurrent and
retrospective probing. Probes were semi-scripted and intended to
cover certain pre-identified topics, but also allowed the interviewer
the flexibility to follow unanticipated problems that surfaced. The
goal of this method is to understand what the respondent was thinking
when answering and how they interpreted the meaning of the question.
The researchers then use the information to identify which questions
could benefit from revision and how they might be revised.
Cognitive
interviews are often focused on three of the four stages of the
cognitive process model - respondent comprehension, information
retrieval, and response (Cannell, Miller, and Oksenberg, 1981;
Tourangeau, Rips and Rasinksi, 2000). The most common probes in
cognitive testing are paraphrase and meaning probes, which elicit
comprehension issues, and process probes that elicit retrieval
problems (Willis, 2005). Response issues are often observed by
watching the respondent attempt to categorize their answers onto the
available response categories. Judgment issues, i.e., when the
respondent comes up with the appropriate response in their mind, but
chooses a different answer based on sensitivity or social
desirability, may go undiscovered in cognitive interviews because of
social and sensitivity issues that may persevere in the cognitive
interview despite attempts to generate rapport with the respondent
(see Willis, 2005 and Childs and Smirnova, 2012). In Test
15 we were particularly interested in how silence or the active
choice to not respond or provide limited information, might be used
when requested information was perceived to be sensitive or private.
Interviewers probed on these issues, asking a combination of direct
information collecting questions, attitudinal and hypothetical
questions. As such, our focus was primarily on retrieval and
response stages of the cognitive process model, though in some
situations, focus was also on the comprehension stage.
Because
we were concerned that the questionnaire may not elicit the requested
data, we additionally used a “verification strategy” at
the end of the protocol. Using the list of cell phone numbers and
email addresses from the Alternate Contact Frame, we confirmed the
information the respondents had already gave during the interview and
probed them on the information found in the list that they had not
already mentioned. Because we suspected this verification strategy to
be a topic of sensitivity, interviewers had leeway in whether or not
this portion of the interview was administered.
Respondents
The
twelve interviews conducted for this study were conducted throughout
the Washington, DC metropolitan area. Although four zip codes were
initially targeted, poor response altered recruitment methods and
resulting household locations. Despite this, the zip codes included
in the sample ended up being diverse in their basic demographic and
economic characteristics, as measured by Census programs, and
provided a measure of diversity across the pool of potential
respondents. More detailed information of demographics can be found
below.
Table 1: Demographic summary of respondents by gender and age (N = 12)
Total Percent
Gender
Female 8 66%
Male 4 33%
Age
Under 40 years 7 58%
40 years and over 5 42%
Race
White 5 42%
Black 5 42%
Asian 1 8%
Hispanic 1 8%
Education
High School or Less 1 8%
Some College 4 33%
College Degree 2 17%
Graduate Degree 5 42%
Results
Overview of Findings
Using a telephone number from the Alternate Contact Frame lists, recruiters called respondents and asked whether they would be interested in participating in a research study conducted by the Census Bureau. The recruiters informed the respondents that they would be compensated forty dollars for their participation in this hour-long interview. Following an initial attempt to call respondents, the team emailed the same respondents using the same lists. The call and email script are attached in Appendix C. The outcomes of these calling and email attempts are documented in Table 2 and Table 3 below.
Table
2. Phone number responses
|
Out of Service |
No Answer |
Left Message |
Refusal |
Fax |
Scheduled |
Total Called |
Recruiter 1 |
33% (66) |
11% (21) |
34% (69) |
11% (22) |
3% (5) |
2% (4) |
187 |
Recruiter 2 |
24% (12) |
46% (23) |
10% (5) |
12% (6) |
8% (4) |
0% (0) |
50 |
Recruiter 3 |
46% (32) |
10% (7) |
25% (28) |
10% (7) |
1% (1) |
3% (2) |
70 |
Total |
36% (110) |
17% (51) |
33% (102) |
11% (35) |
3% (10) |
2% (6) |
307 |
Table
3. Email Responses
|
Bounceback |
Total Emailed |
Recruiter 1 |
54% (109) |
202 |
Recruiter 2 |
60% (85) |
142 |
Recruiter 3 |
48% (119) |
248 |
Total |
53% (313) |
592 |
Because of the high non-response rate of both calling and emailing strategies and the short amount of time we had to do the study, we altered our strategy in the latter stages of recruitment. We posted an advertisement on Craigslist requesting participants in a Census Bureau study for which they would be compensated $40. These respondents called our recruitment line and responded to a set of screening questions in which we asked for their full address. Based upon this information, we conducted a reverse-look using the Alternative Contact Frame to obtain the cellphone numbers and email addresses listed for the particular housing unit. We used this information at the end of our cognitive interviews, during the verification stage. Below is a question-by-question analysis of all those questions on the questionnaire that posed some difficulty for respondents or for which there were interesting findings that may be relevant to the administration of Test 15 in 2013.
The full questionnaire is available in Appendix B.
Question-by-Question Analysis
Only those questions that displayed difficulty during cognitive testing are included in this section. For more information on the full questionnaire, please refer to Appendix B
May I speak with someone at least 15 years old who lives here and knows about the people in this household? Would that be you?
Yes
No – Ask to speak to this person. If such a person is not home, Thank the respondent and hang up.
Respondents did not think this question was appropriate. Several indicated that unless the caller immediately identifies her/himself as a representative of the Census Bureau calling on behalf of “an important study being conducted by the Census Bureau,” that they would probably hang up.
Hello, I’m (your Name) from the U.S. Census Bureau. We are doing research to make it easier for people to respond to the Census. In the past, for the Census, we have mailed most households in the United States a paper census form and visited people who did not respond which cost a lot of government money. In an effort to reduce cost, we are researching alternative ways of contacting people.
Your responses to these questions are protected by law under Title 13 of the United States Code. This is the same law that protects your privacy. No one outside the Census Bureau will see your answers. This interview will take about 7 minutes to complete.
Respondents thought that this introduction was wordy. If we could reduce the length of the opening statement, it might reduce break-offs early in the phone call.
Does anyone in this household share a cellphone?
Yes - Who shares a cellphone?
No
Don’t know
Refusal
Respondents generally understood that this question was referring to a physical device. When probed as to what specifically this question was asking, most respondents said, “It is asking whether two people share a phone”. Only two people thought that this may refer to a calling plan, but also thought it meant the same device.
FOR EACH YES (person that has their own phone or shares a phone):
Do you/does NAME use the cell phone for texting?
Yes
No
Don’t Know
Do you/does NAME use the cell phone to access the internet?
Yes
No
Don’t Know
What is your/NAME’s cell phone number?
____- ____- _________
Each of these questions was not particularly difficult for respondents to answer, though some were reluctant to provide the actual number of other people. This was particularly true in households with tenants or roommates. We prompted them to look in their cellphone if they could not recall them from memory, but some respondents maintained that they did not have the number of the other person in their household. It is unclear whether they did not have this information or whether they felt it inappropriate to disclose someone else’s personal information. Other cognitive tests have revealed similar findings that respondents are hesitant to provide contact information for other people (see Childs and Smirnova, 2012) All respondents besides one offered their own cell phone number.
Are there any other phone numbers for this household, perhaps a work phone number or an additional line?
Yes <go to question 17>
No <go to question 18>
Don’t Know <go to question 18>
Refusal <go to question 18>
All respondents understood this question. When asked for an example, most were thinking of another line that might be used to operate a business out of a home, but one said “maybe a ‘magic jack’ or something like that where you can plug in another line?” There were multiple respondents who used Skype and GooglePhone, but none of them had a separate number for that line, “I just do that through my internet,” said one respondent.
Are any of these cell or landline numbers unlisted or unpublished?
Yes
Which ones?
No
Don’t Know
Refusal
A number of respondents were unsure whether their numbers were unlisted or unpublished, but assumed that there was some action that they would have to do in order to make them unlisted or unpublished. One respondent said, “that means you have to pay someone money to take your number out of a phonebook or other listings, and living on a fixed income, I know we do not do that.” Another said that, “I do not think we are listed in the phonebook, but I don’t think we did anything to make that happen,” and similarly, another person said that “I Googled myself and couldn’t find my number, but I don’t think I am unlisted or unpublished everywhere. Really I am just not sure.” One respondent questioned, “If these numbers are unlisted, then how would you be able to contact me?” She seemed worried that the Census Bureau was being “sneaky.”
This question did cause a number of respondents to think aloud about telemarketers and “spam calls.” One respondent said that she had not actively unlisted her numbers, but that was because she did not receive many unwanted calls. She and others indicated that they would consider taking the time to un-list the numbers if this amount noticeably increased. Returning to this topic at the end of the interview, respondents largely indicated that they would not consider calls from the Census Bureau to be unwanted “spam,” though their caller IDs would have to show that the call came from the Census Bureau for them to be sure. Another respondent, for instance, never picks up calls from people she does not know, but answered the recruiting call because her caller ID indicated the call came from Census Bureau Headquarters.
It might be worth noting that this question caused more sensitivities than many of the other questions in this survey. If it is not a necessary item, then perhaps it should be omitted.
20.
What are your email addresses?
____________________@_______.____ How often? _______________ _H / W
____________________@_______.____ How often? ________________H / W
____________________@_______.____ How often? ________________H / W
____________________@_______.____ How often? ________________H / W
____________________@_______._____ How often? _______________H / W
____________________@_______._____ How often? _______________H / W
About how often you check your email (insert response in lines above).
Let’s look at the addresses that you provided. Which ones of these are work email addresses? Personal or home addresses? (Circle H or W ) Do you sometimes check work email at home? Home email at work?
Some respondents willingly provided all of their email addresses, while others were more reluctant. Some said explicitly, “the one that I give out is…”, or “the one I will share with you is…” or “my main address is…” which indicates that there were other addresses that the respondents did not feel comfortable sharing with the Census Bureau because they were personal, or perhaps because they did not check them. One respondent said that, “I have lots, but only one that I will give you. I like to divide my life, one email address is for old friends, one is for my non-profit work, and another is for the neighborhood. My husband is the same. I will give you my ‘public’ emails.” Respondents varied in their rationales for providing limited lists. Some would tell us the domain of the other email addresses they had, but did not want to provide the exact address.
What is NAME’s primary email address or addresses?
____________________@_______.______
____________________@_______.______
____________________@_______.______
How often do you think NAME checks address <1-3>?
Most days
Most weeks
Occasionally
Don’t Know
Refusal
Similar to the findings in regards to willingness to share cellphone numbers, respondents were reluctant to provide the email addresses of other members of their household. Respondents could easily provide details of how often others in the household checked email and whether they did so from home or from work, but still did not provide the address. Younger respondents tended to check email both at home and at work whereas older respondents tended to have some email addresses they checked at home and others they checked at work. As an example, one respondent indicated that it was not difficult to recall whether or not her daughter and tenant use their phone to text or email because, “I live with them. My daughter texts more than I would like her, too, but I do not let her use the phone for the internet. And I coordinate dinner and other things with [tenant] so I see how often she uses her email address and how often she responds or uses her phone.” In addition, not all respondents had the ability to answer at least the first part of this question. Another respondent had no idea what her tenants’ emails were, because she interacts with them either in person or over the phone. She guessed that they (both in the mid-20s) check their email everyday, but was not familiar enough with their habits. A third respondent noted that she was sure she did not know all of her daughters’ email addresses, “I know their phone number because I pay for them, but I’m sure they have emails I don’t know about.”
As mentioned earlier, in the past the Census Bureau has mailed most households in the United States a paper census form and visited people who did not respond which cost a lot of taxpayers’ money. In an effort to reduce cost, we are considering alternative ways of contacting people. One such way would be to contact people directly on their cell phones. How would you, personally, feel about the Census Bureau calling you on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against, or
strongly against it
Most respondents understood this question. There was concern among a number of respondents about how many minutes would be spent on responding to such a survey on their cellphone. In these cases they felt as though the Census should find a way to pay for the minutes, otherwise they should only call a landline or email. Two of the older respondents (55 and 72) were very opposed to being on their cellphone ever in that they could be caught an inappropriate time, “you could call me in the car” or “when I am in the market, in a store, or sometime when I cannot respond to a survey.” Further, the latter initially interpreted this as a text message, though she was opposed to either a phone call or a text message. She explained, “this costs us a lot of money when you text us. And the screen is too small anyway; we could not fill out the form on the screen.” When I asked her “what if this question meant we would call you?” she said, “I don’t want you to call me on my cellphone because that will also cost me money. To conduct that survey over the phone it would take 20 minutes and that is expensive for people who do not use their cellphone that often. It costs money per text messages and I also think it encourages bad behavior on the road while driving.” She still seemed a bit confused after our discussion, but I think this was due to her strong opinions about text messages and the Census Bureau contacting respondents via their cell phones.
Other respondents appeared to understand this question to mean that the next decennial census would only use cell phones to contact the populace. One respondent said, “I don’t use my cell phone, so you would miss me. Just use paper.” Another indicated that she was worried she would not be counted, but if the cell phones were just used to follow up the normal mail questionnaire, she would be fine with it.
Another way of contacting people would be for the Census Bureau to text you a link to complete your census form. How would you, personally, feel about the Census Bureau texting you a link on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against, or
strongly against the Census Bureau texting you?
There was only a slight confusion among respondents in regards to this question, though we believe that this was due to respondents’ strong opinions towards text messages and hasty responses before the completion of the question. Most respondents were opposed to the Census Bureau texting them. One said that “I only use texting with friends,” while others said that they never texted and that they would not know what to do with a text, they would probably delete it if they were able to read it at all. A number of respondents were similarly concerned about the cost, one respondent explains,
R: “As long as you aren’t texting me each individual question in like 20 texts and asking me to text back. I think this is a great idea. Is that what you will do? Text each question?”
Interviewer: “No, we would just text you the link”
R: My only concern is for those people who do not have unlimited texting plans. Like I have an unlimited plan, but my daughter does not. If the text becomes a financial burden to the respondent then that would be bad. But if the Census can pay for it or negotiate something with the provider and let the people know that they are not paying for it, then I think this should work well.”
Another respondent was against this strategy explaining that,
R: “There is no reason to switch media, if you text me a link you are creating an extra step, just email me. That way I can fill out the form. I hated filling out the form in 2010. By 2020 paper will be completely outdated, everything will be online. You should do it that way but don’t mess with our cellphones.”
A way to clarify that we do not plan to text message respondents the entire survey, question by question, might be to add the language “text you a link to complete your census form online,” to clarify that the survey would not be conducted through text messages, but rather would be used to direct respondents who have smartphones to an online form. For those who do not have smartphones, they could manually type in this address to a web browser on another computer.
Similar to the cell phone question above, some respondents repeated their fears that this new technology would cause the census to overlook people who do not text:
R: “I never turn my phone on. If you text me, I won’t see the form. The Census should count everyone, and I think if you do this, then you’ll miss a large part of the country that does not text.
Interviewer: “What if this was just one way out of many to contact individuals?”
R: “I would be strongly in favor then. As long as I know I can be counted some way.”
Another way of contacting people would be for the Census Bureau to email you a link to complete your census form. How would you, personally, feel about the Census Bureau emailing you a link on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against,
or strongly against the Census Bureau emailing you?
We suggest that the “on your cellphone” part of this question be dropped. For those respondents without a smartphone (and even for some of those with one) found this confusing and often thought it was repeating the question before it (Q24). An example of this confusion is expressed in the exchange below:
R: “Isn’t that the same thing as texting? If you are emailing my cellphone? If you directly email my email, I’d be all for it, but not my cellphone.”
Interviewer: “What if we meant emailing your email?”
R: “Then I would be OK with it. That is how I handle most of my business related matter.”
Interviewer: “Would you question its legitimacy?”
R: Yeah, I would. But I would go to the site first to make sure it is official.”
Another Respondent who is in favor of emailing said,
R: “You are already sitting in front of the computer, you are in the mindset. You can click on the link and continue with the process. I think this will give a higher response rate. This is the way 2020 will be done.” I asked the respondent if she would trust the link or would there need to be something to make her trust it. She responds that “it would be trustworthy if it was from a .gov email address and a campaign announcing a multimedia campaign strategy. The Census would have to make it clear that this was how they planned to conduct the 2020 Census. I think there are several options for how this could work for people. First would be for the Census to issue everyone an email address to use for the system. This is the only way, unless you would buy the email addresses, you don’t plan to do that, do you?” I say that we are unsure what we will do at this time, she continues that, “the second option would be to get information from the IRS.” I ask her if she would be comfortable with this and she laughs, “Oh yeah, it is government sharing with the government (laugh) and besides I’m semi-retired, so I have nothing to hide, I don’t make anything.”
Even those that did understand that an email sent to an account appears on both their smart phones and their computers indicated that they would prefer to fill out the form on their computers. For instance, one respondent said,
R: “You could email me on my cell phone I guess, but I don’t see why I wouldn’t fill it out on my computer. My computer obviously has a larger screen, and it’s easier to type on. I guess I could use the cell phone to fill it out anywhere, but I think I would just wait until I got home and did it on my laptop.”
In order to count people in the right place, the Census Bureau need detailed geographic information. To assist in our efforts to increase the accuracy of the census and potentially save a lot of taxpayers’ money, we could get some of your geographic information from cellphone providers. Would you be willing to allow the Census Bureau to obtain geographic information from your cellular provider?
Yes
No
Don’t Know
Refusal
This question yielded very unfavorable responses. Although attitudinal responses varied in regards to how the respondent interpreted “geographic information,” most respondents opposed it. For those who thought “geographic information” referred to a billing or mailing address, they were generally in favor, though only five people in our sample interpreted the question that way. For those who interpreted it as “GPS or tracking information,” they were opposed to the Census Bureau collecting such information from cellular providers. Some excerpts from respondents’ rationale include:
R: “That makes me very uncomfortable, especially with the recent changes with facebook and Google privacy policy. I don’t use Google maps because then they are able to get my location, I don’t like the idea that they know where I am. I assume that you would ask my provider to look at the towers that get pinged most frequently or which tower was used when I was on the phone with Census or when I was filling out the form. This is creepy. The government knows my social security number and that is OK, but they do not need to know where I am at all times. If you were just asking for my billing address from the phone company, then that would be OK.”
And:
R: “No. I don’t like having the cellphone company have that much control over me to say where I am. I thought that meant tracking. Most people tend to think of how I can pull up google maps which tracks your location and then that would be the geographic information.”
Interviewer: “What is geographic information meant billing address?”,
R: “I might be more favorable in that sense.”
However, some respondents were more favorable:
R: “Yes. Of course, there is so much sharing already and this is critical data. When I ask her what she thinks is meant by “geographic information,” she replies that it is “where I live, where my phone is registered, my billing address, you can glean a lot of information from this. Of course I do not want you to know where I am at all times, but that information is OK.”
Others were unsure of what the Census Bureau could actually get from cell phone providers, indicating that they did not understand the process behind the census and why the Census Bureau needed the geographic information:
R: “What are you going to find from the cell company? Like where I shop and go to work? I don’t understand why you’d want to know that. I guess I trust you, but I don’t get it”
Administrative
Records Matching
At the end of the cognitive interview, interviewers referred to a sheet with listed administrative records data on the respondent. These lists included the housing address, and all phone numbers and email addresses associated with that address from the Alternate Contact Frame. The interviewers compared this list with the data provided by the respondent during the interview and discussed the discrepancies with the respondent. In most of the cases, the phone numbers and email addresses between the administrative data list and the self-reported list did not match up (See Tables 4 and 5). In fact, many respondents did not even recognize most of the phone numbers or email addresses associated with their home address. A number of respondents explained that they lived in houses/apartments with high turnover rates, and therefore even if the information were only a year old it would not be accurate. Others vaguely recognized the numbers as old landlines. A number of respondents guessed that some of the numbers could have been old pre-paid cell phones that they or their children used at some point in the past, but that are no longer active.
In an interesting case, one respondent lives in the house in which she was raised. Her parents have since moved out and sold the house to a new landlord. As such she recognized all five phone numbers and the two email addresses, though she noted that only three of the five phone numbers are still in use (two by the landlord and one by her). Of the email addresses, one belongs to the owner and the other belongs to the respondent’s mother, who has since moved out.
For these reason we believe that a question on length of stay at the residence should be added to the questionnaire so that we can measure the accuracy of the administrative records data for both those respondents who have lived in a house or apartment for some time versus those who are more transient in their stay. Tables 4 and 5 illustrate the accuracy of these lists in the cognitive test.
Table 4. Data Quality of Phone Numbers
RID |
# of Administrative Records Phone Numbers Listed for MAFID |
# of Administrative Records Phone Numbers Recognized by Respondent |
# of Phone Numbers Provided by Respondent |
# of New Phone Numbers provided by Respondent that were NOT on Admin Records List |
R1 |
2 |
1 |
3 |
2 |
R2 |
2 |
0 |
3 |
3 |
R3 |
0 |
0 |
5 |
5 |
R4 |
2 |
1 |
1 |
0 |
R5 |
5 |
2 |
4 |
3 |
R6 |
9 |
0 |
1 |
1 |
R7 |
5 |
4 |
4 |
4 |
R8 |
3 |
1 |
2 |
1 |
R9 |
2 |
1 |
4 |
3 |
R10 |
5 |
2 |
5 |
3 |
R11 |
6 |
4 |
2 |
2 |
R12 |
14 |
3 |
8 |
6 |
Total |
55 |
17 (31%) |
42 |
33 (79%) |
Table 5. Data Quality of Email Addresses
RID |
# of Administrative Records Email Addresses Listed for MAFID |
# of Administrative Records Email Addresses Recognized by Respondent |
# of Email Addresses Provided by Respondent |
# of New Email Addresses provided by Respondent that were NOT on Admin Records List |
R1 |
1 |
1 |
2 |
2 |
R2 |
6 |
0 |
1 |
1 |
R3 |
0 |
0 |
4 |
4 |
R4 |
1 |
0 |
0 |
0 |
R5 |
8 |
7 |
2 |
1 |
R6 |
4 |
0 |
2 |
2 |
R7 |
2 |
2 |
3 |
3 |
R8 |
2 |
2 |
2 |
1 |
R9 |
0 |
0 |
1 |
1 |
R10 |
1 |
1 |
3 |
2 |
R11 |
3 |
2 |
2 |
0 |
R12 |
4 |
4 |
4 |
2 |
Total |
32 |
19 (59%) |
26 |
19 (73%) |
From tables above, we can see that only 31% of the phone numbers listed in our administrative data records file for each MAFID were recognized by the respondent. Email addresses associated with a MAFID were recognized on average 59% of the time. In comparison with information reported by the respondent, 79% of the phone numbers and 73% of the email addresses provided during the interview were new, i.e. they the administrative file did not include them. For those situations where respondents recognized an administrative records email address, but had provided another one in the interview, they explained that addresses we had on file were either SPAM email addresses (addresses created for non-correspondent purposes that are generally not checked frequently) or addresses they have not used in some time. The phone numbers also seemed generally inaccurate in comparison to the information provided directly by respondents.
Recommendations
1.
Reduce the Title 13 explanation at the beginning of the call.
Respondents complained that this introduction was lengthy as is,
though those who had questions later in the survey returned to the
rationale in the opening message as an explanation for the purpose of
this data collection.
2.
Have callers identify themselves as Census Bureau
representatives at the beginning of the call. We
need to make sure that the call centers’ procedures begin with
the interviewer immediately identifying themselves as Census Bureau
staff. A number of respondents throughout the cognitive
interviewing phase indicated that they needed to hear this
immediately or they would discontinue the call.
3. Add “Text a link to complete your census form online” to question 24. There were a number of respondents who were confused by this question, some who thought that the Census would text them each question in a separate message, to which they would have to respond “twenty plus” times. They thought this would be inefficient, costly, and potentially promote bad behaviors such as texting while driving. It is important to specify that the text message will contain a link to a survey they can complete online—on their smartphone if they have access to the internet, or on another computer if not.
4.
Ask
respondents about their attitudes toward the Census Bureau emailing
them, not about emailing them on their cellphone.
We recommend that on Question 27, the qualifier, “on
cellphone” be removed from the question:
“What if we emailed you a link on your cellphone?”
For
those who do not have a smartphone (and even for some of those who
did), this sounded as though it were referring to a text message.
When asked this attitudinal question, several respondents
looked at the interviewer quizzically, “Isn’t that what
you asked in the last question?” The question should
focus on comfort with emails from the Census Bureau, not about which
device they will access this email.
5.
Add a question about the length of stay of the respondent in the
particular household.
For those respondents who indicated that they rented their
apartment/house or have only been there for a year or two, the
administrative data we had for them was generally inaccurate. The
lists of phone numbers and email addresses tended to be longer for
these respondents, but there were almost no matches. This is
probably a result of a high turnover rate at this household or
perhaps outdated data that will not help us in our imputation or
contact strategies.
6. Confirm or request domains of email addresses as opposed to actual email addresses. In our interviews, most respondents did not appear uncomfortable or “creeped out” by the administrative data information that we had associated with their MAFID. That said the population of cognitive interviewing respondents is not representative of the US population or telephone survey respondents. These respondents agreed to meet a Census employee for an hour and they were paid $40 for their time and travel. Even among these respondents there were a number who refused to provide all (or any) of their email addresses. A number of them indicated that they were more comfortable simply providing the domain. As such, we recommend that a matching question be asked in regards to domains. The caller should ask if the respondent has email addresses at the domains for which the Census has data or, more simply, which domains they have email on. This can serve a similar goal of verification without making the respondent uncomfortable.
7. Add a question asking on which type of device respondents would fill out a Census Bureau survey. In order to understand on which device respondents are most likely to fill out a survey, we should add a question on whether they would fill out a questionnaire on their phone or on a computer if they received the link via email or text message. Would they manually type the link or would they click on it directly if they received this on their smartphone?
8.
Provide telephone interviewers with a supplemental script that will
help explain the survey’s purpose. If
the research team decides to eliminate language promising that the
information collected during the survey will not be used to contact
them in the future (such as phone number and email addresses), we
suggest that the telephone interviewers be given a supplemental
script to use at their discretion. They should use this script if
the respondent interjects during the survey asking why the Census
Bureau needs their cell or landline numbers or email addresses. We
purpose the following language, “We do not plan to use the
information you provide today to directly contact you in the near
future. However, we are asking for this information in order to
verify the quality of our contact strategy for the 2020 Census.”
Appendix A
Bibliography
Cannell, C.F., Miller, P.V, Oksenberg L. 1981. “Research on Interviewing Techniques” in S. Leinhardt (Ed). Sociological Methodology. San Franscisco, CA: Jossey-Bass.
Childs, Jennifer Hunter and Michelle Smirnova 2012. “Using Cognitive Interviewing to Detect Privacy and Confidentiality Concerns.” Presented at the Joint Statistical Meetings, July, San Diego, CA.
Tourangeau, R. Rips, L. and Rasinksi, K. 2000. The Psychology of Survey Response. : Cambridge UK: Cambridge University Press.
Willis, G. 2005. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications.
Appendix B
Alternate Contact Information Test Protocol
July 11, 2012
Note: Probes in yellow will be asked directly after the question is answered. Those listed in grey will be asked after the entire interview is completed: the interviewer will go back and probe further on these questions.
Introduction:
Hello, I’m ______________ and I work for _____________. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing. Today we are working on some questions that will be asked of people in a telephone survey. We are very much interested in your own opinions and thoughts today. There aren’t any right or wrong answers, I just want to hear things from your perspective. After I ask you the survey questions, I’m really interested in what the questions meant to you, what you think they are getting at, so I’m going to ask you questions about the survey questions. There aren’t any right or wrong answers there either. I’m just really interested in your thoughts.
Permission to Tape-Record:
Because it would be hard to keep track of everything you say today, I’m going to tape-record this session. HAND RESPONDENT CONSENT FORM. Please read this over and sign it. I want you to know that your responses will be kept strictly confidential and will only be seen by people working on this project. The tapes will be destroyed after we conduct our analysis. Your participation in this study is completely voluntary and you can decline to answer any particular question.
Do you have any questions before we start?
I am going to pretend that I’m calling you on the phone for this part of the survey.
May I speak with someone at least 15 years old who lives here and knows about the people in this household? Would that be you?
Yes
No – Ask to speak to this person. If such a person is not home, Thank the respondent and hang up.
Hello, I’m (your Name) from the U.S. Census Bureau. We are doing research to make it easier for people to respond to the Census. In the past, for the Census, we have mailed most households in the United States a paper census form and visited people who did not respond which cost a lot of government money. In an effort to reduce cost, we are researching alternative ways of contacting people.
Your responses to these questions are protected by law under Title 13 of the United States Code. This is the same law that protects your privacy. No one outside the Census Bureau will see your answers. This interview will take about 7 minutes to complete.
**All Questions are to be blind coded unless otherwise noted**
First, I need to verify your address. Is this _______________?
Yes <go to question 7>
No <go to question 4>
Don’t know <go to question 4>
Refuse <go to question 4>
Are you familiar with this address?
Yes <go to question 5>
No <go to question 6>
Don’t Know <go to question 6>
Refusal <go to question 6>
How are you familiar with it?
I used to live there
My parents live there
My children live there
Another relative lives there
I use this address for billing purposes
I use this as a shipping address
Other, specify_______ [140 characters explanation]
Probe: Do you know who lives at this address? What sort of relationship do you have with the people who live at this address?
What is your address? [WHAT FIELDS DO WE WANT?]
_______________________________________
_______________________________________
_______________________________________
We would like to collect the names and ages of the people that usually live and sleep here. Let’s start with you,
What is your name?
What was your age?
Who else lives here?
Name2, Age2
Name3, Age3
Name4, Age4
Name5, Age5
Name6, Age6
-
Name
Age
How is <NAME> related to you?
Husband/Wife
s/he is my child
S/he is my parent
s/he is my sibling
s/he is my grandparent
s/he is my grandchild
s/he is another relative
S/he is a roomer/boarder
S/her is a housemate/roommate
S/he is an unmarried partner
S/he is an unrelated person
Other
In preparation for the 2020 Census, we are interested in understanding how we might be able to contact people. We do not plan to use this information to contact you, we just want to study the various ways people can be contacted. Do you have a cell phone?
Yes
No
Don’t know
Refusal
Does NAME have a cell phone? (Repeat for each OVER 18 household member)
Yes
No
Don’t know
Refusal
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
Does anyone in this household share a cellphone?
Yes - Who shares a cellphone?
No
Don’t know
Refusal
Probe: What was this question asking? Was it asking about a device, phone number or calling plan?
FOR EACH YES (person that has their own phone or shares a phone):
Do you/does NAME use the cell phone for texting?
Yes
No
Don’t Know
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
Do you/does NAME use the cell phone to access the internet?
Yes
No
Don’t Know
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
What is your/NAME’s cell phone number?
____- ____- _________
If respondent says they do not know, interviewer should probe:
“Do you have this number stored in your cell phone or in a phone book? Would you mind checking there?”
Do you/Does NAME have a work number?
Yes
What is it?
_____ - ____- __________
Do you/Does name have another work number?
Yes
What is it?
_____-_____-________
No
No
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
Does this household have a landline?
Yes <go to question 15>
No < go to question 19>
Don’t Know
Refusal
What is this landline phone number(s)?
____- ____- _________
____- ____- _________
____- ____- _________
Are there any other phone numbers for this household, perhaps a work phone number or an additional line?
Yes <go to question 17>
No <go to question 18>
Don’t Know <go to question 18>
Refusal <go to question 18>
Probe: In your own words, what do you think this question is asking about? Could you give me an example? Do you have any Vontage, Google or Skype phone numbers? If yes, did you list these? Why or why not?
What are they?
____- ____- _________
____- ____- _________
____- ____- _________
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
Are any of these cell or landline numbers unlisted or unpublished?
Yes
Which ones?
No
Don’t Know
Refusal
Probe: If respondent replied “No”, “Don’t Know, or “Refusal” for anyone ask: For REPEAT QUESTION you replied “Don’t Know/No/Refused”. Can you tell me a little bit more why you chose this response? Do you find this question sensitive or do you just not have the information?
And if some are unlisted/unpublished: Why are these numbers unlisted or unpublished?
In looking forward to the 2020 Census respondents may have the option to receive the form via email and fill it out online, as such we also are interested in respondents’ emailing habits. Do you have an email address?
Yes <go to question 20>
No <go to question 22>
Don’t Know No <go to question 22>
Refusal No <go to question 22>
Probe: In your own words, what do you think this question is asking about?
What are your email addresses?
____________________@_______.____ How often? _______________ _H / W
____________________@_______.____ How often? ________________H / W
____________________@_______.____ How often? ________________H / W
____________________@_______.____ How often? ________________H / W
____________________@_______._____ How often? _______________H / W
____________________@_______._____ How often? _______________H / W
About how often you check your email (insert response in lines above).
Let’s look at the addresses that you provided. Which ones of these are work email addresses? Personal or home addresses? (Circle H or W ) Do you sometimes check work email at home? Home email at work?
How often do you check each of these email addresses [Have respondent look at list provided and mark for each address offered]?
Most days
Most weeks
Occasionally
Don’t Know
Refusal
Probe: How did you come up with this answer/list? How was this question different from the last one? Why do you use these emails differently?
Does NAME have an email address?
Yes <go to question 23>
No <go to question 25>
Don’t Know <go to question 25>
Refusal <go to question 25>
What is NAME’s primary email address or addresses?
____________________@_______.______
____________________@_______.______
____________________@_______.______
If respondent cannot think of addresses, ask them if they know the domains or part of the addresses. Are any of these addresses used for work?
Probe: How did you come up with this answer/list?
How often do you think NAME checks address <1-3>?
Most days
Most weeks
Occasionally
Don’t Know
Refusal
Probe: How easy or difficult was this question for you to answer and why? How do you know about this information?
REPEAT 22-24 FOR EACH OVER 18 HOUSEHOLD MEMBER
As mentioned earlier, in the past the Census Bureau has mailed most households in the United States a paper census form and visited people who did not respond which cost a lot of taxpayers’ money. In an effort to reduce cost, we are considering alternative ways of contacting people. One such way would be to contact people directly on their cell phones. How would you, personally, feel about the Census Bureau calling you on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against, or
strongly against it
PROBE: How did you choose your response? In your own words, what was this question asking?
Another way of contacting people would be for the Census Bureau to text you a link to complete your census form. How would you, personally, feel about the Census Bureau texting you a link on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against, or
strongly against the Census Bureau texting you?
PROBE: How did you choose your response? In your own words, what was this question asking?
Another way of contacting people would be for the Census Bureau to email you a link to complete your census form. How would you, personally, feel about the Census Bureau emailing you a link on your cell phone? Would you be
strongly in favor of,
somewhat in favor of,
neither in favor nor against,
somewhat against,
or strongly against the Census Bureau emailing you?
PROBE: How did you choose your response? In your own words, what was this question asking?
In order to count people in the right place, the Census Bureau need detailed geographic information. To assist in our efforts to increase the accuracy of the census and potentially save a lot of taxpayers’ money, we could get some of your geographic information from cellphone providers. Would you be willing to allow the Census Bureau to obtain geographic information from your cellular provider?
Yes
No
Don’t Know
Refusal
PROBE: How did you choose your response? In your own words, what was this question asking? What does “geographic information” mean to you?
Thank you for your participation in this survey. Your feedback is very helpful in our research.
Debriefing–FIRST GO BACK AND ADMINSITER GRAY PROBES
Besides the email addresses that you reported, do you have other accounts, with other domains, like aol, yahoo, or msn, for example? (You don’t have to provide them, I’m just interested as to whether you have other email addresses.)
If new addresses mentioned: Why didn’t you list these addresses? What sorts of activities do you use this/these address(es) for?
Do you think people would be more likely to report that they have an email at a specific domain, like yahoo, than providing their actual email address?
If yes, why? (probe about knowledge vs. privacy)
If children in HH, do the children in the household have cellphones or emails? Did you mention those?
Look at the roster/email list and ask respondents: Can you please describe how you use your emails, you mentioned some that you check daily and some less often. Can you describe why? ]
Verification questions, if appropriate
Pull out list of phone numbers and email addresses and compare it to the Roster list that you made. FAQ: If they ask where this address came from we say:
“For this testing of our questionnaire, the Census Bureau acquired data from a variety of sources.
We are evaluating a number of different possible sources of alternative contact information for households that could possibly save a lot of government money.”
Read: We have these phone numbers and email addresses associated with this address. It appears as though there are some differences between this list and the list you provided.
For phone numbers/emails not mentioned but on our list, ask the respondent: Is this phone number/email familiar to you? How do you know it? Is it a way to contact you? Why did you not mention this phone number/email earlier?
Wrap-Up Questions
Other than what we have already talked about, did you find any of the questions difficult? Are there questions that you think some people would find difficult?
Did you find any of the questions sensitive? Are there questions that you think some people would find sensitive?
Do you have anything else you would like to tell us that you haven’t had a chance to mention?
For analysis purposes, we need to get a little more information from you:
What is your age? ____
Are you of Hispanic origin?
Yes
No
What is your race? You may choose more than one. Are you:
White?
Black or African American?
American Indian or Alaska Native?
Asian?
Native Hawaiian or Other Pacific Islander?
What is the highest level of education you have completed?
Less than high school?
High school diploma or GED?
Some college?
Bachelors’ Degree?
Graduate Degree?
Is your household’s income:
Less than $25,000 a year?
Between $25,000 and $50,000?
Between $50,000 and $100,000?
Over $100,000?
How many people, including yourself and any adults or children live with you? ______
Roster #8 #10/11#13a. #13b. #13c. #20, 23 #20
Name |
Age |
Yes/No |
Text |
Web |
Phone # |
Email address |
Freq |
H/W |
1. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
2. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
3. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
4. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
5. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
6. |
|
|
|
|
|
1. |
|
|
|
|
|
|
|
|
2. |
|
|
|
|
|
|
|
|
3. |
|
|
Appendix C
Test 15 Email Script
Hello,
My name is <insert full name here> and I am a research sociologist from the U.S. Census Bureau. We are conducting research to make it easier for people to respond to the Census. We are contacting you to see if you would be willing to take part in a study to improve the questions that will be used in the 2020 Census Research and Planning. You will be compensated $40 if you are eligible and if you choose to participate in one of our in-person interviews. If you are interesting in participating in our study, please email me with a phone number where we can reach you.
Thank you!
<full
name>
Center for Survey Measurement
US Census Bureau
<email address>
<recruiting phone number>
Test 15 Call Script
If there is an Answering machine: Hello, I’m (your Name) from the U.S. Census Bureau. We are doing research to make it easier for people to respond to the Census. We are calling today to see if you would be willing to take part in a study to improve the questions that will be used in the 2020 Census Research and Planning. You will be compensated $40 if you are eligible and choose to participate in one of our in-person interviews. If you are interesting in participating in our study, please call me back at (Your phone number) and mention “Test 15”. Thank you.
If Someone Answers: Hello, I’m (your Name) from the U.S. Census Bureau. We are doing research to make it easier for people to respond to the Census. In the past, for the Census, we have mailed most households in the United States a paper census form and visited people who did not respond which cost a lot of government money. In an effort to reduce cost, we are researching alternative ways of contacting people. We are calling today to see if you would be willing to take part in a study to improve the questions that will be used in the 2020 Census Research and Planning. You will be compensated $40 if you are eligible and choose to participate in one of our in-person interviews. Do you have time to answer a few screener questions?
Your responses to these questions are protected by law under Title 13 of the United States Code. This is the same law that protects your privacy. No one outside the Census Bureau will see your answers. This interview will take about 2 minutes to complete, and has been approved by the Office of Management and Budget under approval number 0607-0725 and expires August 31, 2013. This valid approval number legally certifies this information collection.
[If at any point they ask how you got their number: “We are evaluating a number of different possible sources of alternative contact information for households that could possibly save a lot of government money. .”
If needed, add: “Due to privacy restrictions within our contractual agreements to acquire the data, we cannot release the names of specific sources”
Great. Before we proceed, what is your age? ____
(If respondent is younger than 18, ask: Is there anyone over 18 years of age who I can speak with? If yes, start again from top. If not, ask: Is there a time I could call back when I could speak to someone over 18? If yes, record time, if not, thank them and hang up.
Can I please have your first and last name?
First Name:________________________
Last Name:________________________
Are you employed by the federal government?
[ ] Yes – Since you are employed by the federal government, you are not eligible to receive payment for this interview. If there is an adult in your household who is not employed by the federal government, they could be interviewed for payment. Are you interested in continuing?
[ ] No
Are you male/female?
[ ] Male
[ ] Female
What is the highest grade of school you have completed, or the highest degree you have received?
[ ] Never completed any school
[ ] Completed 9th grade or below
[ ] Some High School
[ ] Completed High School, or received a GED
[ ] Vocational Training beyond High School
[ ] Some College
[ ] Completed College
[ ] Any Graduated of Professional education
Are you of Hispanic or Spanish origin?
[ ] Yes
[ ] No
Choose one of more of the following races:
[ ] White
[ ] black or African American
[ ] Asian
[ ] American Indian or Alaska Native
[ ] Native Hawaiian or other Pacific Islander
Would you be able to come to our facility in Suitland, MD to be interviewed?
[ ] Yes (go to 9) [ ] No (go to 10) [ ] Possibly (go to 9)
Are you a U.S. Citizen?
[ ] Yes (go to 12)
[ ] No (go to 10)
Is there a metro-accessible coffee shop where it would be convenient for you to meet for an interview, such as a Starbucks, Caribou Coffee, Panera Bread Company or Cosi?
[ ] Yes
Do you know the location/address of this place? What is it? (go to 12)
_________________________________________________________________________________________________________________________________________________________________________________
[ ] No (go to 11)
Would you be able to meet at any of the following locations for an interview?
Caribou Coffee in Dupont Circle NW
Corner Bakery on the main level in Union Station, SE
Panera Bread Co. in Friendship Heights, NW
Panera Bread Co. in Columbia Heights, NW
Cosi 301 Pennsylvania Avenue Southeast
The Blind Dog Café 944 Florida Avenue Northwest
Pound Coffee, 621 Pennsylvania Ave SE
Sova Coffee Shop, 1359 H Street NE
Starbucks, 1100 4th Street SW
What time during the week would be most convenient for you to be interviewed between the hours of 9-5, Monday through Friday?
___________________________________________________________________________________________________________________________________________________________________________________________________
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | scanl009 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |
© 2024 OMB.report | Privacy Policy