OMB Memo

American Housing Survey (AHS) Cognitive Interviews

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct a cognitive pretesting evaluation of new and revised questions about a variety of topics in the American Housing Survey (AHS), and supplemental pretesting research online using Qualtrics. This evaluation will be conducted by staff in the Center for Behavioral Science Methods (CBSM).

Purpose: The purpose of this cognitive research is to test questions added to or revised in the AHS. The question changes were requested by the Department of Housing and Urban Development to address known problems with the questions, to measure new concepts, and to update rotating modules that were not included in the last survey administration. The following modules of the AHS are being tested:

• Mortgage is an existing core module for homeowners that contains new and revised questions about mortgages and loans.

• Recent Movers is an existing core module for those who moved within the past two years that contains new questions about their most recent housing search.

• Wildfire Risk is a new topical module for single unit homes in wildland urban interface areas that contains questions about home structure, nearby vegetation, and how one would be alerted in case of a wildfire emergency.

• Pets is a new topical module that contains questions about the number and types of pets in the home.

The full text of the questions to be tested for in-person interviews can be found in Enclosure 1, and the full text of the questions for online testing can be found in Enclosure 5.

Population of Interest: The AHS samples households from the general U.S. population. There are two types of respondents to the survey.

1. For Occupied Housing Units – A household respondent, who must be a knowledgeable household member 16 years of age or over, provides information on the unit, the household composition, and income.

2. For Vacant Housing Units – A landlord, owner, real estate agent, or knowledgeable neighbor can provide data on the unit.

For our cognitive pretesting evaluation the respondents will be 18 years of age or over, and answering about the household in which they currently live (i.e. for Occupied Housing Units).

Timeline: Cognitive interviews, whether in person or remote, will be conducted in one round between July 2020 and August 2020. Online responses will also be collected through Qualtrics during this time period. Results from testing will be used to inform recommendations for the final wording of the questions.

Language: Testing will be conducted in English only, for both the in-person testing and online testing.

Method: Staff from CBSM will conduct 15 cognitive interviews, either in person or remotely. These interviews are likely to be conducted remotely by phone for a couple of reasons, and will not include video conferencing software since we do not feel this is necessary. In order to test the Wildfire Risk module with the target population, some interviews will be conducted remotely with respondents from the San Francisco, CA, San Antonio, TX, and Birmingham, AL metropolitan areas. For the other modules, interviews will be conducted with respondents from these areas or the Washington, D.C. metropolitan area. Because of the ongoing coronavirus pandemic, the Census Bureau will evaluate Federal, state, and local guidance to determine whether it is safe and appropriate to conduct cognitive interviews in person or remotely. Some interviews may be conducted in person (face-to-face) in the Washington, D.C. metropolitan area, either at the Census Bureau Headquarters or at another location that is more convenient for a respondent, such as a public library, if appropriate. Otherwise, cognitive interviews will be done remotely with respondents recruited from the Washington, D.C. metropolitan area, or the other areas noted. Researchers will conduct supplemental pretesting research online with 150 respondents using Qualtrics, which will occur at the same time.

Sample: For cognitive interviews, our recruiting efforts will target the following groups:

• Respondents who own a home with one or more mortgages, loans, or lines of credit

• Respondents who live in a single-unit home in one of three metropolitan areas: San Francisco, CA; San Antonio, TX; or Birmingham, AL

• Respondents who rent a home and moved within the past two years

In addition to these characteristics, we will attempt to recruit respondents from diverse demographic groups including race/ethnicity, age, gender, and household size to the maximum extent possible. The cognitive interviews will include questions from all four modules mentioned in the Purpose section.

For online testing, we will plan to target respondents who own a home with one or mortgages, or those who live in single unit homes in one of the three metropolitan areas intended for the Wildfire Risk module. We will implement screening questions within the Qualtrics instrument to target the groups aforementioned. The online testing will include questions from the Mortgage module and Wildfire Risk module; this will not include questions from the Recent Movers module or Pets module.

Recruitment: For cognitive interviews, respondents will be recruited through CBSM’s recruitment database, an internal Census Bureau broadcast email, Craigslist.com, and via personal networks. Additional recruitment may be attempted by posting flyers in local libraries or other public places depending on a review of the Federal, state, and local guidance because of the ongoing coronavirus pandemic. Respondents will be screened and selected based on the survey-relevant characteristics of interest listed in the preceding section. Additional questions to be added to the universal recruiting screener and ads are attached (see Enclosure 2 and Enclosures 3a-3b). Respondents will be screened by telephone and may receive an email reminder about the cognitive interview appointment after they agree to participate.

For online testing, Qualtrics will be choosing the sample from their panel and handling the recruitment.

Protocol: For cognitive interviews, we will use a “think-aloud” approach in which respondents are instructed to speak their thoughts aloud as they receive, process, and respond to the survey questions. As the AHS is an interviewer-administered survey, CBSM researchers will administer the survey questions orally with a respondent, but using a mocked-up paper questionnaire rather than a CATI or CAPI instrument. After respondents have given their response to each survey question and finished thinking aloud about it, they will be asked probes intended to elicit more information about their interpretations of the survey questions and of specific terms, and the circumstances they were thinking about when answering the survey questions. Respondents will also be asked to describe how easy or difficult the questions are to answer. The cognitive interview protocol is included in Enclosure 1, as mentioned earlier.

For online testing, we will have respondents complete a self-administered questionnaire in Qualtrics that will include only a subset of questions from the Mortgage module and Wildfire Risk module. Some survey questions will include follow-up questions intended to elicit more information about their interpretations of the survey questions and familiarity with specific terms, and the circumstances they were thinking about when answering the survey questions. We will also be split ballot testing a commitment device prompt to examine whether this affects the number and quality of substantive responses to probing questions. If having a prompt influences respondents to write more quality answers to probing questions, this could be beneficial to embed in future web probing studies. The Qualtrics survey questionnaire is included in Enclosure 5, as mentioned earlier.

Consent: For cognitive interviews, we will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio-taped to facilitate analysis of the results. Respondents may sign a hardcopy form in person for in-person interviews; for remote interviews, they will be emailed a secure link to an electronic form in Qualtrics that allows for signature (see Enclosure 4). Participants who do not consent to be audio-taped will still be allowed to participate.

Use of Incentive: For cognitive interviews, each respondent will be offered $40 for participation in a sixty-minute interview. This incentive will be offered to help ensure that we are able to recruit participants from all desired populations and to offset the costs of participation in this research, such as childcare, phone use, travel, or parking. For in-person interviews, the incentive will be given to the respondent in person after the interview. For remote interviews, the incentive will be mailed to the respondent at their preferred address using Priority Mail. Federal employees who participate in this research are not eligible to receive the cash incentive.

Respondents in the Qualtrics testing will receive an incentive from Qualtrics. Per Qualtrics, the exact amount and form that each respondent receives can vary depending on the participant’s profile, how they were recruited, and the form of incentives they have elected to receive (i.e. e-gift cards, points, cash, etc.). Generally speaking, respondents receive ~$1.00-$1.50 or a relatively equivalent value for completing a 15 to 20-minute consumer survey.

The materials to be used in this study are enclosed with this letter and include:

Enclosure 1 – Cognitive Interview Protocol

Enclosure 2 – Recruitment Screener

Enclosures 3a-3b – Recruitment Advertisement Text

Enclosure 4 – Consent Form

Enclosure 5 – Qualtrics Survey Questionnaire

Length of interview: For cognitive interviews, we estimate that each interview will take approximately one hour (15 hours total). The additional screening questions will take approximately five minutes per person. We estimate that we will screen 3 people for each successful recruit (3.75 hours).

The online testing instrument will be administered to 150 respondents and will take approximately 15 minutes to complete. This results in a burden of 37.5 hours.

Thus, the total estimated burden for this research is 56.25 hours.

Table 1. Estimated Burden

Research Activity	No. of Respondents	Participation Time	Burden
Screening	45	5 minutes	3.75 hours
Cognitive Interviews	15	60 minutes	15 hours
Qualtrics	150	15 minutes	37.5 hours
Total			56.25 hours

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Matt Virgile
Center for Behavioral Science Methods
U.S. Census Bureau

Room 5K028F
Washington, D.C. 20233
(301) 763-4745
[email protected]

2