Memo

Generic Information Collection Request:
Cognitive and Usability Testing of a Self-Administered Current Population Survey (CPS)

Request: The Census Bureau and Bureau of Labor Statistics, through a contract with RTI, plans to conduct cognitive and usability testing under the generic clearance for questionnaire pretesting research (OMB number 0607-0725) for a self-administered Current Population Survey (CPS).

Purpose: Efforts are underway to add an internet self-response option for the CPS, which is currently conducted only in-person and by telephone. A self-administered version of the CPS presents many challenges, particularly with regards to respondent burden and data quality. Several core CPS items require interviewer probing to elicit sufficient detail from respondents, such as descriptions of job titles and job duties. There are also questions that are interviewer-coded based on open-ended responses (e.g., a question on which techniques respondents looking for work used to try to find a job). The high visibility of the data and the complexity of these challenges mean that thorough testing of a self-administered CPS must be completed prior to implementing this mode into production. Therefore, in partnership with the Bureau of Labor Statistics, the Census Bureau intends to conduct cognitive and usability testing of an internet response mode CPS.

Population of Interest: CPS samples a nationally representative sample of households in the United States. We plan to recruit from this general population as described further below.

Timeline: Testing will be conducted from November 2023 through December 2024.

Language: Testing will be conducted in English.

Sample: We plan to conduct two rounds of cognitive testing and two rounds of usability testing. In total, there will be 200 respondents: 70 in cognitive interviews and 130 in usability sessions.

Recruitment: Respondents will be recruited through a variety of methods. RTI will use methods such as posting fliers at local community organizations, putting advertisements in community newspapers, and collaborating with staff at community-based organizations to identify potential respondents. RTI will also post advertisements on Craigslist.com and social media sites such as Facebook and Reddit. Finally, broadcast messages will be distributed to Census Bureau and RTI staff to leverage personal connections.

Recruitment will primarily focus on identifying respondents who meet certain criteria related to critical paths through the instrument. Specifically, we will recruit respondents who are:

• Employed full-time (35+ hours per week)

• Employed part-time (less than 35 hours per week)

• Working more than one job and whose hours generally vary (including gig economy workers)

• Self-employed

• Unemployed and actively looking for work

• Not in the labor force (retired or disabled)

• Not working and not looking for work, or only passively looking for work

If possible, we will also recruit respondents who are discouraged workers marginally attached to the labor force (looked for work in last year but not in last 4 weeks) and people who had a deviation in their usual work schedule during the CPS reference week (e.g., on vacation, in school, temporarily ill, on jury duty). However, we anticipate that the small size of these populations will make recruitment difficult and therefore will concentrate on the bulleted characteristics above.

Most (80%) of the respondents will live in a household with at least one other person since proxy response is a design feature of the CPS. We will also aim for demographic diversity in sex, education, race/ethnicity, geographic region of the country, and household composition. Since this research is focused on a self-administered web version of the CPS, interviews will be conducted with people who are likely to respond online (i.e., those who use the internet frequently). For usability testing, we will recruit respondents to complete the CPS on both laptop/desktop and mobile devices. The screening questions and advertising materials are attached (see Attachments 1 and 2).

Method: Staff from RTI will conduct cognitive interviews and usability sessions remotely using Microsoft Teams.

Protocol: During the cognitive interviews, respondents will first be asked to complete a self-administered version of the CPS as they would if they received it at home (see Attachment 3). Researchers will observe respondents’ interactions with the survey. Respondents will then be probed about their answers, including asking how they came up with their answers and what certain terms and phrases mean to them (see Attachment 4). The interviews will conclude by asking respondents general debriefing probes about their experience. Feedback may inform iterative changes to the protocol.

In the usability sessions, respondents will be asked to think out loud while they complete a self-administered version of the CPS as they would if they received it at home (see Attachment 5). The interviewer will do minimal probing during the survey-taking, saying things such as “keep talking” and “what are you thinking?” After they finish, respondents will be debriefed to evaluate responses the participant gave and get additional feedback on any areas they showed hesitation about or paused on as they were completing the survey. At the end, the respondent will complete a satisfaction questionnaire (see Attachment 6) and be asked general debriefing probes about their experience.

Consent: We will inform respondents that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio and/or videotaped to facilitate analysis of the results (see Attachment 7). Verbal consent will be captured on recordings. Respondents who do not consent to be video and/or audio-taped will still be allowed to participate.

Incentive: Respondents will receive $40.

Length of Interview: We estimate that each of the 200 interviews will take approximately one hour. This results in a burden of 200 hours.

The screening questionnaire specific to this research will take approximately 15 minutes per person. We estimate that we will screen 6 people for each successful recruit for each of the 200 interviews. Therefore, we estimate a total of 1200 people screened for a total of 300 hours (1200 people at 15 minutes each).

Thus, the total estimated burden for the first round of this research is X hours.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	1200	15 minutes	300 hours
Cognitive Interviews	200	60 minutes	200 hours
Totals			500 hours

Below is a list of materials to be used in the current study:

Attachment 1. Screening questionnaire

Attachment 2. Recruitment methodology and ads

Attachment 3. CPS Self-administered questionnaire

Attachment 4. Cognitive interview protocol

Attachment 5. Usability testing protocol

Attachment 6. Usability testing satisfaction questions

Attachment 7. Consent form

Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Jessica Holzberg

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

301-763-2298

[email protected]

3