OMB Memo

Web Survey Design Standards Online Research Submitted Under Generic Information Collection Request

Request: The Census Bureau plans to conduct research under the generic clearance for pretesting (OMB number 0607-0725). We will be conducting research online to gather time-on-task, data quality and satisfaction metrics on different web survey designs by having participants complete a 10-minute online survey. There will be a maximum of five independent studies between February 2020 and June 2021. We are seeking approval for all five online studies.

Purpose: The results of these studies will inform future mobile and PC web designs used at the U.S. Census Bureau for household and economic surveys and censuses. These online studies will also be supplemented by in-person studies (OMB submitted independently) over the next 1.5 years. The compilation of the results from these studies will be used to create a guidelines document that Census Bureau programmers will use to create future online surveys.

All online studies will be fully-crossed between-subject studies, with a minimum of two conditions per web design topic. This first study plans to focus on three web design topics: within screen branching; use of buttons for response options; and predictive text for open-ended text fields. Subsequent studies might repeat or expand on these topics as well as cover text on the submit screen; edit messages; choosing ways to respond; format for write-in fields, including dollar amount fields; grids; field label placement; typeface; icons; and response formats.

Each online study will take approximately 10-minutes to complete and most of the questions will come directly (or with slight modification) from pre-existing surveys such as the American Community Survey, the 2018 Census, the National Sample Survey of Registered Nurses, the National Household Education Survey, the National Survey of College Graduates, the Annual Business Survey, or from questions (including the satisfaction and preference questions) used in the Mobile UI Design Framework for Survey Operations. In each online study, we will not collect detailed health-related information or financial institution information, like bank account information.

The look-and-feel of each question in the survey will include one or more of the design topics being studied in that round of testing. Participants will be randomly assigned to one of combinations of conditions. Each participant will see and interact with one of the design conditions for each of the topics. Topics for each study will be chosen with the assumption that there is no confounding effect between them. Metrics collected include but are not limited to time-on-screen, answer choices, response distributions, break-offs, and clicks. We will compare those measures for each condition within each topic, controlling for the other design topic conditions used in that survey, demographic data, and device-used data. Conditions that take less time to answer and result in more complete and accurate data will be the recommended design. Accuracy will be measured in a variety of ways, including less satisficing, and more detailed, codable, or edited answers. Distributions of answers will also be compared across design conditions. Satisfaction and preference data will be collected and tallied for each design condition.

Population of Interest: The planned research will focus on assessing and improving the user experience for people over the age of 18 who will respond online surveys and censuses.

Timeline: The first study will be conducted February 24, 2020 through March 8, 2020 or until the requested number of completed questionnaires is reached. The four subsequent studies will also use a 2-week window of data collection or until the quota is reached with all studies concluded by June 2021.

Language: Testing will be conducted in English only.

Method: We will use an online Qualtrics survey to gather data. There will be no paper or telephone option.

Sample: We will request no more than 400 completed responses using a Qualtrics panel per study. Qualtrics will select the participants according to our requirements. We require a general population sample over 18 years old.

If we want to study break-offs, we would also need to sample from the Census Bureau affinity panel or contact frame. This affinity panel, comprised of people who have opted into participating in Census Bureau research studies, does not receive any incentive for participation and is more likely to leave the survey prematurely. The contact frame, a frame of emails from the U.S. population gathered through administrative records, also does not receive any compensation.

Recruitment: Qualtrics will select the sample from their panel and will handle their recruitment. For the affinity or contact frame, internal Census Bureau staff will select the sample. We will not conduct any affinity panel or contact frame testing during the 2020 Census window (March – July 2020).

Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project.

Use of Incentive: The Qualtrics sample will be given an incentive by the Qualtrics vendor. Per Qualtrics, the exact amount and form that each respondent receives can vary depending on the participants’ profile, how they were recruited, and the form of incentives they have elected to receive (i.e. e-gift cards, points, cash, etc.). Generally speaking, though, respondents receive ~$1.00-$1.50 or a relatively equivalent value for completing a 15 to 20-minute consumer survey.  The Census Bureau affinity panel and contact frame does not receive any incentive.

Below is a list of materials to be used in the current study.

1. Questionnaire for the online study 1 (Enclosure 1)

The list of materials for studies 2-5 will be sent via a post-hoc addendum after the study dates. However, the questions will come from either Enclosure 1, or one of the following surveys.

2. The American Community Survey (Enclosure 2)

3. The 2018 Census (Enclosure 3)

4. The National Sample Survey of Registered Nurses (Enclosure 4)

5. The National Household Education Survey (Parent and Family Involvement in Education Survey - PFI – Enclosure 5a) and (Early Childhood Program Participation survey - ECPP- Enclosure 5b)

6. The National Survey of College Graduates (Enclosures 6a and 6b)

7. Selected questions from the Annual Business Survey (Enclosure 7)

8. Questions and Satisfaction and Preference type questions from the Mobile UI Design Framework for Survey Operations – Phase 1, Study 2 on approved on 6/15/2017 (Enclosure 8)

Length of interview: We estimate 10 minutes for each online survey.

The total estimated respondent burden for this request is 536 hours.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Online general population sample	400 completes from a Qualtrics panel for each of the 5 studies Estimated 400 completes from the Census Bureau affinity panel or contact frame for no more than 3 studies	10 minutes	67 hours for 400 completes 536 hours total assuming 8 samples maximum at 67 hours each

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Elizabeth Nichols

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1724

[email protected]

3