CPI Letter

June 8, 2017

NOTE TO THE REVIEWER OF:	OMB CLEARANCE #1220-0141 “Cognitive and Psychological Research”
FROM:	Bill Mockovak Survey Methodologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Evaluation of CPI Respondent Letter Study

Please accept the enclosed materials for approval under the OMB clearance package #1220-0141, “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research.

The maximum number of burden hours is estimated to be 50.

If there are any questions regarding this project, please contact Bill Mockovak at (202) 691-7414.

Attachments

I. Introduction and Purpose

The Office of Survey Methods Research has been working with the BLS Boston Regional Office to improve two letters sent to respondents in the CPI Housing Survey. Respondents in this survey are interviewed every 6 months for up to six years, so maintaining ongoing cooperation is important. One of these letters is sent to “hard to contact” respondents, and the other letter is sent to current respondents to maintain rapport and to encourage ongoing cooperation (“maintain cooperation” letter). The purpose of this study is to conduct an evaluation of the letter sent to maintain cooperation. Research objectives are to determine how the general population responds to the letter, and if key features of the letter can be improved (for example, by citing different uses of the CPI).

Although the main purpose of the study will be to evaluate the letter, since attitude questions are being used extensively in the survey evaluation instrument, other research questions will also be pursued that can be applied to BLS customer satisfaction surveys or similar evaluation studies. Specifically, the attempt will be made to answer the question “Do responses to attitude questions vary when they appear in a matrix (table) versus appearing on separate Web pages? And, in addition, does it matter if the response scales are presented horizontally or vertically?

Based on previous research, we can tentatively conclude the following:

• The results are inconsistent regarding which format is better: displaying questions in a matrix vs individually. How individual questions are displayed can also vary, for example, questions can be displayed separately on a single page vs each displayed separately on a different page, but it’s not clear if this makes much of a difference, except possibly on how long it takes to complete the questions. (Couper and Lamias, 2001; Tourangeau et al., 2004, Couper et al., 2013).

• In some studies, matrix questions have been completed more quickly and with higher inter-item corrections, but at least one researcher (Petchev, 2009) has argued that the higher inter-item correlations could be a sign of lower quality responding.

• The orientation; that is, the vertical vs horizontal presentation of the rating scale has also shown inconsistent effects (Toepoel and van Soest, 2009). However, it does matter whether you list response options from positive to negative or from negative to positive, regardless of the orientation (vertical or horizontal), and when positive items are listed on the left, bias is greater (Chan, 1991; Toepoel and van Soest, 2009). In addition, the horizontal format with fully labeled response options seems less susceptible to bias.

A goal of the proposed research is to continue this line of research on the impact of question formatting on response quality.

II. Respondents

Respondents will be recruited from Amazon’s Mechanical Turk. Only U.S. citizens, age 21 or older, will be allowed to participate.

III. Research Design

The analysis will focus on responses to four attitude questions that mention the following uses of the CPI:

• Adjusting Social Security payments

• Adjusting retirement payments

• Adjusting the cost of school lunch programs

• Adjusting food stamp benefits

Table 1 shows how the three instruments vary. In Instrument 1, the four questions of interest are each displayed on a separate page with a horizontal response scale (and responses labeled from most positive to most negative).

In Instrument 2, the four questions are displayed in a matrix, with the column headings containing the response options (therefore, shown as a horizontal scale, but part of a matrix).

In Instrument 3, the four questions are each displayed on a separate page with a vertical response scale (and responses from top to bottom labeled from most positive to most negative).

Table 1. Differences in the Online Data Collection Instruments

	Format for Displaying Four CPI-Use Questions		Format for Displaying Response Scales
Instrument	Matrix Q	Separate Q on separate page	Vertical scale	Horizontal Scale
1		X		X
2	X			X
3		X	X

In addition, in all three versions of the instruments, the order of the four questions mentioning the key uses of the CPI is randomized. And, in Instrument 2, the format of the response scale for Questions 1 and 3 was changed to horizontal to generate more data for the horizontal vs. vertical scale comparison.

IV. Burden Hours

Recruitment. Since participants will be recruited through Amazon’s Mechanical Turk, there are no burden hours for recruitment.

Each instrument is estimated to take an average of 5 minutes to complete. The number of participants that will complete each instrument is shown in Table 2.

Table 2. Number of Participants

Instrument	Number of Participants
1	200
2	200
3	200

The number of burden hours is estimated to be 50 (600 people x 5 minutes divided by 60 min = 50 hours).

V. Payment to Respondents

Participants will be paid $2.10 for completing the survey.

V. Data Confidentiality

BLS will not provide a guarantee of confidentiality.

VI. Attachments

PDFs of the data collection instruments

References

Chan, J. 1991. Response Order Effects in Likert-type Scales. Educational and Psychological Measurement, 51, 531-540.

Couper MP, Traugott M, Lamias M. Web Survey Design and Administration. Public Opinion Quarterly. 2001; 65(2):230–253. 

Couper, M.; Tourangeau, R.; Conrad, F.; and Zhang, C. 2013. The Design of Grids in Web Surveys. Social Science Computer Review. Jun; 31(3): 322–345. doi:  10.1177/0894439312469865

Peytchev A. 2009. Survey Breakoff. Public Opinion Quarterly. 73(1):74–97

Toepoel V, Das M, van Soest A. 2009. Design of Web Questionnaires: The Effect of Layout in Rating Scales. Journal of Official Statistics. 25(4):509–528.

Tourangeau R, Couper MP, Conrad FG. Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions. Public Opinion Quarterly. 2004; 68(3):368–393.

5