Justification Memo for Change Request for Cycle 4

Date: June 3, 2014

To: Office of Management and Budget (OMB)

Through: Darius Taylor, Report Clearance Officer, HHS

Seleda Perryman, Program Clearance Officer, NIH

Karla Bailey, PRA OMB Clearance Liaison, NCI

From: Bradford Hesse, Ph.D., HINTS Project Officer and Chief,

Health Communication and Informatics Research Branch

National Cancer Institute (NCI)/NIH

Subject: Non-Substantive Change Request and Cycle 4 Instrument for,

“Health Information National Trends Survey 4 (HINTS 4)”

(OMB NO. 0925-0538, Expiry Date 10/31/2014)

In accordance with the teleconferences between OMB and the HINTS program staff on November 29, 2010 and May 30, 2012, this memo is a non-substantive change request which summarizes the Health Information National Trends Survey (HINTS) decisions about the survey design and implementation for Cycle 4 data collection.

To date, six sub-studies have been approved conducted under OMB No. 0925-0589 for HINTS to finalize materials and test procedures. They are detailed in Appendix A.

Following cognitive testing, the Cycle 4 instrument has been finalized. Please see Appendix B for the final instrument in English and Appendix C for the final instrument in Spanish.

This memo discusses aspects of Cycle 3 data collection (OMB No. 0925-0538, exp. 10/31/2014) and the resulting decisions that have been made for Cycle 4. Specifically, the memo covers:

• Variations in cover design; and

• Shading of grid questions.

As outlined in the Supporting Statement of the OMB package submitted for HINTS 4, the target population is all adults age 18 or older in the civilian non-institutionalized population of the United States. HINTS 4 uses an address-based sampling frame, selecting the sample from all residential addresses in the U.S., and uses mail data collection procedures and paper questionnaires.

Variations in Cover Design

During HINTS Cycle 3, the Spanish language questionnaire covers were updated to better distinguish them from the English questionnaires. The Spanish questionnaires had a green background on the cover while the English questionnaires maintained the blue background. This change appeared to positively affect Spanish language return rates. In comparing Cycle 2 and Cycle 3 households that were exposed to similar treatments, Cycle 3 saw a higher rate of return of Spanish questionnaires (35.2% of 432 potential Spanish language respondents) as compared to Cycle 2 (25.8% of 422). Although not directly comparable given differential experimental manipulations, Cycle 3 also saw a higher rate of return overall for Spanish language questionnaires as compared to Cycle 2 (5.3% and 3.3%, respectively).

It is likely that the increased rate of Spanish language returns in Cycle 3 was attributable to the change in background color of the questionnaire. This design change made the Spanish questionnaires more distinguishable from the English questionnaires and, in turn, made them more noticeable within the mailed package.

Given the apparent positive effect of changing the color of the Spanish questionnaire on Spanish language return rates, we propose exploring effect of cover design aspects on overall response rates in Cycle 4 of HINTS. The goal of the experiment will be to understand whether and how cover design characteristics impact response rates.

Two cover design characteristics will be manipulated:

1. Overall page contrast – What is the effect of having a predominately white background with dark text versus a predominately blue or green cover background with white text?

There is evidence that a distinctive background may lead to respondents noticing the questionnaire once it is out of the package (e.g., sitting on a desk or table for a few days). Dillman (2007) cites evidence from Nederhof (1988) who compared response rates between two questionnaires, one with a predominately white cover and the other with a predominately black cover, in a sample of Dutch biotechnologists. Nederhof found that the black cover achieved a 10% higher response rate than the white cover. The observed gains in response were achieved after a reminder postcard was sent to respondents who had yet to return the questionnaire. In an attempt to replicate the findings from Nederhof (1988), Dillman & Dillman (1995) found a less dramatic improvement when using colored questionnaire covers relative to white covers (approximately 2 – 4 % improvement) in a general  population survey.

2. Image placement – What is the effect of having images grouped together in a collage versus spread about the cover page?

While the current HINTS cover is simple and attractive (as evidenced by our cognitive interviews and mail survey response rates), we wonder whether the size and placement of the pictures spreads the respondent’s attention more than needed. With fewer larger pictures, the respondent’s attention might be more focused on reading important elements, like the government sponsorship, which adds credibility to the survey.

We propose keeping the current HINTS cover as the control for the large part of the sample and then having four different conditions in the experimental groups. See Appendix D for the versions. We believe changes to the covers represent a low-risk, low-effort attempt at increasing the response. We do not believe any of these changes will significantly decrease the response rate. The experimental conditions do not account for a large percentage of the sample (about 25% of the sample); if the manipulation decreases the response rate, it should not have a big effect on the overall response rate.  (Sample sizes are discussed below).

Shading of Grid Questions

A persistent issue in HINTS questionnaires is higher rates of missing data on grid questions compared to individual questions. Table 1 below provides evidence from Cycle 1 that differences in missing data rates between questions within grids and individual questions are driven largely by respondents with lower levels of education and weaker English language skills.¹

Table 1. Cycle 1 - Percent of missing data for question A10 by form, language ability, and education.
	Proficient English				Less than proficient English			Spanish
Item	More than H.S. (n=2551)	H.S. or less (n=1033)			More than H.S. (n=121)	H.S. or less (n=91)		More than H.S. (n=23)	H.S. or less (n=51)
	Percent
Online newspaper	2.7		10.1		6.6	13.2		4.4	19.6
Print newspaper	1.7		7.4		3.3	13.2		4.4	25.5
Health news	1.5		5.7		1.7	9.9		4.4	9.8
Internet	2.0		10.8		5.8	12.1		8.7	25.5
Radio	2.3		9.9		6.6	14.3		4.4	23.5
Local TV	1.5		6.9		4.1	13.2		4.4	15.7
National TV	1.4		7.4		3.3	11.0		4.4	21.6

It is unclear precisely why there is a difference in response between questions within grids and individual questions for these particular types of respondents. One possibility is that the grids are too compact within the two-column format of the questionnaire. To test this, we experimented with changing to a single column format during Cycle 1. This did not reduce the amount of missing data for these items. A second possibility is that the grids are difficult for respondents to navigate, which increases the likelihood that a respondent will stop answering questions within the grid.

One design feature that might improve navigation in the grid is to add alternating shading to the rows in the table. The shading could help respondents navigate the grid more easily by allowing them to visualize which questions have been answered and where the next response is required. Tourangeau, Conrad, and Couper (2013) present evidence that interactively shading or “graying out” grid questions when they are completed helps to reduce item nonresponse in Web surveys. They recommend providing this type of feedback when possible or shading alternating rows of a grid to help respondents navigate.

We would like to experiment with shading the grids on the HINTS Cycle 4 instrument. See Appendix E for a sample page of the HINTS questionnaire with shading in alternating rows in the grid questions to illustrate how this would look in the actual HINTS questionnaire.

Allocation of Sample to the Experiments

We propose the following design to incorporate both the cover design and grid shading conditions. While we do not expect any interactions between the shading in the grids and either of the cover design experiments, a fully crossed design will help us detect any interactions that occur in the data and provide for more conclusive results.

	Low Contrast	High contrast	Total
Current Design	10,000	na	10,000
Revised Non-Collage	1,000	1,000	2,000
No Shading	500	500	1,000
Shading+	500	500	1,000
Collage	1,000	1,000	2,000
No Shading	500	500	1,000
Shading+	500	500	1,000
	12,000	2,000	14,000

+ All Spanish questionnaires included in the revised non-collage and collage groups will include shaded grids.

The majority of the sample will be allocated to the Cycle 3 design (n=10,000), which has low contrast and a non-collage cover. It also does not have shaded rows in the grids. Allocating most of the sample to the current design will ensure that the experiment can only have a small impact on the overall response rates.

The cover design and grid experiment are proposed to be fully crossed with each other for the remainder of the sample. The starting sample size is designed to ensure that we achieve the targeted number of completes for Cycle 4 (3,500). This allocation will allow detection of differences of between 3 – 5 percentage points in the response rate across the two different cover design features with 80% power, depending on whether there are interactions between Picture Design and Contrast Level.

For the grid experiment, we are proposing to simplify the design by using shaded grids for all of the Spanish questionnaires which are in the experimental groups. This will increase the number of returns we receive from Spanish speakers. We can compare this group to the Spanish returns from the main sample, which does not have shaded grids. We would like to maintain the fully crossed design for the English questionnaires so that we can test for any interactions between cover design and grid design.

For analysis of the grid designs, we will be testing whether there are differences in missing data rates between the shaded and non-shaded grids. Using the return rates from prior cycles, we estimate that we will have 80% power to detect a difference of around 5 percentage points for those with High school or less education.

REFERENCES:

Dillman, D. Mail and Internet Surveys: The Tailored Design Method. 2nd ed. John Wiley & Sons, Inc. New Jersey, 2007. Pp. 135-139.

Dillman, J., Dillman, D. (1995). The influence of questionnaire cover design on response to mail surveys. Paper presented at the International Conference on Measurement and Process Quality, Bristol, England.

Nederhof, A. (1988). Effects of a final telephone reminder and questionnaire cover design in mail surveys. Social Science Research, 17. 353-361.

Tourangeau, R., Conrad, F., Couper, M. The Science of Web Surveys. New York, Oxford University Press, 2013.

LIST OF APPENDICES:

Appendix A – HINTS past approvals

Appendix B – Instrument in English

Appendix C – Instrument in Spanish

Appendix D – Cover versions for embedded experiment

Appendix E – Shading example for embedded experiment

1 This result also was evident in Cycle 2.

5