HINTS memo Change Request Cycle 2_5-18-12

Date: May 18, 2012

To: Office of Management and Budget (OMB)

Through: Keith Tucker, Report Clearance Officer, HHS

Seleda Perryman, Program Officer, Project Clearance Branch, NIH

Vivian Horovitch-Kelley, PRA OMB Project Clearance Liaison, OMAA, NCI

From: Bradford Hesse, Ph.D., HINTS Project Officer and Chief, Health Communication and Informatics Research Branch

National Cancer Institute (NCI)/NIH

Subject: Non-Substantive Change Request and Cycle 2 Instrument for, “Health Information National Trends Survey 4 (HINTS 4)”

(OMB NO. 0925-0538, Expiry Date 10/31/2014)

In accordance with the teleconferences between OMB and the HINTS program staff on November 29, 2010 and May 30, 2012, this memo is a non-substantive change request which summarizes the Health Information National Trends Survey (HINTS) decisions about the survey design and implementation for Cycle 2 data collection.

To date, four sub-studies have been approved conducted under OMB No. 0925-0589 for HINTS to finalize materials and test procedures. They are detailed in Appendix A.

This memo reviews the results of experiments conducted as part of the Cycle 1 data collection (OMB No. 0925-0538, exp. 10/31/2014) and the resulting decisions that have been made for Cycle 2. Specifically, the memo covers:

• the method for selecting a household respondent(s),

• changes to the mailing strategy to improve response from the Spanish speaking population; and

• final questionnaire content decisions.

As outlined in the Supporting Statement of the OMB package submitted for HINTS 4, the target population is all adults age 18 or older in the civilian non-institutionalized population of the United States. HINTS 4 uses an address-based sampling frame, selecting the sample from all residential addresses in the U.S., and uses mail data collection procedures and paper questionnaires.

1. Respondent Selection Methods

Since the HINTS 4 questionnaires are mailed to an address rather than a particular individual, data collection procedures need to include instructions for selecting a respondent within the household living at the sampled address. Cycle 1 tested two different methods for selecting a household respondent:

• All Adult (AA). This method requests that all adults in the household fill out a questionnaire. Two questionnaires are sent to the household and a toll free number is provided to request additional questionnaires. The AA method reflects a probability based method for selecting a household respondent, as all adults in the household have the same chance of selection. Likewise, the instructions to the household are quite straightforward. However, the address frame does not include information about the number of adults in a household, so the data collection operation must include strategies for getting the right number of questionnaires to a household to allow all adults to respond. In addition, all initial mailing packages need to include at least two questionnaires, given that almost 70% of all U.S. households include two or more adults.

• Next Birthday (NB). This method asks that the adult with the next birthday complete the questionnaire. The NB method is a quasi-probability method. Additionally, the NB can result in greater weight variability since only 1 respondent is selected to participate no matter the number of adults living in the household. However, there are some operational advantages to the NB method. For instance, the number of questionnaires sent to each household is fixed at 1. Requiring only 1 questionnaire per household facilitates methods to incorporate Spanish language materials.

For Cycle 1, we mailed 10,280 households a questionnaire using the NB method and another 2,105 households were sent a mailing package with 2 questionnaires using the AA method.

1.1 Results from Cycle 1

To evaluate the AA and NB methods, our analysis looked at several metrics:

• Response rates by method

• Demographic composition of respondents in each method

• Key survey estimates by method

• Estimates of data collection costs by method.

Each of these is discussed below.

Response Rates

Table 1 shows the two components of the overall response rate, as well as the overall response rate by respondent selection method. Response at a household level, meaning whether or not at least one completed questionnaire for the address was returned, resulted in a slightly higher response rate with the NB method relative to the AA ( t=2.03, p<0.05). When accounting for the within household response rate, meaning whether or not we received the expected number of returns from a household, the difference between the two methods in the overall response rate becomes more dramatic. The NB resulted in about a 9 percentage point higher overall response rate than the AA method.

Table 1: Response rates by selection method

Response type	Next Birthday Method	All Adult Method
Household	38.72%	35.26%
Within-household	NA	84.57%
Overall response rate	38.72%	29.82

Demographic Composition

The analysis indicates differences in several demographic characteristics related to socio-economic status – education, income, homeownership. In addition, the results identify a meaningful difference in terms of age of the respondents. The analysis used weights reflecting the probability of selection only, without any nonresponse adjustments, in order to capture differences in the responding populations.

• Age: The AA method results in significantly more senior participants (65+) than NB (30.8% versus, 24.2%), though in general the age distributions for both methods significantly differ from the American Community Survey (ACS).

• Education: The AA education distribution comes closer to the ACS distribution, but both under-represent less educated respondents and over-represent the more educated relative to the ACS. However, the NB method results in more, higher educated respondents than the AA method does with almost 46% of the respondents with a college degree or higher. This compares to 39% for AA.

• Marital status: The NB method has significantly more respondents who are currently married and fewer never married respondents as compared to the AA method.

• Number of household members: More respondents report living in one person households in the AA method than in the NB method. The NB method resulted in a significant over-representation of three or more adult households. Both methods underrepresent one person households, likely a correlate of the underrepresentation of young adults.

• Income: Overall household income also differed by respondent selection method. Both NB and AA result in distributions skewed towards the higher incomes, but the NB method attains responses from significantly more people with household incomes of $75,000 or more. In NB, just about 37% of respondents reported a household income of $75,000 or more, but about 10 percentage points fewer (27%) reported incomes of $75,000 or more in the AA method.

• Home ownership: Significantly more respondents in the NB method report owning their homes (75.8%) than in the AA group (72.0%).

In general, these demographic comparisons suggest that the NB methodology results in getting cooperation from respondents from a generally higher SES group. NB respondents tended to have higher education (college or more), higher incomes, more likely to be married and homeowners relative to the AA respondents. For income and to a lesser extent education, the AA distribution comes closer to the ACS distribution. We did not observe any differences in terms of race or gender, though both AA and NB result in an underestimate of minorities and males relative to the ACS.

Key Survey Estimates

We compared estimates by respondent selection method for 24 selected HINTS variables that reflected information seeking behaviors, communication topics, internet use, health care and health behaviors and one cancer-specific variable. Across these 24 variables in which we compared the estimate from the AA respondent selection group to the NB group, we found significant differences in two related variables: Source of information and Internet Use. Other than for these two variables, we did not find any other meaningful differences between the two selection methods.

The distributions for the question about the source used first in the most recent search for health or medical information are largely similar between the AA and NB methods, with the exception of the Internet category. Close to 10% percentage points more respondents in the NB method selected the internet as the first choice when looking for health or medical information. This difference makes sense given the differences observed in the responding populations by method because of the correlation of income with access to the internet.

Similarly, when specifically asked about ever using the internet for any purpose, close to 81% of NB respondents responded positively as compared to 74% of AA respondents (Chi square=13.48, p=0.001).

As a last step in this portion of the analysis, we reran the comparisons but used the final weights incorporating all nonresponse adjustments. With these weights, the differences for these two items between the two selection groups diminish greatly, losing statistical significance.

This analysis suggests that while the NB method does seem to result in response from higher socio-economic groups relative to the AA method, those demographic differences do not seem to systematically affect HINTS survey estimates (assuming the variables analyzed here reflect well on other HINTS variables). The two estimates having to do with Internet use that did seem to reflect the demographic difference disappear once weighting to account for nonresponse.

Costs

To estimate the cost per complete, we included the following direct costs:

• Printing

• Postage

• Receipt and scanning labor

• Incentives

The cost per complete were not hugely different between the two methods. On average, the cost per complete for the AA method was about $52. For the NB, the cost was about $48. With a slightly higher response rate from the AA method, this cost differential would likely reduce to something negligible.

1.2 Decision for Cycle 2

The results from the respondent selection experiment did not definitively point to either of the two procedures. Given the small empirical differences between the two selection methods, we have decided to use the NB method for Cycle 2. While the AA has somewhat better demographic distributions, the differences between the two methods are not great. The biggest advantage of NB for Cycle 2 is that it can best accommodate the planned mailings of the Spanish instrument (described below).

2.0 Mailing Strategy for Spanish Instruments

2.1 Results from Cycle 1

For Cycle 1, households in linguistically isolated areas (LI areas) and those that had a Spanish surname were not included in the respondent selection experiment. These households used the NB method of respondent selection by first sending an English Language (EL) questionnaire. If no questionnaire was returned, two questionnaires were mailed to the household at the follow-up mailing. One was the EL and one was a Spanish-Language (SL) version. A label was attached to the outside of the outgoing envelope with a message in Spanish indicating that Spanish-language materials were included in the envelope.

This procedure was somewhat successful. A total of 79 SL questionnaires were returned using this procedure. This is about 2% of all completes for Cycle 1. It is difficult to know precisely how many questionnaires should be returned in Spanish, since there are not good estimates of individuals who prefer Spanish for responding to a survey such as HINTS. Nonetheless, this is relatively low when compared to the HINTS 3 telephone survey, which completed around twice as many interviews in Spanish.

One would expect that with a low return of the SL questionnaires, the percentage of Hispanics who return a questionnaire will be lower than desired. This actually was the case for Cycle 1, which had 8.5% of the completed interviews being from Hispanic respondents. This compares to 13% in the population as indicated by the ACS.

The lower return of Hispanics is due partly to an under-representation of SL respondents. The procedure used on Cycle 1 did not send a SL questionnaire until the second mailing, which may have restricted the number of people that were potentially exposed to that questionnaire. Furthermore, the SL questionnaire was restricted to only those in LI and Spanish Surname matched portions of the sample. This is only about 10% of the population.

2.3 Results from NHES

We also examined the results from a recent experiment conducted as part of the National Household Education Survey (NHES OMB No. 1850-0768). This experiment tested different variations of sending SL questionnaires to households including: 1) sending two questionnaires (SL and EL) vs. a dual-language version, 2) the timing of when the SL questionnaire was sent (e.g., first mailout; follow-up mailouts) and 3) the type of sample (e.g., surname matched, general population). The results indicate that sending two questionnaires at the first mailing significantly increases the number of SL questionnaires completed. The results also indicated that sending two questionnaires to the entire sample, without targeting the mailing to particular addresses, yielded the largest number of SL questionnaires. This is a surprising result, given common practice which avoids sending SL material to a general population sample. The fear has been that non-SL households will be annoyed when receiving the non-English materials. However, the NHES experiments did not find a significant drop in the response rate for this treatment. The final results, based on relatively small samples, found that approximately 9% of returns in the national sample completed an SL questionnaire. This was approximately half of those who reported being of Hispanic Origin. Furthermore, the response rate was approximately equal to a control group which only sent a single EL questionnaire.

2.2 Decision for Cycle 2

Although there are some significant differences between NHES and HINTS, we have decided to conduct an embedded experiment during Cycle 2 based on the NHES experience. The experiment would have two levels. One is mailing both a SL and EL questionnaire to Spanish surname and Linguistically Isolated households. The second is to mail the two questionnaires to all households. The design and suggested sample sizes are shown in Figure 1. We are recommending that the bulk of the sample be in the “Surname” cell. This minimizes the risks that the new mailing procedure will significantly impact the overall response rate.

Target for Mailing of Spanish Questionnaires
SL mailed only to surname and Linguistically Isolated	10,000
SL mailed to all addresses in Sample	2,000
Total	12,000

Most of the sample (approximately 10,000) would receive two surveys at each mailing to those addresses that have a Spanish surname and/or are in Linguistically Isolated area. If the results mirror those found on the NHES experiment, the number of SL questionnaires would go up to approximately 30% for this treatment. This will theoretically raise the number of completed SL questionnaires from 79 to approximately 105. The second treatment, with approximately 2000 allocated, will send an SL and EL questionnaire to all households at all mailings. Based on NHES results, we would expect this to result in approximately 50 SL questionnaires, or approximately 9% of all returns.

The primary outcomes from the experiment to be analyzed will be:

1. The number of SL questionnaires returned

2. The national level response rate

3. The percent of completes that are Hispanic

4. The socio-demographic composition of the Hispanic population returning questionnaires

5. Differences in selected health outcomes for Hispanics returning questionnaires

The results will be reported to OMB prior to the Cycle 3 testing.

3. Instrument Content Decisions

Like Cycle 1, content for Cycle 2 data collection comes from the all-inclusive items list submitted as part of the original HINTS 4 OMB submission (Appendix B). As planned, approximately half of the Cycle 2 instrument is unchanged from Cycle 1. As noted above, the new items on the Cycle 2 instrument were cognitively tested through OMB No. 0925-0589-03, Expiry Date 4/30/14. Changes as a result of cognitive testing included dropping some proposed questions or sub-questions for inclusion in the Cycle 2 instrument and altering the wording of some questions to improve data quality. The content of the full-length Cycle 2 instrument is shown in Appendix C.

For all data collection instruments, HINTS is using the TRAPD method¹ of translation. This method involves two main steps in order to get the best translation possible:

1. Documents are independently translated by two different translators that are not in contact with each other. In the case of HINTS, there are two separate subcontractors, each doing their own independent translations of the documents. The translators both have been provided with a way to track places where they are unsure of an appropriate translation or where they feel translations are awkward.

2. Both translations and tracking sheets are provided to a third translator who acts as an adjudicator between the two translations. This translator determines the best Spanish version of the document based on the two translations received. Where there are questions or inconsistencies, the adjudicator is in contact with the two translators to discuss the issues and come to a resolution.

Only once all issues have been resolved are the letters and data collection instruments considered final.

This process is time consuming. Two to three weeks are provided to the subcontractors for the initial translations of documents depending on the complexity of the documents (with instruments taking longer than letters). Then, an additional two to three weeks are provided for the adjudication process. It therefore takes approximately four to six weeks from the time that the translators receive the initial English documents to the time that a Spanish translation is finalized.

Because of time limitations, the version of the Spanish instrument attached here is just an initial translation. It has not yet been through the adjudication process. We anticipate there will be some changes to the translation through adjudication, but these changes should be minor in nature. The Spanish language version is shown in Appendix D.

LIST OF APPENDICES:

1. HINTS 4 sub-study approvals

2. List Of Potential Items

3. Instrument – English

4. Instrument – Spanish

1 Harkness, Janet A., Ana Villar and Brad Edwards (2010) “Translation, Adaptation, and Design” in Survey Methods in Multinational, Multiregional, and Multicultural Contexts J. Harkness, M. Braun, B. Edwards, T. Johnson, L. Lyberg, P. Mohler, B. Pennell, and T. Smith (eds), Chapter 7, pgs 117-140.

8