FINAL PROTOCOL CHANGES_9-12-11-td

Date: September 12, 2011

To: Office of Management and Budget (OMB)

Through: Mary Forbes, Report Clearance Officer, HHS

Seleda Perryman, Program Officer, Project Clearance Branch, NIH

Vivian Horovitch-Kelley, PRA OMB Project Clearance Liaison, OMAA, NCI

From: Bradford Hesse, Ph.D., HINTS Project Officer and Chief, Health Communication and Informatics Research Branch

National Cancer Institute (NCI)/NIH

Subject: Final Protocol Changes and Cycle 1 Survey for, “Health Information National Trends Survey 4 (HINTS 4)” (OMB NO. 0925-0538, Expiry Date 3/30/2009)

In accordance with the pre-submission teleconference of November 29, 2010, this memo summarizes the Health Information National Trends Survey (HINTS) decisions about the survey design and implementation for Cycle 1 data collection.

Three sub-studies have been conducted under OMB No. 0925-0589 for HINTS to finalize materials:

1. 0925-0589-06: Focus groups were conducted to identify the most effective messages to use with the HINTS 4 mailing materials to encourage the participation of all sampled households, improving overall response rate and diminishing the chances of systematic nonresponse specific to certain subpopulations. These focus groups resulted in the mailing materials included in the pilot study described below.

2. 0925-0589-09: Cognitive testing was conducted to identify problems in question wording, context or order effects, as well as response difficulties resulting from the design and layout of the new mail form. The instrument changes that resulted from the cognitive testing were included in the pilot instrument in sub-study OMB No. 0925-0589-10 below.

3. 0925-0589-10: A pilot test was conducted with a sample of 1,000 households nation-wide. The goals of the pilot test were to: 1) conduct a methodological experiment comparing respondent selection options; 2) conduct a test of questionnaire variations; 3) determine how long respondents take to complete the questionnaire in order to more accurately estimate burden; and 4) test the operational procedures that will be used on HINTS.

This memo reviews the results of the pilot study that was conducted under OMB No. 0925-0589-10 and the decisions that have been made. Specifically, the memo covers:

• the method for selecting a household respondent(s),

• changes to the mailing strategy to improve overall response and in particular, response from the Spanish speaking population

• changes to the cover letters that accompany each questionnaire package, and

• the final questionnaire content decisions

As outlined in the Supporting Statement of the OMB package submitted for HINTS 4, the target population is all adults age 18 or older in the civilian non-institutionalized population of the United States. HINTS 4 will use an address based sampling frame, selecting the sample from all residential addresses in the U.S., and will use mail data collection procedures and paper questionnaires. The pilot study included a split ballot comparison for a subset of HINTS items. HINTS 4 will include only one version of each question included in the split ballot comparison.

1. Respondent Selection Methods

Since the HINTS 4 questionnaires will be mailed to an address rather than a particular individual, data collection procedures need to include instructions for selecting a respondent within the household living at the sampled address. The Pilot study (OMB No. 0925-0589-10, Expiry Date 5/31/2011) tested three different methods for selecting a household respondent.

1. All Adult (AA). This method requests that all adults in the household fill out a questionnaire. Two questionnaires are sent to the household and a toll free number is provided to request additional questionnaires. This method was used for HINTS 2007, based on the results from Battaglia, et al (2008) which found it to provide better coverage of young persons. In addition, for HINTS 4, multiple person households with 2 or more adults that did not return a questionnaire for each adult are sent a follow-up mailing, targeting the individuals that did not return the questionnaire.

2. Hagan-Collier (HC). In this method, each address is randomly assigned to one of four respondent selection rules: 1) youngest male in household as respondent, 2) youngest female as respondent, 3) oldest male and 4) oldest female. The instructions request that the person in the household that fits the designated category fill out the questionnaire. If the household does not include someone of the designated gender, the instructions indicate to select the youngest/oldest person of the opposite gender. A disproportionate allocation is given to the first three groups to increase response from populations that are typically under-enumerated in general population samples.

3. Next Birthday (NB). This method asks that the adult with the next birthday complete the questionnaire. This is a common method used on telephone surveys because it is relatively easy to implement over the phone (Rizzo et al., 2004). Prior HINTS RDD data collections successfully used this method for respondent selection. (To our knowledge only the Battaglia (2008) study has tried this method for respondent selection in a mail survey.)

1.1 Results

Westat computed response rates for each respondent selection method using the weights that account for the probability of selection. For the HC and the NB groups, the response rates vary significantly by strata, with the high minority strata having significantly lower response rates than the low minority strata. In both cases, the high minority strata had response rates of approximately 24% and the low minority strata of 37%. This yields a final response rate across the two strata of approximately 34%. This pattern across the strata is similar to what was found for both the HINTS 3 and the Pilot for HINTS 3. The pattern is considerably different for the AA, which had a nominally lower household response rate for the low minority strata (25.2% vs. 24.4%). This yields a total household response rate of 24.5%, which is significantly lower than the HC and NB methods.

The household-level rates for the HC and NB are the final rates, since only one person per household was sampled. For the AA method, the within-household response rate must also be taken into account. The total within-household response rate across the two strata was 82.6%, with a higher rate in the low minority strata. Sixteen households were mailed a follow-up request targeting household members that had not returned a questionnaire. The follow-up request attained at least one additional questionnaire per household approximately 50% of the time. The final response rate for the AA method is computed by multiplying the household rate (24.5%) by the within household rate (82.6%). This results in a final response rate of 20.5%. The high minority stratum has a slightly lower rate (19.6%) than the low minority stratum (23.6%).

The HC/NB rates are statistically different from the AA rate (p<.01). The difference is driven by the much lower response rate in the lower minority stratum for the AA method (24.4%) as compared to the response rate in that same stratum for the HC/NB methods (37.4% and 37.0%, respectively).

1.2 Decisions

The plan submitted to OMB for the Cycle 1 data collection anticipated not being able to draw definitive conclusions about the respondent selection method from the Pilot survey. The goal was to select two methods for Cycle 1 and to use the comparative results from Cycle 1 to determine a single method for later cycles. Following this planned logic, we will implement both the NB and AA methods in Cycle 1, allocating most of the sample to the NB method, (approximately 11,000 addresses), with a smaller allocation to the AA method (n=2000 addresses). Given the previous success of the AA procedure (i.e., HINTS 3 and Battaglia, et al, 2008) and the success of the follow-up procedure used for the AA, we believe continuing to test the AA method in Cycle 1 will provide a more accurate assessment of the selection method both in terms of response rates and the composition of those who cooperate, and thus will better inform the selection method for HINTS moving forward. By allocating a smaller portion of the sample to the AA methodology we minimize the risk of significantly lowering the response rate for Cycle 1 while at the same time providing a large enough sample to adequately compare the two methodologies. We selected the NB method over the HC method because both methods attained about the same response rates, but the NB method is simpler to implement since it only requires a single version of the questionnaire.

Because we made some minor edits to the wording on the AA and NB respondent selection pages, the new versions are included in Appendix M_FINAL.

2. Updated Mailing Strategy

The Pilot mailing strategy was as follows:

• Advance letter – This was mailed to all sampled households in English only.

• First questionnaire mailing package - Mailed in English to all sampled households.

• Reminder/Thank you postcard – Mailed to nonresponding addresses in English,

• Second questionnaire mailing package – Mailed to nonresponding addresses in English

• Third questionnaire mailing package – A third questionnaire package, with both English and Spanish language forms, was sent to nonresponding addresses in linguistically isolated stratum, or addresses with an associated Hispanic surname.

• Spanish language questionnaires mailed immediately following a request from a sampled household – ongoing throughout field period.

• Targeted follow-up for addresses with AA respondent selection method, for which we did not receive a questionnaire for all reported adults in the household (English and Spanish, as appropriate)

For Cycle 1, we plan to use a modified version of these mailing procedures as a strategy for increasing response rates. The modified strategy includes replacing the prenotification letter with a mailing of the questionnaire. This results in an additional mailing of the questionnaire to all households, providing one more opportunity for participation. In the Pilot, we only mailed a third questionnaire mailing package to addresses within the Linguistically Isolated stratum or addresses with a Hispanic surname match. The modified strategy also provides further motivation to the most reluctant respondents by reducing the length of the questionnaire to just the key HINTS content for the third and final mailing. Last, we include a Spanish language questionnaire with the 2^nd mailing (rather than the 3^rd) for households with a Hispanic residents’ surname to make it easier for Spanish speakers to respond. However, all addresses with an associated Hispanic surname are excluded from the AA sample by design in order to avoid sending a mailing package with 4 questionnaires to a given address (2 English and 2 Spanish). Thus, this design choice means that households assigned to the all-adult method that can only read in Spanish will have to take the initiative to request a Spanish form in order to participate. However, based on the Pilot we anticipate a small number of Spanish language returns making a statistical comparison across methods infeasible, even if we had included the extra operational cost of mailing both Spanish and English questionnaires in the all-adult method. Table 1 below shows the modified mailing strategy planned for Cycle 1 data collection with anticipated dates, pending OMB approval.

Table 1: Mailing Strategy Planned for HINTS 4, Cycle 1 Data Collection

Mailing	Materials	Recipients	Change from Pilot?
October 24	First class mailing package: Cover letter English questionnaire (1-NB, 2-AA) $2 bill Return envelope (1-NB, 2-AA)	All sampled addresses	Yes – dropped the advance letter as first mailing
November 7	Reminder/Thank you postcard	All addresses for whom we haven’t yet received a completed questionnaire	No
November 28 – Spanish & English	Priority mail package: Spanish cover letter Spanish questionnaire English cover letter English questionnaire Return envelope	NB method only: Nonrespondents living in a LI area, or with an Hispanic surname	Yes – including both Spanish and English materials in 2nd mailing
November 28 - English only	Priority mail package: English cover letter English questionnaire (1-NB, 2-AA) Return envelope (1-NB, 2-AA)	NB and AA method: Nonrespondents without an Hispanic surname, living outside of a LI area.	No
December 12 – Spanish & English	First class mailing package: Spanish cover letter Spanish – abbreviated questionnaire English cover letter English – abbreviated questionnaire Return envelope	NB method only -Nonrespondents living in a LI area, or with an Hispanic surname	Yes – reduced content questionnaire was not used in Pilot; sending both English and Spanish questionniare
December 12 – English only	First class mailing package: English cover letter English – abbreviated questionnaire (1-NB, 2-AA) Return envelope (1-NB, 2-AA)	NB and AA method: Nonrespondents without an Hispanic surname, living outside of a LI area.	Yes – reduced content questionnaire was not used in Pilot

Throughout the field period, we also will mail Spanish language questionnaires to households calling the toll-free number to request Spanish questionnaires. This applies to both the NB and AA assigned addresses. Within the AA assigned addresses, we will continue with the tailored follow-up procedure implemented in the Pilot. We will mail additional questionnaire packages to any household assigned to the AA group that sends at least one completed questionnaire indicating that a larger number of adults live in the household than the number of completed questionnaires receipted for that household. The tailored follow-ups will begin approximately 7-10 days after the mailing of the second questionnaire mailing package.

We expect the modified mailing strategy to result in a slightly higher response rate than what we observed in the Pilot. As noted above, the NB method in the Pilot resulted in a 34% response rate and the AA method resulted in a final response rate of 20.5%. Moving forward to Cycle 1, even with the changes presented above, we are assuming a conservative response rate not much different from the rates attained in the Pilot. Thus, we assume about a 21% response rate for the all adult method, and about a 35% response rate with the Next Birthday method. The response rate assumption for the all-adult method may be an underestimate given that HINTS 3 which also used the all-adult selection method in the mail data collection attained a 32% response rate. However, since the Pilot all-adult method occurred with the same general content and in the same year as Cycle 1 data collection, we decided to use the Pilot rates for the Cycle 1 design assumptions. In fact, the difference between HINTS 3 mail response rates and the Pilot response rates for the all-adult method underscores the need to continue the evaluation and comparison of the two methods in Cycle 1 with a larger sample. The smaller number of cases allocated to the all-adult method reduces the risk to the overall response rate. See Table 2 below for the sample size and expected number of completed questionnaires by stratum for Cycle 1. The number of completed questionnaires remains the same as in the original clearance memo for Cycle 1.

Table 2: Cycle 1 Samples Sizes and Expected Completes by Stratum

	Respondent Selection Method: Next Birthday			Respondent Selection Method: All Adult			Total
	Sampled addresses	Household responses	Sampled addresses		Household responses	Sampled addresses		Completed interviews
High minority areas	4,650	1,570	965		368	5,615		1,938
Low minority areas	5,630	1,290	1,140		305	6,770		1,595
Total	10,280	2,860	2,105		673	12,385		3,533

3. Changes to Cover Letters

The cover letters for the Cycle 1 data collection were updated to better reflect the new mailing strategy. Specifically, the cover letters no longer include the three bullets giving examples of statistics generated from HINTS data. In the Pilot the invitation letter sent prior to the first questionnaire mailing package used bulleted examples as a way to clarify the type of information collected in HINTS. With the new strategy respondents will always receive a letter and a questionnaire in the same package. Since the questionnaire shows the actual information collected by HINTS, we dropped the bulleted examples of HINTS data from the cover letters, shortening the letter.

As noted, the new strategy includes a third questionnaire mailing package. We developed a third version of the cover letter to include with the third questionnaire mailing package. In this version we highlight the fact that we’ve shortened the questionnaire to just the most important elements as a way to reduce the amount of time required to participate. As noted above, we made this change to motivate participation from more reluctant households.

The only other change to the cover letters was to add language inviting non-English and non-Spanish speakers to call a toll-free number if they’d like to participate in HINTS in another language. We anticipate very few of such calls. Any callers who speak a language other than English or Spanish will be connected to a bilingual interviewer who will translate the mail questionnaire and complete the form over the phone for the caller.

All cover letters and the reminder postcard are enclosed as Appendix K_Final. Each letter in the attachment includes the language tailored to both the Next Birthday and the All Adult method, though each household will only see the language appropriate to their assigned respondent selection method. We’ve included the language for both methods in a single letter to reduce the number of attachments.

We did not make any changes to the targeted follow-up letter mailed to households in the AA method for whom not all adults returned a completed questionnaire. Nor did we change the thank-you/reminder postcard. Both these letters are included in the appendix.

4. Instrument Content Decisions

Content for Cycle 1 data collection comes from the all-inclusive items list submitted as part of the original HINTS 4 OMB submission (Appendix B1). As noted above, the Cycle 1 instrument was cognitively tested through OMB No. 0925-0589-09, Expiry Date 5/31/2011. Changes as a result of the cognitive testing were included in the split ballot instrument that was included in the pilot study. The Cycle 1 content reflects two categories of changes from the Pilot content:

• Selection of one version of a question included in the Pilot split ballot. Based on the results of the split ballot test implemented in the Pilot, NCI selected a single version of the tested questions for inclusion in Cycle 1. NCI HINTS management group selected the version of each question that best met the quality requirements and analytic goals in terms of preserving trends, and reduction of item level missing data.

• Further refinement of HINTS measurement goals across all 4 cycles of data collection. As the Pilot data collection began, NCI continued working with the key data users and subject matter experts to further refine and enhance the analytic goals across the four cycles of HINTS 4 data collection. As a result of those discussions, the NCI HINTS management group dropped some questions or subquestions for inclusion in the Cycle 1 instrument and added others. In addition, the wording of some questions was slightly changed or refined to improve data quality.

The content of the full-length Cycle 1 instruments is shown in Appendix B2_Full. Additionally, the reduced content version that includes only the questions that cover the key HINTS data elements is shown in Appendix B2_Short. This short version is a sub-set of items from the full-length version.

References:

Battaglia, Michael P., Michael W. Link, Martin R. Frankel, Larry Osborn and Ali H. Mokdad (2008) “An Evaluation of respondent selection methods for household mail surveys” Public Opinion Quarterly, Vol. 72: 459–469

Rizzo L, Brick JM, Park I.  (2004)  “A minimally intrusive method for sampling persons in random digit dial surveys”  Public Opinion Quarterly   68(2):267-74.

8