PAPERWORK REDUCTION ACT SUBMISSION (statistical methods) - 8.25.07

COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Potential Respondent Universe

The respondent universe for this collection of information will be all non-institutionalized adults 18 and older who reside in households with landline telephones in the 50 states and the District of Columbia. A response rate of approximately 60% is anticipated in this collection of information.¹ This estimate is based on historical data as well as measures that will be employed to maximize response rate discussed below.

2. Statistical methodology for collection and sample selection

Five of the seven studies in this collection have identical sample sizes and will use identical procedures for sample selection, interviewing, data processing, tabulation, analysis and reporting. These five studies are identified in Table 1 and designated by the names Messaging Survey, Benchmark Survey, Tagline Testing, Tracking Survey One and Tracking Survey Two. The surveys will be conducted using computer-assisted telephone interviewing (CATI) technology, which effectively reduces many interviewing, recording and tabulation errors.

Households will be selected using a Random Digit Dialing (RDD) procedure through a database-assisted sampling methodology that contains working residential telephones for the entire United States to produce a single-stage random sample of residential telephone numbers. Although the number of households that rely exclusively on cell phones is increasing, recent estimates from the Centers for Disease Control and Prevention indicate that approximately 175 million Americans currently have a landline while 15.8% have no landline, with about 13% relying only on cell phones.²

The database is used to generate a random sample in which every telephone number has an equal probability of selection. Identification of the respondent will be achieved by the most recent birthday method. Only one respondent per household will be interviewed. If the respondent is unavailable throughout the study period, the household will become ineligible.

Information will be collected by experienced and specifically trained telephone interviewers. Quality control will be assured by periodic monitoring of on-going interviews throughout the study.

1. Lu, R.P., Response Rates Achieved in Government Surveys: Results of an OMB Study, FCSM Statistical Policy Working Paper 35.

2. Blumberg, SJ, Luke, JV, Wireless Substitution: Early release of Estimation based on data from the National Health Interview Survey, July-Dec. 2006, NCHS, May 14, 2007 www.cdc.gov/nchs/nhis.htm

3. Estimation Procedure

Following data collection and cleaning, data will be weighted to match the most recent Census estimates for sex, education, age, and race/ethnicity.

For analyses of the full sample in three of the five telephone surveys, the proposed sample size (900 adults) will provide a margin of error based on sampling of approximately ± 3.26 percentage points at the 95 percent confidence level. For the survey with 800 respondents, the margin of error based on sampling is ± 3.46, and for the survey with a sample of 750, the margin of error is ± 3.57.For analyses of subgroups, the margin of error will be larger but within acceptable limits for standard demographic variables such as age, gender, education, income and race/ethnicity).

Two studies, the In-depth Interviews and the Visual Testing study, are of a different nature than the five studies that will use the methodology described above. The In-depth interviews are a more qualitative investigation into a sub-population of special interest. The audience consists of bank tellers and individuals employed in businesses in which they handle cash transactions relatively frequently. This audience differs from general consumers in that they are more likely to be focused on currency-related issues as a result of their occupation, and because they also serve as transmitters of information to other consumers.

To better understand this audience, the collection includes a qualitative set of in-depth telephone interviews to be conducted with 150 bank tellers and “cash handlers.” Potential respondents will be identified through a sample selected randomly from a broad nationwide list developed by Dun & Bradstreet. Respondents will be screened to ensure they meet the specified criteria. Of the 150 respondents, approximately half will be bank tellers and half employees who operate a cash register or handle cash at retail outlets. While not a statistically reliable and projectable sample, this sample will provide the insight needed.

The second study that would use an alternative procedure is the Visual Testing study, designed to evaluate alternative executions of education materials. Because the core of this study is to obtain feedback on visual materials, such as brochures or posters, a telephone-only approach is not feasible. Because the essence of materials testing requires that respondents be shown draft materials so they can react to visual designs, colors, layout, flow etc., this study cannot be conducted by telephone.

While qualitative approaches such as focus groups are often used to evaluate materials of these types, easy access to a large proportion of American consumers on the Internet has made this a cost effective alternative approach that allows respondents to see and evaluate materials.

Available quantitative options include in-person interviews, a combination of phone and mail, and online surveys. Although conducting these interviews in person or via a combination of phone and mail would offer the advantage of a random sample of consumers, the costs and time required to conduct a sufficiently large number of in-person interviews would be both very expensive and inefficient. Conducting this study online offers several advantages, including the efficient use of federal funds, timeliness as well as being less intrusive to respondents and offering a lower respondent burden. Showing materials to respondents online can be done much more efficiently than in person, and can also be done at the respondent’s convenience.

Although Internet access today is very broad and is believed to reach a majority of Americans, selecting a random sample of adults via the Internet remains problematic. However, given the need to balance costs, limit respondent burden, as well as time constrains built into the education program schedule by the fixed timing of the introduction of new notes, we believe an online approach would be most efficient and adequately serve the purposes of the study. Although conducting the study online through a large panel of Internet users would not be projectable to the larger population of American consumers with a specific degree of accuracy, it will allow us to show materials to a much larger number of consumers than would be possible using a qualitative approach such as a series of focus groups.

In addition, the study will be used to look at the within sample variances in responses to different visual executions, which are likely to produce useful comparative data. Further, the online survey will contain numerous questions that are identical to questions that will be asked in surveys that will be conducted using an RDD telephone methodology. Having this data from a general consumer population may provide a way to appropriately weight the data produced by the Internet survey.

4. Use of periodic data collection cycles to reduce burden

As noted earlier, the various studies will be conducted sequentially to collect data as needed as the public education program develops and is implemented, as well as to avoid overburdening individual respondents.

5. Methods to Maximize Response Rates

In an effort to increase response rate, the agency plans to instruct its contractor to take the following measures:

• up to 12 call attempts to complete an interview;

• extend data collection period to 30 days; and

• conduct a non-response study to identify potential non-response biases and

adjust estimates statistically, if necessary.

Increasing the number of call attempts and extending the data collection period have often been used by survey organizations as part of an effort to increase telephone survey response rates. Studying non-response may help the agency in identifying significant non-response biases. Existing research has shown that non-response biases in random digit-dialing national telephone surveys may not be as severe as commonly suggested. For example, Keeter et al. (2000) found no measurable differences in findings between a survey with a response rate of 36% and an identical survey with a response rate of 61%, even though potential respondents in the latter were sent advance letters and a $2 incentive.³ Other studies suggest that the large majority of completed interviews will have been after 10 callbacks, and that 12 callbacks is the optimum number to balance cost and potential response bias. ^{4, 5}

Calls will be staggered over times of the day and days of the week to maximize the chances of making contact with a respondent. No-answers after five attempts at initial contact will be regarded as non-households and eliminated from the sample.

In addition, all interviewers and supervisors will be trained on the specifics of the survey by a member of the project's professional staff. This will include an explanation of the importance and purpose of the collection of information as well as a thorough review and practice reading of the entire information collection instrument. Respondents who initially refuse to participate will be assigned to conversion specialists who have high completion rates, who will attempt to complete the interview on a different day.

Interviewers will be monitored by a supervisor throughout the course of the collection of information. Production rates and call dispositions will be monitored each day to detect and resolve any problems or discrepancies quickly. Callback appointments will be made with respondents who are unable to complete the interview when initially contacted.

To ensure quality control, the contractor will maintain complete call disposition records on every household contacted. In no case will telephone numbers be abandoned prior to achieving one of the following: (1) completed interview, (2) completed conversion attempt or refusal, (3) exhaustion of callbacks, (4) determination that a household is not eligible, and (5) exhaustion of initial contact attempts. When a household is determined to be ineligible, the basis for the determination will be recorded.

3 Keeter, S., Miller, C., Kohut, A., Groves, R.M., and Presser, S. 2000. "Consequences of Reducing

Non-response in a National Telephone Survey." Public Opinion Quarterly 64: 125-148.

4 Dennis, M., Mathiowetz, N.A., Saulsberry, C., Frenkel, M., Srinath, K.P., Roden, A.-S., Smith, P.J., and

Wright, R.A. 1999. "Analysis of RDD Interviews by the Number of Call Attempts: The National

Immunization Survey, 1999 Conference of the American Association for Public Opinion Research, May 13-16, 1999

5 Srinath, K.P., Battaglia, M.P., Cardoni, J., Crawford, C., Snyder, R., and Wright, R.A. (2001) Balancing Cost and Mean Squared Effort in RDD Telephone Surveys: The National Immunization Survey. 2001 Proceedings of the Section on Survey Research Methods, American Statistical Association, Alexandria, VA.

Survey Response and Follow-Up

The response rate will be measured and evaluated as defined by the Council of American

Survey Research Organizations (CASRO). The CASRO response rate (CRR) is considered the industry standard response rate formula for this type of RDD survey. It is defined as follows:

CRR = (number of completed interviews)/(Estimated number of eligibles)

= (number of completed interviews)/(Known Eligibles + Presumed Eligibles)

Presumed eligibles are determined by applying the same ratio of the known eligible to known ineligible to those whose eligibility status is unknown

A test for non-response bias will be conducted upon completion of the surveys by examining the responses of those who responded quickly compared to those who required many callbacks or contact by a conversion specialist. Differences identified by the tests may result in the need for adjustments to the weighting to account for these differences.

Pre-testing

For each study, the data collection contractor will conduct interviews with 25 randomly-selected adults after OMB approval of the collection of information to identify and correct any problems connected with item response, respondent selection, interviewer, instructions, skip patterns, and design of the computer-assisted-telephone interview program. Results of these 25 interviews will be reviewed and adjustments made as warranted before proceeding with the data collection. The review process includes reviewing the overall design, skip patterns, proofing, CATI programming, and timing of the survey. Thus, potential problems associated with the questionnaire design are expected to be minimal. The pretest will be conducted during the initial stage of the interviewing process. After these initial interviews are completed, the interviewing process will be stopped and examined for potential problems.

Individuals Involved in Statistical Consultation and Information Collection

The contractor, Burson-Marsteller, and its subcontractor, Penn Schoen & Berland, will collect the data on behalf of the Bureau of Engraving and Printing. The research process for all studies will be overseen by Amy Leveton, Managing Director at Penn, Schoen & Berland, and by Steven Einhorn, Managing Director at Burson-Marsteller. Consultation on statistical issues is provided by Oleg Urminsky, Chief Statistician, Penn, Schoen & Berland.

Amy Leveton

Managing Director

Penn, Schoen & Berland

1110 Vermont Ave. NW

Washington DC 20005

202-962-3045

Steven Einhorn

Managing Director

Burson-Marsteller

1101 Vermont Ave. NW

Washington DC 20005

202-530-4527

Oleg Urminsky

Chief Statistician

Penn, Schoen & Berland

245 East 92nd Street
New York, NY 10128

212-534-4000.