Supporting Statement Part B_120823_clean

SUPPORTING STATEMENT B

U.S. Department of Commerce

U.S. Census Bureau

Generic Clearance for Internet Panel Pretesting and Qualitative Survey Methods Testing

OMB Control No. 0607-0978

SUPPORTING STATEMENT PART B - (Questions from OMB are in black text, guidance from DOC for how to respond to the questions is in blue text; remove all blue text before submission.)

Agencies are instructed to complete Supporting Statement Part B if they are using statistical methods, such as sampling, imputation, or other statistical estimation techniques; most research collections or program evaluations should also complete Part B. If an agency is planning to conduct a sample survey as part of its information collection, Part B of the ICR supporting statement must be completed, and an agency should also complete relevant portions of Part B when conducting a census survey (collections that are sent to the entire universe or population under study). For example, an agency doing a census of a small, well-defined population may not need to describe sampling procedures requested in Part B, but it should address what pretesting has taken place, what its data collection procedures are, how it will maximize response rates, and how it will deal with missing unit and item data.

Agencies conducting qualitative research studies or program evaluations, including case studies or focus groups, should also complete the relevant sections of Part B to provide a more complete description of the use of the information and the methods for collecting the information.

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Question 17 in ROCIS is checked "Yes", responses to the following questions should be included in the Supporting Statement to the extent that it applies to the methods proposed. If there are no statistical methods involved, a response stating this should be provided.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The response should describe the respondent universe and how that universe was or will be selected. The method of sampling should also be explained. With respect to the response rate, the narrative should cover the actual percentage response rate that is anticipated. Additionally, the agency should describe the efforts that will be undertaken to ensure a high response rate including pre-survey telephone calls or correspondence, post-mailing reminders, etc. If correspondence will be used to boost the response rate, copies of all letters, telephone scripts or other materials should be included in the package. If the collection has been conducted previously, then a summary of that activity should be provided including the response rate achieved. Collections anticipating a response rate less than 80 percent need complete descriptions of how the expected response rate was determined, a detailed description of steps that will be taken to maximize the response rate, and a description of plans to evaluate nonresponse bias. Agencies also need a clear justification of why the response rate is adequate based on the purpose of the study and the type of information that will be collected.

The data collected will be used for questionnaire development and pretesting activities rather than to produce estimates about populations; thus, the anticipated response rates to these surveys would be less than 80 percent. Some of the small-scale testing activities undertaken as part of this clearance will involve nonprobability or opt-in samples, with respondents who self-select to participate in the survey or usability test. In other instances, a probability sample may be drawn, for example, for an opinion survey or online surveys, that would permit statistical inferences about the effectiveness of alternative advertising treatments. For usability testing, we might also send emails to a specified group, such as the Census Data Center staff. A description of the plans for selecting respondents for each individual test will be provided to OMB at the time the individual clearance requests are submitted.

2. Describe the procedures for the collection of information including:

   • Statistical methodology for stratification and sample selection,

   • Estimation procedure,

   • Degree of accuracy needed for the purpose described in the justification,

   • Unusual problems requiring specialized sampling procedures, and

   • Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The response should include an answer to each of the subparts of this question. Where the collection is considered ongoing - i.e., has been conducted previously and is continuing for the foreseeable future - a description of any changes that have been made in the procedures or statistical methodology of the collection since the last approval should be discussed.

There are no substantive changes to methodology from the last renewal. Data will be collected via the Internet, telephone, mail and in-person studies. Remote usability testing will be through the Internet. Statistical results will include response rates, click rates, item nonresponse rates, usability paradata, frequency distributions of data items, and analysis of opinion data. More specific information about data collection procedures will be contained in the description provided to OMB at the time the individual clearance requests are submitted.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

A thorough discussion of what the agency's plan of action for dealing with non-response must be provided. If the collection is categorized as qualitative and does not necessarily employ a sampling frame or other proven statistical methods, then an explanation of exactly what practical utility the collection will have for the agency must be discussed. This discussion should include a listing of the specific uses the agency will plans for the data collected.

In general, reminder emails, calls, mailing or text messages may be used to maximize response rates in surveys. These may be the topic of a given experiment, or may be used to maximize overall response. Tallies will be kept of the number of nonrespondents to all testing activities that involve an invitation to the individual survey (if respondents are invited via a link on our website, we will not have a measure of nonresponse). More specific information will be contained in the description provided to OMB at the time the individual clearance requests are submitted.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The response should fully describe any planned tests of the collection including who is in the test sample, how they were chosen, what evaluation criteria for the test will be or were used, and specific comments from the participants of the test. If applicable, a summary of how the collection instrument or statistical methods were changed as a result of the test phase should be provided.

This entire submission consists of tests of data dissemination websites, advertising, data collection instruments and survey/census procedures. We expect that all the tests conducted under this clearance will result in more usable websites, simpler questionnaires, easy-to-understand advertisements and/or procedures and thus reduced user or respondent burden.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Include all individuals who have contributed to or commented on the survey, sample frame, statistical methods or other aspects of the collection. Make sure that all representatives are referenced, or the submission will be returned to the agency for this review to occur.

Nancy Bates, Dr. Paul Beatty and Dr. Peter Miller have served as general methodological consultants for this project. Reg Baker, Consultant at Market Strategies International, has also provided external guidance on this topic previously. Additional advice on statistical aspects of each individual survey will be sought as the testing program proceeds. Depending on the nature of the research, staff from subject-matter divisions, operational divisions, and the Center for Behavioral Science Methods will have primary responsibility for data collection and analysis. The specific research project will also determine whether the data will be collected by the Census Bureau or through a contractor. Aleia Clark Fobia (301-763-4075) is the contact person for general questions about data collection and analysis. Other contact persons for questions regarding data collection and statistical aspects of the design will be provided to OMB at the time the questionnaires are submitted.

Page | 2