Pretesting Generic Clearance - Statement B

U.S. DEPARTMENT OF HOMELAND SECURITY

Generic Clearance for Pretesting Instruments/Procedures for Evaluation, Research, and Evidence-Building

OMB Control Number, 1601-New

Part B Supporting Statement

B. STATISTICAL METHODS

The objectives of the information collection requests submitted under this generic clearance are to develop and test information collection instruments and procedures for a wide range of evaluation, research, and evidence-building activities. The activities will help the Department of Homeland Security assess and improve the quality of the data gathered through surveys for DHS’s evaluation, research, and other evidence-building activities requiring statistical analysis and employing statistical methods.

The ICRs submitted under this generic will employ methods, as described in Supporting Statement A (see A2, Study Design). These methods are appropriate for the pretesting of data collection instruments and procedures. Each individual information collection requested submitted under this generic clearance will address appropriateness of study design and methods for the specific project.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Public sharing of this data is limited to the context described in Supporting Statement A.

1. Universe and Respondent Selection

DHS will use the data collected for the purpose of developing and testing information collection instruments and procedures. Given the nature of the information collection activities under this request, the universe and respondent selection varies considerably. Generally, the testing activities undertaken as part of this clearance have and will involve purposive samples with respondents selected either to cover a broad range of demographic subgroups or to include specific characteristics related to the topic of the survey. In some instances, a statistically representative (probability) sample may be drawn, for example, for mail surveys or to permit statistical comparisons about the effectiveness of alternative procedural treatments.

Respondents could include participants in DHS programs being evaluated; participants in DHS pilots and demonstrations, many of which are supported by DHS program grants; recipients of DHS grants and individuals served by DHS grantees; comparison group members; and other relevant populations, such as individuals eligible for DHS services.. A description of the plans for selecting respondents will be provided to OMB as part of each Gen IC request.

The individual information collection requests submitted under this generic clearance are intended to pretest or pilot data collection efforts. The data collected under this generic clearance will not be used to generate population estimates, either for internal use or dissemination. The results are not intended to promote statistical generalizations, but DHS will work to select respondents with characteristics typical of each study’s target population. For example, for instruments that are likely to be used with respondents who speak different languages, DHS may perform cognitive testing to ensure different populations understand questions as translated. Each individual information collection request will address expected generalizability of results.

2. Procedures for Collecting Information

Data collection procedures for the testing conducted under this clearance will vary but are likely to include in-person or telephone interviews, mail surveys and discussion guides. Efforts will collect data using well established methodologies, including: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments, (f) focus groups, and (g) pilot studies/pretests. Statistical results will address a variety of issues including response rates, item non-response rates, frequency distributions of data items, and analysis of behavior coding and respondent debriefing data. Depending on the nature of the research, DHS staff and research and evaluation contractors will have responsibility for data collection and analysis. Each ICR will provide specific information about data collection procedures for each individual information collection.

Development of Data Collection Instruments

The purpose of this information collection request is developmental data collection efforts that enable DHS to test and improve data collection instruments and procedures prior to submitting information collection requests for full, non-developmental data collection. DHS expects that all the tests conducted under this clearance will result in improved instruments and/or procedures and thus reduced respondent burden.

Advice on statistical aspects of each individual survey will be sought, as appropriate, as the testing program proceeds. Additional information about consultation and contact information will be provided to OMB within each individual information collection request.

Data Handling

Individual information collection requests submitted under this generic clearance will describe, as appropriate for the proposed activities, the following:

• Procedures for editing to mitigate or correct detectable errors, including checks built into computerized instruments.

• Procedures to minimize errors due to data entry, coding, and data processing.

Data Analysis

Individual information collection requests submitted under this generic clearance will describe data analysis plans for the proposed activities. This will include, as appropriate, methods for any statistical tests, planned analytical techniques to be used, and how the information collected will be used or interpreted in conjunction with any other sources of information. DHS anticipates that data tabulations will be used to assess the results of instrument testing.

Data Use

DHS will use the results internally to inform subsequent information collection requests.

Results of these methodological studies may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. The results of these pretesting activities may be prepared for presentation at professional meetings or publication in professional journals. When necessary, results will be labeled as exploratory in nature and any limitations will be described.

Individual information collection requests submitted under this generic clearance will describe all planned uses of data, including a description of plans to share of the information and any documentation that will be included to improve understanding of how to interpret, analyze, and assess information from the collection.

3. Methods to Maximize Response

Response Rates: Expected response rates will vary for individual information collection requests. Information about expected response rates will be provided with individual information collection request.

NonResponse: In general, callbacks will be used to maximize response rates in telephone surveys; reminder phone calls, letters, or second questionnaires will be used to maximize response rates in mail surveys. Reminder phone calls and/or letters to participants will be used to encourage them to keep their appointments. Tallies will be kept of the number of non-respondents to all testing activities. More specific information will be provided to OMB at the time the individual information collection request are submitted. Each individual information collection request will provide specific information about methods to maximize response rates and deal with nonresponse, as appropriate.

4. Testing of Procedures

The purpose of this information collection request is to test data collection instruments and survey procedures.

5. Contact Persons

Individual information collection requests submitted under this clearance will include the name and contact information for the lead individual(s) who can answer questions about the data collection activities.

3