Pretesting Generic Clearance - Statement A

U.S. DEPARTMENT OF HOMELAND SECURITY

Generic Clearance for Pretesting Instruments and Procedures for Evaluation, Research, and Evidence-Building

OMB Control Number, 1601-New

Justification – Part A Supporting Statement

Overview of Information Collection:

This Information Collection Request establishes a new generic clearance to develop and test the Department of Homeland Security’s (DHS) information collection instruments and procedures for a wide range of evaluation, research, and evidence-building activities. DHS intends to use the resulting data to assess and improve the quality of data gathered through surveys used in DHS’s evaluation, research, and other evidence-building activities. These pretesting and piloting efforts will collect data using well established methodologies, including: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests. The populations to be studied include DHS grantees, program and potential program providers and participants, researchers, practitioners, and other stakeholder groups involved in DHS programs, experts in fields pertaining to DHS evaluation and research, or others involved in conducting DHS evaluation, research, or evidence-building projects. Data collection may occur in-person, online, or by telephone, video, or other audio technology. Data will be analyzed using well established methods, such as data tabulations to assess the results of instrument testing.

Conducting pretesting and piloting activities is necessary to improve future DHS information collections, resulting in higher quality studies. The information collected is meant to contribute to the body of knowledge on DHS programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

1. Need & Method for the Information Collection.

The Department of Homeland Security (DHS) requests approval of the Generic Clearance for Pretesting of Evaluation Data Collection Activities to allow DHS to use samples of more than nine participants in applying methods useful for identifying instrument and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions.

DHS studies its programs and the populations they serve, through rigorous evaluation, research and evidence-building activities. These include evaluations of existing programs, evaluations of innovative approaches to allow the Department to respond to its evolving threat environment with effective strategies and operations that ensure a safe, secure, and prosperous Homeland, research syntheses, and descriptive and exploratory studies. Pursuant to Executive Orders 13985, Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, and 14058, Transforming Federal Customer Experience and Service Delivery to Rebuild Trust in Government, the DHS continuously seeks to ensure that the Department's programs are effective, designed and delivered in a manner all people can navigate, reach underserved communities, promote equitable delivery of services, and meet customers' needs. This generic clearance allows us to identify if and when an instrument may be simplified for respondents, respondent burden may be reduced, among other possible improvements.

The information collections conducted under this generic are intended to be informative in nature; the studies may be iterative, as variation in questions or procedures are proposed, assessed, and retested. The pretesting and piloting of instruments is necessary to improve future DHS information collections, resulting in higher quality studies. The core methodology and target populations will be consistent and burden caps and token of appreciation structure are proposed in this request. The efficient and timely pretesting and piloting efforts allow feedback to contribute directly to more targeted and improved study designs. Conversely, the failure to engage in pretesting and pilot data collection limits DHS's ability to improve the quality of evidence about programs, pilots, initiatives, and services while reducing administrative burden to the public.

2. Use of the Information.

DHS will use the information collected to develop and test data collection instruments and procedures to assess and improve the quality of the data gathered through DHS’s research and evaluation studies. Pretesting and piloting information collection instruments and/or procedures is the main objective of the activities in this clearance. The goal of developing these instruments and refining procedures is to improve the evaluations, research, and evidence-building for DHS programs, pilots, initiatives, and services. DHS will use the results of information collections internally to inform subsequent information collection requests.

The information collected is meant to contribute to knowledge on evaluation, research, and evidence building for DHS. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

Individual information collection requests submitted under this generic clearance will include the guiding questions for the specific proposed activities.

Study Design

All of the methods proposed for instrument and procedure development will be used with either purposive or statistically representative samples of participants in DHS programs being evaluated; participants in DHS pilots and demonstrations, many of which are supported by DHS program grants; recipients of DHS grants and individuals served by DHS grantees; comparison group members; and other relevant populations, such as individuals eligible for DHS services. Data tabulations will be used to evaluate the results of instrument testing. All information collection activities conducted under this generic clearance will be voluntary and low-burden.

DHS will test a variety of instruments and procedures under this clearance. The exact nature of the instruments and the samples is dependent on each individual project and details will be provided for each individual information collection requests submitted. The particular samples included in future generic information collection requests will vary based on the content of the instrument being tested. The proposed methods for use under this clearance include the following:

• Cognitive and Usability Laboratory and Field Techniques: A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted one-on-one with respondents. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand the physical features of a survey, for instance, its display and navigational features. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire.

• Behavior Coding: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.

• Exploratory Interviews: A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign.

• Respondent debriefing questionnaires: In this method, standardized debriefing questionnaires are administered to respondents who participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.

• Split sample questionnaire experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 100 persons in a split sample test.

• Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.

• Pretests/Pilot Studies: These methodologies are used to test a preliminary version of the data collection instrument. Pretests are used to gather data to refine questionnaire items and scales and assess reliability or validity. Pilot studies are also used to test aspects of implementation procedures in addition to testing survey measurement issues. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.

Procedures for Clearance

Since the types of instruments included under the umbrella of the clearance are so varied, we cannot specify at this point the exact activities that will be involved in any particular information collection request. With each information collection request, we will provide OMB with a copy of instruments, supplementary materials, and a brief justification package in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. When iterative testing is conducted, initial instruments will be submitted for review and approval and any revised materials will be uploaded to ROCIS as a nonsubstantive change between each round of testing. A memo will detail any changes.

DHS understands that OMB will make every effort to review materials for individual generic information collection requests within 10 working days of submission. All information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. DHS will make separate information collection requests for full, non-developmental data collection efforts.

DHS will provide a report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance with subsequent overarching generic information collection renewals.

Other Data Sources and Uses of Information

Individual information collection requests submitted under this generic clearance will include information, as appropriate, about how the information collected may be used in concert with other sources of information (e.g., administrative data sources, prior data collections).

3. Use of Information Technology.

The DHS and its contractors will collect information electronically and/or use online collaboration tools, as appropriate, to reduce the burden. Specific information regarding the use of technology will be submitted with each individual information collection request.

4. Non-duplication.

No similar data are gathered or maintained by DHS or are available from other sources known to DHS. The purpose of this clearance is to better inform and improve the quality of DHS’s evaluation, research, and evidence-building. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.

5. Burden on Small Businesses.

Small business or other small entities may be involved in these efforts but DHS will minimize the burden on them of information collections approved under this clearance by sampling, asking for readily available information, and using short, easy-to-complete information collection instruments.

6. Consequences of Less Frequent Collection.

This generic clearance involves instrument and procedure development activities for each study connected with the clearance. This may include one-time collections or iterative testing, based on the specific situation. In all cases, without the proposed information collection activities, the quality of the data collected for DHS studies would suffer. Pretesting of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pretesting plan.

7. Paperwork Reduction Act Guidelines.

There are no special circumstances. The information collected will be voluntary and will not be generalizable. Further, the information collected will not be labeled as statistical activities or used for statistical purposes as defined by 44 U.S. Code § 3561.

8. Consultation and Public Comments.

Per 5 CFR 1320.8(d), a 60-day notice for public comment was published in the Federal Register on March 14, 2023 [88 FR 15731 - Generic Clearance for Pretesting Instruments and Procedures for Evaluation, Research, and Evidence-Building (FR Doc. 2023-0012)]. No comments were received. 30 Day notice was published in the Federal Register on June 14, 2023, 88 FR 38878

DHS did consult with other federal agencies engaging in pretesting and piloting data collections using a generic information clearance. Additionally, consultation with DHS staff and contractors engaging in pretesting and piloting activities will occur when preparing the individual data collections submitted under this generic information collection request.

9. Gifts or Payment.

Per OMB guidance, incentives are generally not appropriate for contractors, cooperators, grantees or program participants because they already have a pre-existing relationship with the agency. Incentives are most appropriate where participants are being asked to travel to a site to participate in a focus group or cognitive interview. Incentives are generally not appropriate for questionnaires/surveys.

If an incentive is proposed, a detailed justification based on the type of collection, population of respondents, and other circumstances will be provided in the individual information collection request. Per the Office of Information and Regulatory Affairs, Office of Management and Budget guidance document Questions and Answers when Designing Surveys for Information Collections (Updated Oct. 2016)¹, justifications will focus on data quality, burden on the respondent, past experience, improved coverage of specialized respondents, rare groups, or minority populations; reduced survey costs; and/or equity.

Each justification will cite the research literature that demonstrates significant improvements in response rates and non-response bias when applied to similar participants, data collection methods, and data collection contexts. OMB does not consider it appropriate to use private sector market rates as a justification for incentives in government information collections. Where no evidence is available, DHS may propose a field test or experiment to evaluate the effects of the incentive.

The following includes expected ceiling amounts for different types of collections:

• Focus groups where participates are expected to travel to a central site: Up to $75

• Cognitive Interviews or similar exercises (intensive one-on-one probing of basis for thoughts) in which participants are expected to travel to a central site: Up to $40

• Questionnaires/Surveys: TBD, under special circumstances

For any collection over 90 minutes, participants may be offered an incentive to account for incidental expenses (transportation, child care, lost wages, etc.)

10. Privacy & Confidentiality.

If the Privacy Act does apply to a collection, DHS will provide a Privacy Act statement, SORN, or other associated documentation, as appropriate. Participation in any formative data collection effort will be voluntary, and personally identifiable information will only be collected to the extent necessary. Respondents will be informed of all planned data uses, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. All data collection shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Agency regulations for private information.

Individual statements will be included with each generic information collection request submitted under this generic clearance, but in general, the government employee or contractor performing the data collection shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Agency regulations for private information. DHS and contractors shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. Any specific pledges evaluation staff must sign, as required by the contractor, will be described in individual information collection requests.

As necessary, DHS and contractors shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. DHS and contractors shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. DHS and contractors shall: ensure that this standard is incorporated into the Agency’s or contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Agency regulations. In addition, DHS and contractors must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.

If a confidentiality pledge is deemed necessary, the Agency will only include a pledge of confidentiality supported by authority established in statute or regulation, supported by disclosure and data security policies that are consistent with the pledge. Any such pledge does not unnecessarily impede sharing of data with other agencies for compatible confidential use. The term confidentiality will not be used when no statutory basis exists, and instead, the process to protect respondent data and anonymity will be explained. A voluntary and informed consent statement will be submitted with each request when appropriate based on the data collection effort. Finally, information will not be maintained in a paper or electronic system from which they are directly queried by an individuals’ personal identifier.

11. Sensitive Questions.

Most of the questions included in these pretesting activities will not be of a sensitive nature. However, it is possible that some potentially sensitive questions may be included in instruments tested under this clearance, such as demographic or socio-economic information. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them as much as possible before the actual survey is administered. Information about and justification for any sensitive questions will be included in the justification statement for each information collection request submitted under this generic clearance.

12. Burden Estimate.

Various data collection techniques and instruments will be used to conduct pretesting and piloting activities under this clearance. However, the exact number of respondents and data collection instruments are not known at this time. The estimated burden for this new generic clearance for information collections considers evaluation, research, and evidence-building identified in DHS’s Learning Agenda and Annual Evaluation Plans and other Agency evaluation planning activities.

Based on projections of projects to be conducted over the next three years, 3,825 annual burden hours are estimated with the annual respondent cost of $107,139. Respondent cost estimates are based on the Bureau of Labor Statistics May 2021 median wage for all occupations of $28.01 per hour. No additional cost burdens to respondents for reporting, recordkeepers, or third-party disclosures are anticipated. The estimated number of respondents also assumes an industry-standard 40% response rate with 8,975 individuals invited to participate in the data collections.

Annual Burden Estimate:
Instrument or Activity Type	Estimated No. of Respondents	Frequency of Response	Estimated Annual Responses	Estimated Average Minutes/ Response	Estimated Annual Hour Burden	Estimated Annual Cost
Interviews/Debriefings	715	1	715	90	1075	$30,111
Questionnaires	1000	1	1000	30	500	$14,005
Focus Groups	750	1	750	90	1125	$31,511
Usability Tests	1125	1	1125	60	1125	$ 31,511
Totals	3590		3590		3825	$107,139

13. Respondent Costs Other Than Burden Hour Costs.

There are no additional costs to respondents.

14. Estimated Annualized Costs to the Federal Government.

Based on previous experience, the anticipated cost to the Federal Government is approximately $2,245,000 annually. The estimated costs consider Agency staff time, contractor payments, and any other necessary expense to collect the information approved under this generic clearance. Contractor and other expense costs will be covered by the data collection budgets from each individual evaluation, research, or evidence building project, and the costs will be fully described in the individual information collection requests.

15. Reasons for changes in burden.

This is a request for a new collection.

16. Publicizing Results.

The primary purpose of data collected under this generic clearance is not for publication. However, because the pretesting and piloting data collection efforts are intended to inform DHS’s decision-making related to evidence-building and programmatic activities, results of these methodological studies may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. The results of these pretesting activities may be prepared for presentation at professional meetings or publication in professional journals. Results will be labeled as exploratory in nature and any limitations will be described. Although not anticipated, the DHS may receive requests to release the information (e.g., congressional inquiry, Freedom of Information Act requests) and will disseminate the findings when appropriate, following the Department's guidelines.

17. OMB Not to Display Approval.

All instruments submitted under this generic clearance will display the expiration date for OMB approval.

18. Exceptions to “Certification for Paperwork Reduction Submissions”

No exceptions are requested, and these activities proposed under this generic clearance comply with the requirements in 5 CFR 1320.9.

19. Surveys, Censuses, and Other Collections that Employ Statistical Methods.

Data collection requests under this generic clearance are not intended for statistical purposes as defined by 44 U.S. Code § 3561. Additionally, each pretesting and piloting data collection effort is expected to use a unique instrument, which will be submitted with the individual information collection request. All sampling methods used with requests made under this generic clearance are to gain insight from a broad and diverse set of stakeholders while simultaneously minimizing the burden, not intended to make inferences about a population.

1 https://obamawhitehouse.archives.gov/sites/default/files/omb/inforeg/pmc_survey_guidance_2006.pdf

8