U.S. DEPARTMENT OF HOMELAND SECURITY
Generic Clearance for Formative Data Collections for Evaluation, Research, and Evidence-Building
OMB Control Number, 1601-New
Justification – Part A Supporting Statement
Overview of Information Collection:
This Information Collection Request establishes a new generic clearance to conduct formative studies that inform the U.S. Department of Homeland Security’s (DHS) evaluation, research, and evidence-building activities. DHS’s formative studies will collect data using well-established methodologies, including but not limited to questionnaires and surveys, semi-structured small group discussions or focus groups, observation, interviews, and cognitive interviews and user testing. The populations to be studied include DHS grantees, program and potential program providers and participants, researchers, practitioners, and other stakeholder groups involved in DHS programs, experts in fields pertaining to DHS evaluation and research, or others involved in conducting DHS evaluation, research, or evidence-building projects. Data collection may occur in-person, online, or by telephone, video, or other audio technology.
Conducting formative evaluation, research, and evidence-building activities will help the DHS better understand emerging needs and issues, identify evidence gaps, and ensure that DHS leadership and program offices have current data and information to implement DHS programs and initiatives successfully. The data from formative studies will be used to improve internal decision-making and inform future studies but will not be highly systematic nor intended to be statistically representative. Findings from these formative studies will not be generalized to the broader population and are not intended to inform major policy decisions. Information from data collection efforts may be shared in design and method documents; process maps, journey maps, conceptual frameworks, and logic models; performance metrics; as background materials for technical workgroups, informational documents, technical assistance plans; and evaluation or research reports. Shared findings will describe the study methods and limitations regarding generalizability and intended use.
Need & Method for the Information Collection.
Advancing the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) call to use data to build evidence and inform program design, the DHS requests approval of the Generic Clearance for Formative Data Collections for Evaluation, Research, and Evidence-Building to allow the DHS to conduct a variety of formative data collections. The DHS intends to design and conduct evidence-building activities, including formative evaluations of existing programs, processes, and new initiatives; logic model development and testing; process or journey mapping; research syntheses; survey, questionnaire, and metric development; analysis; and foundational fact-finding through descriptive and exploratory studies.
The DHS anticipates undertaking various new evaluation, research, and evidence-building activities related to the priority questions identified in the Agency's Learning Agenda and Annual Evaluation Plans, including on topics such as target violence and terrorism prevention, immigration, cybersecurity, infrastructure security, national preparedness, climate, equity, COVID‐19, and the federal workforce. Pursuant to Executive Orders 13985, Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, and 14058, Transforming Federal Customer Experience and Service Delivery to Rebuild Trust in Government, the DHS continuously seeks to ensure that the Agency's programs are effective, designed and delivered in a manner all people can navigate, reach underserved communities, promote equitable delivery of services, and meet customers' needs. In accordance with the DHS's commitment to advancing equity, improving service delivery, and promoting trust, the information collected under this generic clearance is necessary to enable the Department to gather customer and stakeholder feedback in a timely and efficient manner.
The DHS's formative studies will collect data using well-established methodologies, including but not limited to semi-structured small group discussions or focus groups, questionnaires and surveys, observation, interviews, and cognitive interviews and user testing (e.g., in-person, video, and audio collections). These data collections will allow for collaborative, ongoing, and actionable communications between the Agency and its customers and stakeholders and allow the DHS to identify the strengths and weaknesses of current programs, pilots, initiatives, and services. The efficient and timely formative collection efforts allow feedback to contribute directly to rapid cycle improvements of program management and the delivery of products and services. Conversely, the failure to engage in formative data collection substantially limits the DHS's ability to build evidence about programs, pilots, initiatives, and services and inform the development of future impact studies.
Use of the Information.
Under this generic clearance, the DHS plans to engage in a variety of formative and exploratory data collections with DHS grantees, program and potential program providers and participants, researchers, practitioners, and other stakeholders to fulfill the following goals:
maintain a rigorous and relevant evaluation and research agenda,
inform the development of DHS’s future evidence-building activities,
inform the delivery of targeted assistance and workflows related to program and grantee processes,
inform the development and refinement of recordkeeping and communication systems,
plan for the provision of programmatic or evidence-capacity-related training or technical assistance,
obtain grantee or stakeholder input on the development or refinement of program logic models, evaluations, and performance measures,
test activities to strengthen programs, and
preparation for summative evaluations.
The DHS’s formative studies will collect data using well-established methodologies, including but not limited to semi-structured small group discussions or focus groups, questionnaires and surveys, observation, interviews, and cognitive interviews and user testing (e.g., in-person, video, and audio collections). In accordance with OMB requirements, the DHS will submit information about individual information collection activities proposed under the generic clearance, including the data collection method, sampling strategy, a copy of the individual instruments or questionnaires, recruitment materials, protocols, and as appropriate, other supplementary materials describing the project. The DHS understands that OMB will make every effort to review materials for individual generic information collection requests within ten working days of submission. The DHS will make separate submissions for the clearance of full, non-formative data collection efforts and for data collections intended for statistical purposes as defined by 44 U.S. Code § 3561.
The data collected from formative evaluation, research, and evidence building activities will be used to improve internal decision-making, such as improvements of program management and the delivery of products and services, and to inform future studies but will not be highly systematic nor intended to be statistically representative. The data collection efforts are also not intended to produce influential information that is expected to have a genuinely clear and substantial impact on major policy decisions. Information gathered may inform future evaluation, research, and evidence-building, which could inform future influential public policy decisions.
Data collected under this generic clearance are not intended for use by other Federal agencies. However, the collected data may be shared with other Federal agencies where an evidence-building Interagency Agreement (IAA) exists, given that the data shared is for a specific program covered by the IAA. The study methods, limitations regarding generalizability, and intended used will be described in any produced work product.
Information from data collection efforts may be shared in design and method documents; process maps, journey maps, conceptual frameworks, and logic models; performance metrics; as background materials for technical workgroups, informational presentations, technical assistance plans; and evaluation or research reports. The aggregated results of formative work may also be prepared for presentation at professional meetings, professional journals, or an agency report or report to Congress. However, findings from these formative studies will not be generalized to the broader population and are not intended to inform major decisions. Additionally, shared findings will describe the study methods and limitations regarding generalizability and intended use. When necessary, the results will be labeled as formative or exploratory.
If these conditions are not met, the Agency will submit an information collection request to OMB for approval through the normal PRA process.
Use of Information Technology.
The DHS and its contractors will collect information electronically and/or use online collaboration tools, as appropriate, to reduce the burden. Specific information regarding the use of technology will be submitted with each individual information collection request.
Non-duplication.
No similar data are gathered or maintained by DHS or are available from other sources known to DHS. The purpose of this clearance is to inform DHS’s decision-making related to evidence-building and programmatic activities. To the maximum extent possible, the DHS will use existing data sources before seeking clearance under this generic information collection request, and the data collected through the formative studies approved under this generic clearance will not duplicate data collected elsewhere or otherwise obtainable.
Burden on Small Business.
Small business or other small entities may be involved in these efforts, but DHS will minimize the burden on them of information collections approved under this generic clearance by using sampling techniques, asking for readily available information, and using short, easy-to-complete information collection instruments. If an individual collection involves small organizations, the justification package will include further discussion to address this involvement.
Less Frequent Collection.
The DHS anticipates that all data information collected under this generic clearance will involve a one-time data collection. However, if a data collection effort involved a more frequent collection, the rationale and detail will be provided in the individual information collection request. Without the collection of formative data under this generic clearance, the Agency’s ability to solicit feedback from a broad and diverse stakeholder base will be limited, impacting our ability to identify the strengths and weaknesses of current programs, initiatives, and services and engage in rapid cycle improvements of program management and the delivery of products and services. DHS would not have timely information to understand emerging needs and issues, identify evidence gaps, and ensure that DHS leadership and program offices have current data and information to implement DHS programs and initiatives successfully.
Paperwork Reduction Act Guidelines.
There are no special circumstances. The information collected will be voluntary and will not be generalizable. Further, the information collected will not be labeled as statistical activities or used for statistical purposes as defined by 44 U.S. Code § 3561.
Consultation and Public Comments.
Per 5 CFR 1320.8(d), a 60-Day notice for public comment was published in the Federal Register on March 14, 2023, 88 FR 157331 Generic Clearance for Formative Data Collections for Evaluation, Research, and Evidence-Building (FR Doc. DHS-2023-0011. No comments were received. A 30-Day Notice was published on June 14, 2023, 88 FR 33878 were published in the Federal Register to obtain public comment on this collection.
The DHS did consult with other federal agencies engaging in formative data collections using a generic information clearance. Additionally, consultation with DHS staff and contractors engaging in formative evaluation, research, and other evidence-building activities will occur when preparing the individual data collections submitted under this generic information collection request.
Gifts or Payment.
Per OMB guidance, incentives are generally not appropriate for contractors, cooperators, grantees or program participants because they already have a pre-existing relationship with the agency. Incentives are most appropriate where participants are being asked to travel to a site to participate in a focus group or cognitive interview. Incentives are generally not appropriate for questionnaires/surveys.
If an incentive is proposed, a detailed justification based on the type of collection, population of respondents, and other circumstances will be provided in the individual information collection request. Per the Office of Information and Regulatory Affairs, Office of Management and Budget guidance document Questions and Answers when Designing Surveys for Information Collections (Updated Oct. 2016)1, justifications will focus on data quality, burden on the respondent, past experience, improved coverage of specialized respondents, rare groups, or minority populations; reduced survey costs; and/or equity.
Each justification will cite the research literature that demonstrates significant improvements in response rates and non-response bias when applied to similar participants, data collection methods, and data collection contexts. OMB does not consider it appropriate to use private sector market rates as a justification for incentives in government information collections. Where no evidence is available, DHS may propose a field test or experiment to evaluate the effects of the incentive.
The following includes expected ceiling amounts for different types of collections:
Focus groups where participates are expected to travel to a central site: Up to $75
Cognitive Interviews or similar exercises (intensive one-on-one probing of basis for thoughts) in which participants are expected to travel to a central site: Up to $40
Questionnaires/Surveys: TBD, under special circumstances
For any collection over 90 minutes, participants may be offered an incentive to account for incidental expenses (transportation, child care, lost wages, etc.).
Privacy & Confidentiality.
If the Privacy Act does apply to a collection, DHS will provide a Privacy Act statement, SORN, or other associated documentation, as appropriate. Participation in any formative data collection effort will be voluntary, and personally identifiable information will only be collected to the extent necessary. Respondents will be informed of all planned data uses, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. All data collection shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Agency regulations for private information.
Individual statements will be included with each generic information collection request submitted under this generic clearance, but in general, the government employee or contractor performing the data collection shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Agency regulations for private information. DHS and contractors shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. Any specific pledges evaluation staff must sign, as required by the contractor, will be described in individual information collection requests.
As necessary, DHS and contractors shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. DHS and contractors shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. DHS and contractors shall: ensure that this standard is incorporated into the Agency’s or contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Agency regulations. In addition, DHS and contractors must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
If a confidentiality pledge is deemed necessary, the Agency will only include a pledge of confidentiality supported by authority established in statute or regulation, supported by disclosure and data security policies that are consistent with the pledge. Any such pledge does not unnecessarily impede sharing of data with other agencies for compatible confidential use. The term confidentiality will not be used when no statutory basis exists, and instead, the process to protect respondent data and anonymity will be explained. A voluntary and informed consent statement will be submitted with each request when appropriate based on the data collection effort. Finally, information will not be maintained in a paper or electronic system from which they are directly queried by an individuals’ personal identifier.
Sensitive Questions.
Most of the questions included in these formative evaluation, research, and evidence-building activities will not be of a sensitive nature. Data collection requests may ask for demographic and socio-economic information, which can be considered sensitive. However, all responses are voluntary, and the majority of questions included in DHS’s formative data collection efforts will not be sensitive in nature. If an individual data collection effort includes potentially sensitive questions beyond demographic and socio-economic information, an additional explanation will be provided in the individual clearance request.
Burden Estimate.
Various data collection techniques and instruments will be used to conduct evidence-building activities under this clearance. However, the exact number of respondents and data collection instruments are not known at this time. The estimated burden for this new generic clearance for formative data collections considers DHS’s Learning Agenda evidence needs. The calculations are based on previous experience with formative data collection efforts and consultation with evaluation and research contractors with whom the DHS partnered.
Based on projections of projects to be conducted over the next three years, 12488 annual burden hours are estimated with the annual respondent cost of $349,788. Respondent cost estimates are based on the Bureau of Labor Statistics May 2021 median wage for all occupations of $28.01 per hour. No additional cost burdens to respondents for reporting, recordkeepers, or third-party disclosures are anticipated. The estimated number of respondents also assumes an industry-standard 40% response rate with 56,875 individuals invited to participate in the data collections.
Annual Burden Estimate:
Description of Data Collection Method |
Estimated Number of Respondents |
Frequency of Response |
Estimated Annual Responses |
Estimated Average Minutes/ Response |
Estimated Annual Hour Burden |
Estimated Annual Cost |
Small group discussions or focus groups |
3600 |
1 |
3600 |
90 |
5400 |
$151,254 |
Surveys or questionnaires |
6750 |
1 |
6750 |
30 |
3375 |
$94,533 |
Interviews |
2025 |
1 |
2025 |
60 |
2025 |
$56,720 |
Cognitive interviews or user testing |
1125 |
1 |
1125 |
90 |
1688 |
$47,281 |
Total |
22750 |
|
22750 |
|
12488 |
Respondent Costs Other Than Burden Hour Costs.
No additional costs are anticipated.
Estimated cost to the Government.
Based on projections of projects to be conducted over the next three years, the anticipated cost to the Federal Government is approximately $5,250,000 annually. The estimated costs consider Agency staff time, contractor payments, and any other necessary expense to collect the information approved under this generic clearance. Contractor and other expense costs will be covered by the data collection budgets from each individual evaluation, research, or evidence building project, and the costs will be fully described in the individual information collection requests.
Reasons for changes.
This is a request for a new collection.
Publicizing Results.
The primary purpose of data collected under this generic clearance is not for publication. However, because the formative data collection efforts are intended to inform DHS’s decision-making related to evidence-building and programmatic activities, the findings may be incorporated into documents and presentations available to the public. Such documents may include design and method documents; process or journey maps, conceptual frameworks, or logic models; performance metrics; background materials for technical workgroups, informational presentations, technical assistance plans; and evaluation or research reports. Although not anticipated, the DHS may receive requests to release the information (e.g., congressional inquiry, Freedom of Information Act requests) and will disseminate the findings when appropriate, following the Agency's guidelines. Shared findings will include a discussion of the limitations regarding generalizability and intended use, and when necessary, results will be labeled as formative or exploratory.
OMB Not to Display Approval.
All instruments submitted under this generic clearance will display the expiration date for OMB approval.
Exceptions to "Certification for Paperwork Reduction Submissions."
No exceptions are requested, and these activities proposed under this generic clearance request comply with the requirements in 5 CFR 1320.9.
Data collection requests under this generic clearance are not intended for statistical purposes as defined by 44 U.S. Code § 3561. Additionally, each formative data collection effort is expected to use a unique instrument, which will be submitted with the individual information collection request. All sampling methods used with requests made under this generic clearance are to gain insight from a broad and diverse set of stakeholders while simultaneously minimizing the burden, not intended to make inferences about a population.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Formative ICR_SSA |
| File Modified | 0000-00-00 |
| File Created | 2023-08-02 |