Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Healthy Marriage and Responsible Fatherhood Performance Measures and Additional Data Collection
OMB Information Collection Request
Supporting Statement
Part B
FEBRUARY 2021
Submitted By:
Office of Planning, Research, and Evaluation (OPRE)
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Dr. Katie Pahigiannis, Senior Social Science Research Analyst, OPRE
Ms. Pooja Gupta Curtin, Social Science Research Analyst, OPRE
Part B
B1. Objectives
Study Objectives
The Administration for Children and Families (ACF), Office of Family Assistance (OFA), has had administrative responsibility for the federal funding of Healthy Marriage and Responsible Fatherhood (HMRF) Grant Programs since 2006. The authorizing legislation for the programs may be found in Section 403(a)(2) of the Social Security Act. OFA and the Office of Planning, Research, and Evaluation (OPRE) in ACF are proposing to repeat performance measure and other data collection activities for the HMRF grant program, while also expanding the number of participating grantees. A key objective is to facilitate the collection and reporting of quality and consistent performance measures data for all HMRF grantees funded in fiscal year (FY) 2020. For further background, please see Supporting Statement A.
Generalizability of Results
The performance measure data collection and reporting described in this ICR is intended to provide internally-valid descriptions of all HMRF programs in the 2020 cohort, not to promote statistical generalization to other programs or service populations.
Appropriateness of Study Design and Methods for Planned Uses
ACF requires HMRF grantees to collect performance measures data on their own grant operations and clients and to prepare and submit reports to ACF using those data. To facilitate these processes for grantees, ACF has developed a range of data collection and reporting instruments that are programmed into the nFORM (Information, Family Outcomes, Reporting, and Management) system. This web-based system allows for streamlined and standardized data collection and submission of grantee performance data through quarterly progress reports. nFORM is provided free of charge to grantees.
The proposed performance measures design and methods follow the previously approved approach that allows grantees to collect their own performance measures data using the nFORM data collection system (#0970-0460). nFORM was implemented with the 2015 cohort of HMRF grantees to improve the quality of data and the efficiency of data collection and reporting. The performance measures, data collection instruments, and data collection system were revised in 2020 to reflect ACF’s priorities for the 2020 cohort, better align the data with the characteristics of HMRF programs and clients, and improve the grantee user experience. For the 2020 cohort of HMRF grantees, ACF is also providing a continuous quality improvement plan template to support grantees’ program improvement efforts. Grantees will create plans using this template on paper and submit them to OFA.
The initial OMB approval and first renewal of the previously-approved ICR also included cross-site analyses of grantee implementation and impacts for the 2015 cohort of grantees. Similar activities for the 2020 cohort will be included in separate OMB packages as applicable. This request includes only descriptive analyses of performance measures across grantees to support grantee and ACF program monitoring and improvement.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
Using the nFORM system, each grantee will collect data on the applicable target populations associated with their grant type. These include healthy marriage (HM) adult individuals, couples, and youth; and responsible fatherhood (RF) community and reentering fathers. (Programs serving RF community fathers may serve mothers, but they are not a target population.) The nFORM system provides data at the grantee, program, and individual levels. For the 2020 cohort, grantees are estimated to serve over 90,000 clients per year across these target populations.
Sampling
No sampling techniques are employed for the performance reporting system. All HMRF grantees are required to collect and report on their performance using the nFORM system. A sample survey cannot be used, because it would not provide the full and complete performance measures data reflecting the experiences of all clients served across the 2020 cohort of grantees.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
ACF is proposing to repeat collection and reporting of a revision of the existing performance measures for the 2020 cohort of HMRF grantees to have standardized information on applicant characteristics, program operations, service delivery, and client outcomes. These measures were initially developed in 2014 after extensive review of the research literature and grantees’ past measures. The performance measures, instruments, and nFORM system were revised in 2020 to reflect ACF’s priorities for the 2020 cohort and further improve (1) the quality, utility, and clarity of the information collected, and (2) the user experience and access to data and reports.
The updated performance measures, instruments, and nFORM system clarify and align question, response, and item wording with programs’ and clients’ experiences, drop unnecessary items, and add new items to capture information that is most useful to grantees and/or ACF. Proposed item revisions for the 2020 cohort were cognitively tested with a small number of current or former clients in the 2015 cohort to identify areas of misinterpretation or confusion and inform further revisions. In addition, multiple enhancements to nFORM were implemented to align with best practices; support the collection of accurate, complete, and timely data; and improve the user experience while adhering to security requirements.
All of the instruments covered by this ICR are used to achieve the objectives described in B1. Each instrument is described in Table A.1 in Part A of this ICR, including the instrument name, respondent type, content and periodicity of instruments, and mode of administration. The data from the survey instruments and service data are used to generate the quarterly reports to ACF; data from the survey instruments, service data, and quarterly reports are also used for developing and implementing CQI plans.
B4. Collection of Data and Quality Control
Who will be collecting the data? HMRF grantee staff will use the nFORM system for most of the data collection and reporting activities described in Table B.1 above. Program clients will self-administer the client surveys (Instruments 1 and 4) after grantee staff launch the applicable survey for each client. In some cases, client surveys will be administered on paper and grantee staff will later enter the responses into nFORM. Grantee staff will access and complete the CQI plan template outside of the nFORM system.
What is the recruitment protocol? Grantees are responsible for developing their own recruitment plans based on the populations they are serving. Grantees may use nFORM data regarding how clients were referred to the program to inform future recruitment strategies.
What is the mode of data collection? All grantees are required to use the web-based nFORM system for performance measures data collection and reporting, to improve data quality and consistency across grantees and reduce the burden of data collection and reporting for the instruments in Attachments E through J. nFORM has a user-friendly interface accessible to authorized users from any computer with internet access, allowing for ease of data entry without purchasing or installing additional software or changing the configuration of their computers. The 2020 grantee cohort will use an enhanced version of the nFORM system used by the 2015 cohort. Through nFORM, grantee staff generate unique survey passcodes for each client when they are ready to complete an instrument. Program clients use computers or tablets (often provided by grantees) to self-administer the applicant characteristics, entrance, and exit surveys in nFORM. This method presents several advantages over interviewer-administered surveys. It ensures greater privacy, and respondents will be less likely to give socially desirable responses, particularly with sensitive questions (Turner et al. 1998; Tourangeau and Smith 1996). It also reduces burden for grantee staff who would otherwise need to administer the surveys. To address possible literacy limitations, respondents have the option to wear headphones and listen to a recording of the questions, known as Audio Computer-Assisted Self-Interview (ACASI). nFORM allows grantees to generate the required quantitative performance measures for quarterly reporting to OFA with only the touch of a button, thereby minimizing grantee burden while maximizing cross-grantee consistency and quality of performance data. Grantees will complete the CQI plan template on paper, outside of nFORM, and submit the completed plan directly to OFA via email.
How are the data collection activities monitored for quality and consistency? The nFORM system will facilitate the collection and reporting of high-quality and consistent performance measures for all 2020 HMRF grantees at the grantee and client levels. nFORM includes automatic data quality checks for numerous fields to improve data reliability. If grantee staff enter unusual or unlikely values in a particular field, the system will prompt users to check the value. For some fields, the response values are restricted; for others, grantee staff are able to override the check. In addition, nFORM includes reporting features for grantees that can be used to assess data quality, such as a query tool and operational reports.
What data evaluation activities are planned as part of monitoring for quality and consistency in this collection? Grantees are required to designate data managers who are responsible for routinely monitoring the quality and consistency of their data collection activities using nFORM. Improvement goals can be incorporated into grantees’ CQI plans and strategies. nFORM system users will receive detailed training and technical assistance on nFORM prior to their initial use and throughout their grants. They will also have continuous access to nFORM and CQI help desks to raise any questions regarding the performance measures, nFORM, or CQI. Grantees will submit quarterly reports to ACF, which will use the data to assess grantees’ progress and performance.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The instruments described in Table B.1 are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. The data collection procedures, the nFORM system, and training and technical assistance provided to grantee staff described in B4 are designed to maximize response rates and data reliability. These efforts have been successful in ensuring strong response rates with the 2015 cohort of HMRF grantees and will continue for the 2020 cohort. As of July 1, 2020, grantees in the 2015 cohort had completed enrollment for 228,199 clients. Of these, 214,927 (94.2 percent) had completed an entrance survey and 144,483 (63.3 percent) had completed an exit survey.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.
B6. Production of Estimates and Projections
The data will not be used to generate population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
Previous responses in Section B4 of this supporting statement describe how nFORM was designed to standardize and improve the reliability of the data collected and reported by grantees. Design features include automatic data quality checks during data entry and automated production of quarterly performance reports using data entered in nFORM.
Data Analysis
Descriptive statistics (such as counts, percentages, means, and modes) will be computed to summarize grantee and client experiences for program monitoring and improvement. These analyses include client enrollment, characteristics, services received, and outcomes. Quarterly grantee reports and ad hoc analyses will not be made public. Comprehensive annual reports and a final report with summary statistics across grantees will be made public.
Data Use
The comprehensive performance measure data collected by HMRF grantees is a valuable resource for program monitoring and improvement, as well as for research designed to broaden understanding of HMRF programs. To support future research efforts, ACF and the nFORM contractor will develop a data archiving plan that identifies the most appropriate entity with which to archive a restricted use file of de-identified data at the end of the grants, so that the data are accessible to interested researchers.
Existing data documentation, including the nFORM user manual and data dictionary, will be updated to reflect enhancements to the system for the 2020 cohort of grantees. Additional documentation will be developed as part of ACF’s plan for archiving de-identified data for research purposes. This will include a communication plan with the identified archiving entity, a description of the data and supporting documentation, storage and security requirements, access information, analytic file formats, and a publishing schedule for the data.
B8. Contact Person(s)
The performance measures, instruments, and reports were developed by staff in OFA and OPRE in partnership with Mathematica. Staff at Mathematica were consulted on statistical methods, including:
Dr. Sarah Avellar
Senior Researcher
Mathematica
Inquiries regarding statistical aspects of the study should be directed to:
Senior Social Science Research Analyst
Office of Planning, Research, and Evaluation
Administration for Children and Families
U. S. Department of Health and Human Services
Ms. Pooja Gupta Curtin
Social Science Research Analyst
Office of Planning, Research, and Evaluation
Administration for Children and Families
U. S. Department of Health and Human Services
Further consultations will be made with statistical experts in developing the data archiving plan.
Attachments
ATTACHMENT A: Legislative Authority
ATTACHMENT B: 60-DAY Federal Register Notice
ATTACHMENT C: Institutional Review Board Exemption for ACF-Sponsored Analysis of Performance Measures
ATTACHMENT D: Assumptions for Calculations of Burden Estimates
ATTACHMENT E: Instrument 1: Applicant Characteristics
ATTACHMENT F: Instrument 2: Program Operations
ATTACHMENT G: Instrument 3: Service Delivery Data - nFORM Service Receipt
ATTACHMENT H: Instrument 4: Entrance and Exit Surveys
ATTACHMENT I: Instrument 5: Semi-annual Performance Progress Report (PPR)
ATTACHMENT J: Instrument 6: Quarterly Performance Report (QPR)
ATTACHMENT K: Instrument 7: CQI Plan
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Pooja Curtin |
| File Modified | 0000-00-00 |
| File Created | 2021-03-02 |