Final Supporting Statement A 10-21-20

Supporting Statement A

Maternal Health Portfolio Evaluation

OMB Control No. 0906-XXXX-New

Terms of Clearance: None.

A. Justification

1. Circumstances Making the Collection of Information Necessary

This is a new Information Collection Request (ICR) requesting approval to collect data for a portfolio-wide evaluation of Maternal Health (MH) programs funded by the Health Resources and Services Administration.

Maternal mortality (MM) and severe maternal morbidity (SMM) are significant public health issues in the United States. According to the most recent CDC MMWR from 2019, 700 women die each year from pregnancy-related complications,¹ and the rate of women dying from pregnancy-related complications has increased steadily during the past three decades.² Researchers cite the lack of standardized approach to emergency obstetric care and hospital quality of care as contributory factors.³ Another factor is the closure of many obstetric units in rural counties in the United States, which can be a barrier to pregnant women receiving obstetric care.⁴

HRSA’s Maternal Health Innovation Strategy aims to address MM and SMM through the life course perspective and the health equity perspective through a suite of five maternal health programs (MH Programs). The MH portfolio includes 15 awardees across five programs: 1) State Maternal Health Innovation Support and Implementation Program (Supporting MHI); 2) State Maternal Health Innovation Program (State MHI); 3) Alliance for Innovation on Maternal Health (AIM); 4) Alliance for Innovation on Maternal Health Community Care Initiative (AIM-CCI); and 5) Rural Maternity and Obstetrics Management Strategies Program (RMOMS).

The proposed evaluation will identify overarching strategies that are common across the grantees, collect primary and secondary data elements for each of these strategies, and conduct analyses using a mixed-methods approach. This will help the HRSA Maternal and Child Health Bureau identify individual and/or collective strategies, interrelated

activities, and common themes within and across the MH programs that may be contributing to or driving improvements in key maternal health outcomes.

The MH evaluation will use a variety of primary and secondary data sources to answer each evaluation question. The MH evaluation will include primary data collection through four primary data collection activities: 1) semi-structured interviews with grant staff, 2) semi-structured interviews with HRSA Project Officers, 3) quantitative data collection through tailored web-based data collection tools; and 4) a partnership survey administered to each grantee and their partners.

• 1. Annual semi-structured interviews with HRSA Project Officers. These interviews will provide important context for the MH portfolio and document grantee progress, challenges, and successes. Because each of the project officers oversees multiple grantees, and the evaluators will gather information for all of their grantees on one call, these interviews will last approximately 90 minutes. Protocols for each year can be found in Attachment B1.

• 2. Annual semi-structured interviews with grantee staff. These 60-minute interviews will address descriptive data elements (e.g., grantee activities, implementation processes, contextual information); barriers and facilitators; perceptions on the ability to scale and spread of strategies implemented by programs; and how the program addresses health equity. Protocols for each year can be found in Attachment B2.

• 3. Tailored web-based data collection tools. Data will be collected from grantees using a 30-minute web-based data collection tool. Because the programs are so diverse and are implementing a wide range of program activities, the tool will be tailored to each of the 15 grantees. This tailoring approach ensures that the data collected is relevant to the grantees’ activities and minimizes burden to the grantees. To tailor the tool, the evaluator will use the most recent program documents, and work in partnership with HRSA, to determine the grantees activities. Once the activities have been determined for each grantee, the evaluator will select data elements from the question pool that are relevant to the grantees’ activities. Only the relevant questions will be included on the tailored web-based tool for each grantee. The question pool that will be used to develop the 15 tailored web-based tools can be found in Attachment B3.

• 4. Partnership survey. A 15-minute survey will be administered to each grantee and their partners to assess the overall quality and effectiveness of each partnership. The partnership survey questionnaire can be found in Attachment B4.

In order to minimize data collection burden for this evaluation, a large number of secondary data sources will be utilized to gather information on the implementation and outcomes of the MH Programs. The secondary data sources include:

• HRSA Program Documents: Grantee Applications, Grantee Reports (HRSA Performance Measures), Grantee Evaluation Reports, and Grantee Program Call Templates.

• AIM Data Center: Contains data from AIM subawardees on several baseline, outcome, process, and structure measures per each bundle type throughout the course of the program.

• AIM-CCI Data Portal: Contains data for both of the existing patient bundles to assess implications for maternal safety through the continuum of care in the outpatient and community settings, as well as the soon-to-be developed non-hospital focused Maternal Safety Bundles (MSBs).

• RMOMS External Evaluation Report: An external evaluation will be conducted to document the implementation of the RMOMS program, assess how many women and infants were served, and assess broader program impact. The MH evaluation will utilize findings from the RMOMS evaluation documented in the RMOMS final report.

• State MHI Strategic Plans: Each State MHI Maternal Health Task Force will develop a state level strategic plan to address maternal mortality with specific recommendations and plans.

• State policies related to maternal health: Supporting MHI plans to create a “policy database” on their website to identify and document the variety of policy interventions at the local, state, and national levels aimed at reducing MM and SMM.

• State-level maternal health data: State MHI grantees will produce annual data reports on state maternal health outcomes.

2. Purpose and Use of Information Collection

The purpose of this project is to gather credible evidence on the activities and impact of the HRSA MH Portfolio. This evidence can be used to assess how programs were designed and implemented and evaluate their associated outcomes. These findings will be used for future decision making on efforts to address maternal mortality and severe maternal morbidity.

The evaluation is guided by six overarching evaluation research questions that assess the effectiveness of grantees’ activities, barriers and facilitators to implementation, opportunities for scaling and spreading effective program interventions, the degree to which programs address the elements of the prevention framework, whether the programs address health equity, and the overall impact of the portfolio on maternal health outcomes. The evaluation research questions are the following:

1. What program components (strategies and activities) were effective in addressing relevant maternal health outcomes? Why or why not?

2. What are the barriers and facilitators of implementing the MH programs’ strategies?

3. What individual and/or collective strategies or activities of the MH programs are likely to succeed if elevated to the national level or replicated in different settings?

4. To what extent did the MH programs address factors that have the greatest impact on maternal mortality outcomes based on the prevention framework?

5. To what extent did the MH programs address health equity?

6. What is the overall impact of the MH portfolio?

This information is critical for HRSA to understand the impact of these investments on maternal mortality and severe maternal morbidity. The evaluation will also provide important insights into which components of these programs should be elevated and replicated to the national level. To that end, the data will provide actionable information for future HRSA investment strategies that address maternal outcomes. It will also be used by HRSA to assess program and evaluation progress across the portfolio, so that HRSA can make adjustments as needed throughout the project period.

3. Use of Improved Information Technology and Burden Reduction

Data will be collected using web-based software. Grantees will complete the web-based data collection tool annually. Grantees and their partners will be asked to complete a one-time web-based partnership survey. The use of web-based surveys reduces respondent burden in numerous ways. The web tool will allow for the evaluator to minimize the number of questions that the grantee is asked by tailoring the tool to show questions that pertain to their unique program. The web tool will also allow for skip patterns to be implemented so that grantees will avoid seeing questions that are not relevant to them based on their answers to the previous questions. Survey respondents may stop the survey if necessary and return to it. To reduce burden when a respondent stops taking the survey, the survey will be programed so that the respondent does not have to start the survey from the beginning when they resume. This also reduces burden by allowing them to divide the time needed to complete the survey into increments of their choosing.

4. Efforts to Identify Duplication and Use of Similar Information

The purpose of the interviews, web-based data collection tool, and partnership survey is to obtain information about MH grantee programs that is not available from any other source. The information collected will complement the secondary data sources (discussed in item 1 above), not duplicate or replace them. A thorough review of these existing secondary data sources has been conducted and determined that there is no duplication in data collection efforts between the data collection tools (Attachments B1-B4) and the secondary data sources.

5. Impact on Small Businesses or Other Small Entities

This study will have minimal impact on small organizations. Grantee organizations include state governments, universities, associations, and healthcare systems and grantees are expected to participate in the evaluation activities as part of their grant.

It is possible small businesses or organizations may be included in data collection if they are one of the grantee partners asked to complete the partnership survey. To limit burden, the partnership survey will only be administered once and takes 15 minutes to complete.

6. Consequences of Collecting the Information Less Frequently

The consequences of less frequent data collection would result in a lack of essential and timely information to inform HRSA activities. The study has been designed to minimize the frequency of data collection when possible while ensuring that sufficient data will be collected to allow for the evaluation of the MH portfolio. Interviews with grantee staff and HRSA Project Officers will occur annually. Grantees will complete the web-based data collection tool annually. The partnership survey will be administered once during the study period. The use of secondary data also reduces the frequency of primary data collection by reducing the overall amount of data that needs to be collected through primary data collection activities.

Collecting data less frequently would result in insufficient data to understand the progress of the programs that HRSA would like to use to make adjustments to the program during the grant periods, if necessary. It would also impede HRSA’s ability to assess the impact of the programs, such as gathering important annual data regarding the implementation of the components of these programs that should be elevated and replicated to the national level and to inform future investments in programs addressing maternal outcomes.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The request fully complies with the regulation.

8. Comments in Response to the Federal Register Notice/Outside Consultation

Section 8A:

A 60-day Federal Register Notice was published in the Federal Register on June 8, 2020, vol. 85, No. 110; pp. 35099-35100 (see Attachment A). There were no public comments.

Section 8B:

The study team consulted with Roy Ahn, MPH, ScD, global maternal health researcher and Vice President, Public Health Research at NORC at the University of Chicago. Contact information: (312) 759-4068; [email protected]. Additionally, the study team met with each MH grantee to assess their evaluation capabilities and early plans for program activities, data collection, and data storage, which informed the development of the evaluation design.

9. Explanation of any Payment/Gift to Respondents

Respondents will not receive any payments or gifts.

10. Assurance of Confidentiality Provided to Respondents

Data will be kept private to the extent allowed by law. Respondents will be informed that their information will be kept private to the extent permitted by law. They will be told that their names will not be used in reports. This information will be provided verbally to interview respondents and verbal consent will be requested. Partnership survey respondents will be provided a written statement at the beginning of the web survey with this information (Attachment B4). This study will be reviewed by the Institutional Review Board of the contractor selected to conduct this evaluation.

11. Justification for Sensitive Questions

The data collection instruments do not include any questions of a sensitive nature. No personally identifiable information will be collected from respondents. Individual-level data will be not obtained from the grantees.

12. Estimates of Annualized Hour and Cost Burden

Estimates of annualized burden are below. The total number of estimated respondents is 387 and the total number of burden hours is 165.5.

12A. Estimated Annualized Burden Hours

Type of Respondent	Form Name	No. of Respondents	No. Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours
Grantee Staff	Instrument 1: Interview guide for grantee staff	75	1	1	75
Federal Project Officers	Instrument 2: Interview guide for HRSA POs	7	1	1.5	10.5
Grantee Partners	Instrument 3: Partnership Survey	290	1	.25	72.5
Grantee Staff	Instrument 4: Web-based data collection tool	15	1	.50	7.5
Total		387			165.5

12B.

Hourly wage rates were determined using the Department of Labor website. For grantee interviews and partnership interview respondents, we use the mean hourly age of Medical and Health Services Managers.⁵ For the web-based data collection tool, we use the mean hourly wage of Social and Community Service Managers.⁶

For federal project officers, we used the hourly rate for a GS-13 employee in the Washington, DC, area.⁷

Estimated Annualized Burden Costs

Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Grantee Staff	75	$54.68	$4,101.00
Federal Project Officers	10.5	$49.19	$516.50
Grantee Partners	72.5	$49.19	$3,566.28
Grantee Staff	7.5	$34.46	$258.45
Total			$8,442.23

13. Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

There are no capital and start-up costs to respondents associated with this data collection.

14. Annualized Cost to Federal Government

The cost to the Federal Government for this 5-year project is $4,876,103, or $975,221 per year on average. These costs cover all aspects of database development, testing, data collection, analysis, results interpretation, as well as providing technical assistance to all grantees. The method used to estimate the cost includes preparation of a detailed line-item budget that specifies all staff/consultant rates and labor hours by task, along with operational and other direct costs (e.g., telephone calls, reproduction).

In addition, it is estimated that two full-time equivalent HRSA staff member (Grade 12, Step 10; and 04/SG 12) will spend 30% of her time to manage, provide subject matter expertise, and administer the project. Assuming an annual salary of $112,240 and $93,000 respectively, government personnel costs will be $61,572 annually.

Total costs to the Federal Government are, therefore, $975,221 annually.

Estimates of annualized cost to the Federal Government

Item	Grade/Salary	Percent Effort	Annualized Cost
HRSA/MCHB/DHSPS/Project Staff/SME/Oversight	GS-12-10 ($112,240)	30%	$33,672
HRSA/MCHB/DHSPS/Project Staff/COR/Oversight	O4/SG-12 ($93,000)	30%	$27,900
Contract cost			$913,649
Total			$975,221

15. Explanation for Program Changes or Adjustments

This is a new information collection.

16. Plans for Tabulation, Publication, and Project Time Schedule

Plans for Tabulation: Analyses will be conducted at the activity- and strategy-levels in order to draw conclusions regarding implementation and outcomes of activities and strategies. Analysis will also be conducted at the grantee-level and the portfolio-level in order to develop findings across the various strategies and grantees. Descriptive statistics will be used to analyze quantitative data and content analysis will be used to analyze qualitative data. A mixed-methods analysis will incorporate the quantitative and qualitative data. A gap analysis, contribution analysis, trend analysis, and social network analysis are also planned.

Reports and Publication: Annual reports will be produced that will describe data collection methods, preliminary results for each research question, and any limitations to results. A final report will be produced that will discuss final results from data collection as well as the results of the gap analysis, contribution analysis, trend analysis, and social network analysis. The evaluator will also submit abstracts for presentation at national conferences of prominent associations such as the American Public Health Association, American Evaluation Association, American College of Obstetrics and Gynecology, and Association of Maternal and Child Health Programs. Additionally, the evaluator can develop special topic papers for state and national policy makers, practitioners, and the general public.

Project Timeline: The majority of data collection and reporting is designed to occur on annual cycles with some data elements expected to be collected towards the beginning and end of program implementation. Since the evaluation will begin 1-2 years after the start of implementation, data collection is scheduled to occur very soon after the start of the evaluation period of performance (2021). The evaluation is proposed to continue one year after all grantees’ period of performance in order to account for time for final data to be submitted by grantees, reviewed and analyzed by the contracted evaluator. The table below outlines the anticipated evaluation timeline, including data collection and reporting, for each year of the proposed evaluation.

Task	Timing
Collect and analyze HRSA Program Documents (grantee applications, reports, program call templates)	January 2021- August 2024
Collect and analyze additional secondary data sources (AIM Data Center, AIM-CCI Portal, RMOMS report, State MHI plans and other polices and data)	January 2021– November 2021 September 2021 – November 2021 September 2022 – November 2022 September 2023 – November 2023
Interviews with program staff and HRSA Project Officers	January 2021-May 2021 October 2021-May 2021 October 2022-May 2022 October 2023-May 2023
Web-based Data Collection Tool	April 2021-May 2021 April 2022-May 2022 April 2023-May 2023 April 2024-May 2024
Partnership Survey	November 2023-February 2024
Annual Reports	September 2020- August 2024
Final Report	November 2024-August 2025
Conferences and Special Topics Papers	November 2024-August 2025

17. Reason(s) Display of OMB Expiration Date is Inappropriate

The OMB number and Expiration date will be displayed on every page of every form/instrument.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

Attachments

Federal Registrar Notice

Attachment A: A 60-day Federal Register Notice was published in the Federal Register on June 8, 2020, vol. 85, No. 110; pp. 35099-35100

Data Collection Instruments

Attachment B1: Interview Protocol for HRSA Project Officers

Attachment B2: Interview Protocol for MH Portfolio Grantees

Attachment B3: Web-based data collection tool

Attachment B4: Partnership Survey

Recruitment Materials

Attachment C1: Project Officer Interview Recruitment Letter

Attachment C2: Grantee Interview Recruitment Letter

Attachment C3: Grantee Web-based Data Collection Recruitment Letter

Attachment C4: Partnership Survey Recruitment Letter

1 Petersen EE, Davis NL, Goodman D, Cox S, Mayes N, Johnston E, et al. Vital signs: Pregnancy-related deaths, United States, 2011-2015, and strategies for prevention, 13 States, 2013-2017. MMWR Morb Mortal Wkly Rep. 2019 May 10;68(18):423-9.

2 https://www.cdc.gov/reproductivehealth/maternalinfanthealth/pregnancy-mortality-surveillancesystem.htm

3 Howell, E. A., Egorova, N., Balbierz, A., Zeitlin, J., & Hebert, P. L. (2016). Black-white differences in severe maternal morbidity and site of care. American journal of obstetrics and gynecology, 214(1), 122-e1.

4 Hung P., Henning-Smith C.E., Casey M.M., & Kozhimannil K.B. (2017) Access to obstetric services in rural counties still declining, with 9 percent losing services, 2004-14. Health Affairs, 36(9), 1663-1671.

5 Occupational Employment and Wages, May 2018, 11-9111 Medical and Health Services Managers. Bureau of Labor Statistics. https://www.bls.gov/oes/current/oes119111.htm.

6 Occupational Employment and Wages, May 2018, 11-9151 Social and Community Service Managers. Bureau of Labor Statistics. https://www.bls.gov/oes/2018/may/oes119151.htm.

7 Salary table 2020-DCB. U.S. Office of Personnel Management (OPM). https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB.pdf.

7