FINALSupporting Statement A_0915-0338_Revision_11-2-16

Supporting Statement A

Healthy Start Evaluation and Quality Improvement

OMB Control No. 0915-0338

Highlighted text is the information changed from the last request.

Terms of Clearance: None

A. Justification

1. Circumstances making the collection of information necessary

This statement from the Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) requests Office of Management and Budget (OMB) approval for data collection to support monitoring and evaluation of the transformed Healthy Start (HS) program. The information collection is a revision to OMB# 0915–0338 and is authorized under the Healthy Start Reauthorization Act 2007 (Public Health Law No. 110-339), which includes appropriations for the Healthy Start initiative and its evaluation through fiscal year 2013 (Attachment A). Please note; in May 2016 a bill to amend the Public Health Service Act to reauthorize the Healthy Start program (Healthy Start Reauthorization Act of 2016) for FY2017-FY2022 was introduced.

The revision includes a modification of the Preconception, Pregnancy and Parenting (3Ps) Information Form, which has been redesigned from one form into six forms. The six forms include: (1) Demographic Intake Form; (2) Pregnancy Status/History; (3) Preconception; (4) Prenatal; (5) Postpartum; and (6) Interconception/Parenting (Attachment B1-B6). The purpose of this redesign is to enhance the 3Ps Information Form to ensure collected data is meaningful for monitoring and evaluation, as well as screening and care coordination, and to streamline previously separate data systems. The 3Ps Information Form was also redesigned to allow questions to be administered in accordance with the participant’s enrollment/service delivery status and perinatal period. In addition to redesigning the 3Ps Information Form, HRSA/MCHB deleted questions that are neither critical for evaluation nor programmatic purposes. HRSA/MCHB also added questions to the 3Ps Information Form to allow the Form to be used as an all-inclusive data collection instrument for MCHB and Healthy Start grantees. The additional questions extend and refine previously approved content, allowing for the collection of more granular and/or in-depth information on existing topics. Adding these questions allows Healthy Start grantees to better assess risk, identify needed services, provide appropriate follow-up activities to program participants, and improve overall service delivery and quality. HRSA/MCHB submitted a Change Memo request (Attachment C1-C3) for the modification to the 3Ps Information Form in May 2016. However, we were informed by the OMB Desk Officer to submit this request through a full clearance because of the extensive changes to the 3Ps Information Form. The remaining data collection instruments – National Healthy Start Program Survey; Community Action Network Survey; Healthy Start Site Visit Protocol; Healthy Start Participant Focus Group Protocol – have not been modified.

The purpose of the monitoring and evaluation is to assess the implementation of the program; measure the effect of the program on individual-, organizational-, and community-level outcomes; and identify best and promising practices for the program. Results from monitoring and evaluation efforts will provide actionable evidence to support the improvement, sustainability, replication, and dissemination of the program. In addition, monitoring and evaluation of the transformed Healthy Start program is consistent with the needs of HRSA/MCHB to meet its Government Performance and Results Act requirements.

The data collection effort to support monitoring and evaluation is of interest to HRSA/MCHB as the federal agency for promoting and improving the health of women and children. HRSA/MCHB will use the results of the monitoring and evaluation to improve interventions for reproductive age women, their children, and families, and help reduce health disparities, decrease infant mortality, and improve perinatal health outcomes.

Background of Healthy Start

The national Healthy Start program aims to reduce disparities in infant mortality and adverse perinatal outcomes. The program began as a demonstration project with 15 grantees in 1991 and expanded over the past two decades to 100 grantees across 37 states and Washington, DC. Today, Healthy Start has evolved from a program framework of nine service and systems core components to five approaches: (1) improving women’s health, (2) promoting quality services, (3) strengthening family resilience, (4) achieving collective impact, and (5) increasing accountability through quality assessment, performance monitoring, and evaluation.¹

Need for monitoring and evaluation of the Transformed Healthy Start program

The transformation of Healthy Start, based on recent science, innovations, and legislation, will necessitate revised methods for monitoring and a reassessment of the program. Information from a strong monitoring and evaluation effort will contribute to the program’s continued evolution and transformation by shaping key programmatic decisions, identifying successful implementation strategies, and strengthening the evidence base for the program model. In addition, results from monitoring and evaluation can be used to meet Government Performance and Results Act requirements.

Healthy Start benefits from more than two decades of experience that included an evaluation of its demonstration and two previous national evaluations. This experience has influenced the design and priorities for the monitoring and evaluation of the transformed program.² Specifically, previous monitoring and evaluations revealed important information about Healthy Start’s implementation and contribution to improvements in birth outcomes. However, the monitoring and evaluations were fundamentally limited by a lack of consistently collected and high quality data on outcomes to assess the association between program components and outcomes. Although grantees collected administrative data on all of their clients at the individual level, their data collection was not standardized and only reported in the aggregate to MCHB. In addition, the lack of a comparison group has made it challenging to develop an assessment of the program’s effect.

Considering the lessons learned from the previous funding cycles of the Healthy Start program and its evaluations, HRSA/MCHB seeks to conduct uniform individual-level data collection across grantees for programmatic monitoring purposes and a mixed-methods evaluation of the transformed Healthy Start program that includes the following three design components: 1) implementation; 2) health services utilization; and 3) outcome evaluations. The purpose of the implementation evaluation is to describe HS programs and strategies and to identify program factors that are associated with effective implementation. The purpose of the health services utilization evaluation is to examine the characteristics of participants and non-participants and factors that help explain differential penetration, or service rates. The purpose of the outcome evaluation is to assess the overall effectiveness of the program with regard to producing expected outcomes among the target population and factors that help explain variation in the program’s impact on individual level outcomes. The outcome evaluation will employ a quasi-experimental method, which will include two types of comparisons:

1. A matched individual comparison analysis of linked vital records for HS participants and non-participants in the same general geographic service area for all 100 HS grantees, which maximizes generalizability and will allow for assessment of the key outcome of interest, infant mortality, with adequate statistical power.

2. A matched individual comparison analysis of HS participants and non-participants by oversampling of the Center of Disease Control and Prevention (CDC) Pregnancy Risk Assessment and Monitoring Survey (PRAMS) for a random sample of 15 HS grantees. This component of the evaluation data collection strategy will maximize internal validity with a broader set of outcomes and control for matching characteristics that can influence selection into the program.

Underlying the monitoring and evaluation of Healthy Start is the program logic model (Figure A.1). This framework was used to identify the data elements for collection related to program implementation, outputs, and outcomes. The longer-term outcomes—such as improved birth outcomes and decreased maternal and infant morbidity and mortality—are unlikely to be observed during the five-year study period. However, the logic model identifies the short- and intermediate outcomes that are known to be associated with the longer-term outcomes. The individual-, organizational-, and community-level outcomes prioritized for study in the evaluation are specified in Attachment D.

Figure A.1. Transformed Healthy Start program logic model

Data collection activities under the monitoring and evaluation of the Transformed Healthy Start program

To support uniform data collection and the mixed-methods evaluation design, five types of data collection activities will be implemented: the redesigned 3P’s Information Form, National Healthy Start Program Survey, CAN Survey, Healthy Start Site Visits, and Healthy Start Participant Focus Groups (Figure A.2). Below, a description of each of these data collection activities is provided.

In addition to these data collection methods and instead of using 15 matched comparison sites, the evaluation will build linkages to existing datasets such as vital records (e.g., birth and death certificates) and PRAMS to compare Healthy Start participants and non-participants using key benchmarks and outcomes available in vital records, such as infant mortality, low birth weight, preterm birth, and breastfeeding initiation, as well as additional pre- and post-partum benchmarks available in PRAMS, such as perinatal depression screening, the postpartum visit, and safe sleep practices. The evaluation is designed to link Healthy Start participant data to vital records for all 100 Healthy Start grantees and to PRAMS data for 15 randomly selected Healthy Start grantees. All Healthy Start grantees will be asked to provide individual identifiers (e.g., mother’s name, mother’s date of birth, infant sex, date of delivery, delivery hospital) for Healthy Start participants that give birth in calendar year 2017 to state/jurisdiction Vital Records Offices. The Vital Records Offices will link the Healthy Start participants to 2017 infant birth certificates and any subsequent infant death certificates. For the 15 Healthy Start grantees selected for PRAMS oversampling, all Healthy Start participants that are linked to birth certificates—the PRAMS sampling frame—will be selected to receive a PRAMS survey.

Figure A.2. Summary of the Transformed Healthy Start Program Data Collection Activities

MONITORING:

All 100 Healthy Start Grantees

E

V

A

L

U

A

T

I

O

N

*We are not seeking approval of Vital Records Linkage and PRAMS Oversampling in this ICR package.

• The Redesigned Preconception, Pregnancy and Parenting (3P’s) Information Form (Attachment B1 – B6) will collect uniform information at the individual level about women eligible for Healthy Start and their children (up to 2 years of age) and families for monitoring and evaluation purposes. These data have traditionally been collected by Healthy Start grantees at the individual level within their own administrative data systems; however, they have not been collected in a standardized format and have only been reported to MCHB in the aggregate. Under this grant cycle, MCHB is making an effort to improve the quality of information already being collected by grantees by supporting standardization of key program data elements, which will also support assessments of program performance and evaluation. The data elements on the redesigned 3P’s Information Form are limited to those considered necessary to describe the reach of the program and the services provided, and to develop measures as specified in the funding opportunity announcement (file:///C:/Users/jbanks/Downloads/oppHRSA-14-020-cfda93.926-cidHRSA-14-020-instructions.pdf). The women eligible for Healthy Start include those of reproductive age (ages 15–44 years) living in communities with the poorest perinatal outcomes in the nation, including low birth weight and infant mortality.

• The National Healthy Start Program Survey (NHSPS) (Attachment E) will collect information about implementation of the program across the five key approaches of Healthy Start. All 100 Healthy Start projects will be asked to complete this survey. Project directors may delegate completion of sections of the survey to other Healthy Start staff. The survey is designed for self-administration through a web-based application that will allow the respondent to stop in the middle and resume the survey at another time. Healthy Start projects will be asked to complete the survey two times—in the second and fourth grant years.

• The Community Action Network Survey (Attachment F) will collect information about the health networks and social networks that support maternal and child health and social capital within the community.³ Approximately 10 to 15 active CAN board members and committee chairs for 15 Healthy Start projects that are selected for PRAMS oversampling will be asked to complete the survey. CAN members include representatives of organizations and agencies in the community that range from state and local government to community-based organizations. The CAN may also include individual consumers and community leaders. However, the instrument is designed to assess the relationships between agencies and organizations in the community to address maternal and child health, and as a result, individuals without relevant organizational/agency ties will not be included among respondents. Consumer and community leader involvement will be captured through the NHSPS and site visits. The CAN survey is designed for self-administration through a web-based application. Active CAN members will be asked to complete the survey two times—during the third and fifth grant years. Methods for selecting the 15 Healthy Start sites for PRAMS oversampling are described in Supporting Statement B.

• Healthy Start Site Visits (Attachment G) will collect in-depth qualitative information about program implementation and achievements. The information collected will also give context to quantitative outcomes and help identify best practices. Site visits will be conducted at the 15 Healthy Start projects selected for PRAMS oversampling. During the site visits, in-person interviews will be conducted with four types of informants: Healthy Start project directors and administrative staff, Healthy Start core service staff, health care providers, and Healthy Start CAN members. Site visits will be conducted once during the fifth grant year. Methods for selecting Healthy Start sites for PRAMS oversampling are described in Supporting Statement B.

• Healthy Start Focus Groups (Attachment H) will collect participants’ perspectives on program implementation, individual-level networks, and social capital within the community. The focus groups will be conducted in the communities of the 15 Healthy Start projects selected for PRAMS oversampling. Similar to the site visits, they will provide context to quantitative outcomes. Each focus group will include 10 to 12 Healthy Start participants. The focus groups will be conducted in English and Spanish at a minimum; other languages will be determined by the populations served. These focus groups will occur during the fifth grant year at the same time as the site visits.

Information collected through these activities will be used together to monitor implementation and evaluate the effectiveness of the Healthy Start program in improving perinatal health among disadvantaged populations. The mixed-modes data collection approach will capture both quantitative measures of program activities, outputs, and outcomes as well as qualitative impressions of program implementation and lessons learned. This data collection approach will generate results useful to policymakers and practitioners, informing them about the implementation and value of Healthy Start as an intervention working at multiple levels to reduce infant mortality.

2. Purpose and use of information collection

The purposes of the monitoring and evaluation are aligned with Healthy Start program needs and goals for accountability, programmatic decision making, and ongoing quality assessment at the grantee and national levels. The monitoring and evaluation of the transformed Healthy Start Program are focused around the following goals:

• Provide information to assess implementation of the program and enable identification of issues at earliest possible stages for midcourse corrections among individual grantees and for the program as a whole

• Provide credible and rigorous evidence of program effect on outcomes at multiple levels and across the life course of participants

• Assess the relationship between program components and outcomes to identify the relative contribution of components to desired outcomes for programmatic decision making

• Identify best and promising practices in implementation for replication and dissemination of the program⁴

• Strengthen the evidence base for multipronged initiatives to improve maternal and child health

To reach these goals, the monitoring and evaluation will address seven key research questions. Aligned with the purposes stated in Section A.1, each question has an evaluation design component associated with it in parentheses.

1. What components (e.g., activities, services, interventions) did grantees implement in the transformed HS program and to what extent did the components align with the five HS approaches? (Implementation Evaluation)

2. What factors (e.g., program and organizational) help explain effective implementation of the transformed HS program? (Implementation Evaluation)

3. How many women and infants participated in the transformed HS program? (Health Services Utilization Evaluation)

4. To what extent were Healthy Start services delivered to the highest risk target populations (women and infants), as intended? (Health Services Utilization Evaluation)

5. What factors (e.g., personal, program, and organization level) help explain the volume of services used? (Health Services Utilization)

6. What impact did the transformed HS program have on HS participants when compared to non-HS controls? (Outcome Evaluation)

7. What factors (program/organizational) of the transformed HS program are associated with improved participant behaviors, health services utilization, and health outcomes? (Outcome Evaluation)

The strength of the data collected for the monitoring and evaluation will be critical in the development of credible results. Table A.1 summarizes each data collection method and the monitoring and evaluation components into which they will feed.

Table A.1. Data collection efforts and design component

Data Collection Effort	Program Monitoring	Implementation Evaluation	Health Services Utilization Evaluation	Outcome Evaluation
3P’s Information Form				
National Healthy Start Program Survey				
Community Action Network Survey		
Site Visits			
Focus Groups			

Below, we discuss the specific use of the information collected under each method.

• The Redesigned 3P’s Information Form (Attachment B1-B6) is designed to collect information about Healthy Start participants/clients across all Healthy Start grantee sites. The client data is the primary data source for the health services utilization and outcome evaluations. The client data provides data on individual-level socio-demographics, service needs, services received, and follow-up visits and enables HRSA/MCHB to understand the HS population and to track outcomes and progress at the participant level. The redesigned 3Ps Information Form is divided into six tools, including a Demographic Intake Form, and assessments of key reproductive phases, including Pregnancy Status/History, Preconception, Prenatal, Postpartum, and Interconception/Parenting. They also facilitate aggregate or crude benchmarking and comparison with national databases on various health behaviors, health services received, and perinatal outcomes.

• The National Healthy Start Program Survey (Attachment E) is designed to provide high quality information about the implementation of the Healthy Start program across its five key approaches. Accordingly, after the first section of the survey asking for general program information, the five subsequent sections correspond to each of the five approaches, and the final section ends with questions about program achievements. The sections of the survey are (1) overview of services, staffing, outreach, and retention; (2) improve women’s health; (3) promote quality services; (4) strengthen family resilience; (5) achieve collective impact; (6) increase accountability through quality, performance monitoring, and evaluation; and (7) Healthy Start project achievements. These data will be used for conducting the implementation and health services utilization studies to provide variables related to program components and intervention models that may explain outcomes. For the implementation study, the data will provide information about the nature and extent of Healthy Start projects’ collaboration and linkages in the community. For the health services utilization study, the information will be used to assess services offered and provided, intervention models used by projects, aggregated outcomes for the population served, and achievements at the grantee and national levels.

• The Community Action Network (CAN) Survey (Attachment F) is designed to collect information about implementation of the Healthy Start program as related to the health and social networks to support maternal and child health, and social capital within the community. The sections of the survey are (1) organizational information, (2) CAN participation, (3) infrastructure for collaboration, (4) quality of collaboration, (5) progress toward achieving goals, and (6) perspectives on the community. Information from the CAN survey will be used mainly in the implementation study to provide variables to describe aspects of program implementation as related to partnerships and resources in the community.

• Healthy Start Site Visits (Attachment G) will include key informant interviews that will cover several aspects of program activities, including staffing, services provided, populations served, partnerships, networks, and reflections on challenges and successes. Qualitative information from the site visits will be used mainly to assess program implementation and identify and describe best and promising practices. By providing information about the nuances of program implementation, it may provide context to quantitative outcomes.

• Healthy Start Focus Groups (Attachment H) are designed to capture the participants’ perspectives on program implementation, individual-level networks, and social capital within the community. The focus group protocols include the following areas for discussion: outreach and participation, services received, case management and service coordination, home visiting, counseling and support, health education/promotion, medical home, and perspectives about the community. Similar to the site visits, the focus groups will provide information about program implementation and utilization, specifically about outreach, populations served, and services provided, but from the participant perspective. It may also provide context to interpret quantitative outcomes.

3. Use of improved information technology and burden reduction

Redesigned 3P’s Information Form, National Healthy Start Program Survey, and Community Action Network Survey. These three data collection efforts will comply with the Government Paperwork Elimination Act (Public Law 105-277, Title XVII) by employing technology efficiently in an effort to reduce burden on respondents. HRSA/MCHB will use an online, web-based application to obtain information from respondents for all three instruments. The application will include automated range checks and branching and will enforce consistency among critical questions to optimize resources and facilitate collection of high quality data. The programming will allow the collection of information specific to each respondent by skipping respondents out of questions not pertinent to them, thereby eliminating undue time burden on respondents. The application will also allow respondents to stop and return to the instrument so that they can complete it at their convenience. The instruments solicit only information that corresponds to the specific research items discussed in Section A.2, above. No superfluous or unnecessary information is being requested of respondents. In addition to the web-based application for the redesigned 3Ps Information Form, grantees requiring a paper form will also have the option to use PDF forms modeled after the redesigned 3Ps Information Form. Grantees using the PDF forms may save these forms locally and can complete them for clients on a laptop or other device in the field, saving them for upload later when an internet connection is available.

Healthy Start Site Visits and Focus Groups. As these are qualitative data collection efforts, HRSA will not use information technology to collect information from 90 staff and stakeholders during site visits (four key informant interviews at each of the selected 15 Healthy Start sites) and the approximately 180 Healthy Start participants during the focus groups (12 participants at each of the 15 selected sites). The data collection is qualitative in nature and requires information from a relatively small number of individuals; therefore, it is not appropriate, practical, nor cost-beneficial to build electronic instruments to collect the information. All information will be collected orally in person using discussion guides, supported by digital recordings. Site visit and focus group transcripts will be analyzed using Atlas.ti, a software system used for the qualitative analysis of large amounts of data collected in text format.

4. Efforts to identify duplication and use of similar information

There are no current HRSA/MCHB data collection activities for monitoring and evaluating the transformed Healthy Start Program. The information that we are requesting to collect described in this OMB package is not available elsewhere.

5. Impact on small businesses or other small entities

This activity does not impact small entities.

6. Consequences of collecting the information less frequently

Table A.2 summarizes the data collection efforts, including the frequency of information collection. After the table, we discuss the consequences of collecting the information less frequently for each data collection activity.

Table A.2. Summary of data collection efforts

Data Collection Method	Data Collected	Respondents	Administration	Rounds of Data Collection	Consequences of less frequent data collection
3P’s Information Form	Individual-level data	40,675 Healthy Start participants across 100 Healthy Start projects	Web-based or PDF form, administered by Healthy Start staff to participants	Ongoing for Healthy Start participants	Limit the ability to assess outcomes and overall program and individual grantee performance Limit the ability to assess changes as the program matures
National Healthy Start Program Survey	Program implementation and aggregate outcomes data	All Healthy Start projects each round (100)	Web-based survey, self-administered by Healthy Start project director and staff	2 rounds during second and fourth grant years	Limit the ability to link changes in outcomes to the implementation of program components and identify the best and promising practices associated with better outcomes.
Community Action Network (CAN) Survey	Organizational-level data	~225 CAN members across 15 selected Healthy Start projects	Web-based survey, self-administered by CAN members	2 rounds during third and fifth grant years	Limit the ability to assess changes in community-level and systems outcomes of the program and link them to changes in individual-level outcomes.
Site Visits	Qualitative program implementation information	~90 key informants across 15 selected Healthy Start projects	Interviews with Healthy Start project director, core service staff, providers, and CAN members	1 round during the fifth grant year
Focus Groups	Participants’ perspectives of implementation	~180 participants across 15 selected Healthy Start projects	Group discussions led by national evaluation staff	1 round during the fifth grant year

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

This request fully complies with 5 CFR 1320.5. There are no special circumstances.

8. Comments in response to the federal register notice/outside consultation

Section 8A:

The 60-day Federal Register Notice required in 5 CFR 1320.8(d) was published in the Federal Register on June 24, 2016, Volume 81, Number 122, Page Numbers 41314-41315. Public comments were received (Attachment I1). The public comments generally focused on the number and length of the instruments for the redesigned 3Ps Information Form, and the time burden to administer the 3Ps Information Form. Public comments also included recommendations for removing, revising, and adding questions to the 3Ps Information Form. There were also comments regarding the sensitivity of questions, especially questions that were assessing medical/clinical conditions.

To address the public comments the Healthy Start program contractor and Healthy Start Collaborative Improvement and Innovation Network (CoIIN), in collaboration with HRSA/MCHB, reviewed the 3Ps Information Form and determined which questions to remove and/or revise to streamline the instruments and reduce burden on Healthy Start participants and grantee staff. The revisions resulted in a substantial reduction of burden for the redesigned 3Ps Information Form. Please see Attachment I2 for a summary matrix of public comments and how they are addressed/resolved.

Section 8B:

In an effort to consult with experts both inside and outside the Department of Health and Human Services, HRSA/MCHB presented a description of the planned evaluation of the transformed Healthy Start program to HS grantees, HS CoIIN, CDC PRAMS programs, Vital Records Offices in states that have HS grantee sites, and to HRSA/MCHB staff and leadership. In addition, HRSA/MCHB staff and the HS CoIIN reviewed and provided feedback on the instruments during a number of meetings and conference calls.

HRSA/MCHB pilot tested the redesigned 3Ps Information Form with nine participants at one grantee site in 7/11/2016 – 7/15/2016. A training webinar was provided to the grantee prior to the pilot test period. The training provided background on and an overview of the forms, and guidance for the grantee in administering the redesigned 3Ps Information Form and completion of a pilot program evaluation form. The redesigned 3Ps Information Form pilot test was conducted using the electronic version of the forms. Because there are no revisions to the NHSPS and the CAN survey, a pilot test of these instruments was not conducted again. The initial pilot test of the NHSPS and the CAN survey occurred during the period of 1/28/2014 – 2/14/2014. The pilot test for the NHSPS and the CAN survey were conducted using a paper version.

The results of the pilot tests and recommendations for finalizing the instruments are presented in Attachments J1 (redesigned 3Ps Information Form) and J2 (NHSPS and CAN survey). The pilot tests allowed us to validate the length of the instruments (which was determined to be shorter than the initial estimates) and, thus, reduce the public burden. The pilot test also allowed us to refine and clarify the instructions and language; responses collected during the pre-test were not and will not be analyzed.

The instruments were revised based on results of the pilot test and feedback from HRSA/MCHB staff and the public. Contact information for the one grantee that participated in the pilot for the redesigned 3Ps Information Form is provided in Table A.3.

Table A.3. Pre-test grantee contact information

NAME	CONTACT INFORMATION
Raymond Howard	[email protected] 210-207-4725

9. Explanation of any payment/gift to respondents

Redesigned 3P’s Information Form. Healthy Start participants will not be compensated for completing the 3P’s Information Form, as information will be collected as part of the enrollment and participation process and will be essential for providing, targeting, and improving services for these women.

Healthy Start Focus Groups. To encourage attendance, focus group participants will be provided an incentive of a $25 gift card when they attend the focus group. In addition to each focus group taking 90 minutes, participants must travel and potentially hire somebody for child care.

National Healthy Start Survey, Community Action Network Survey, and Healthy Start Site Visits. Healthy Start staff and stakeholders will not receive incentive payments because most are participating as part of their professional positions.

10. Assurance of confidentiality provided to respondents

Redesigned 3P’s Information Form. HRSA/MCHB has embedded protections for privacy in the study design. The information collection will fully comply with all aspects of the Privacy Act. Individuals and agencies will be assured of the privacy of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). All participants will be told during the consent process that the data they provide will be treated in a secure manner to the extent allowed by law.⁵ They also will be informed that participation is voluntary, that they may refuse to answer any question, and that they can stop at any time without risk to their receipt of services outside of Healthy Start. In addition, their name will not be provided to the federal government. A unique ID will be assigned to each participating woman.

At Healthy Start sites, individuals must have a form record in the data collection system to be considered a program participant. The form can be incomplete if the individual refuses to provide all information requested. However, without a form in the data collection system, grantees will not have a means for accounting for individuals recruited (and Healthy Start services provided) to HRSA/MCHB. Organizations will ask program/study participants to sign an informed consent (please see Attachment K for the Healthy Start evaluation IRB package) form to authorize participation in the program/study. A staff member will read the consent elements and record the participant’s response in the data collection system. HRSA/MCHB will work with grantees to customize consent procedures so they are acceptable to HRSA/MCHB, and the Institutional Review Board (IRB).

National Healthy Start Program Survey, Community Action Network Survey, Healthy Start Site Visits. As part of establishing communication for the remaining data collection efforts, potential participants will be sent information about the study and what is required for participation. The elements of consent will be explained in these communications. In addition, we will develop consent forms and procedures for participants to sign at the time of data collection. We will develop consent forms as part of the web-based instrument for the NHSPS and CAN surveys that will request electronic signatures from respondents and paper-based consent forms for key informants to sign during site visits. No personally identifiable data will be collected from these data collection methods. The requested information is at the aggregate or organizational level.

Healthy Start Focus Groups. When Healthy Start participants arrive at the focus group location, they will be given a consent form to read, sign, and return to the moderator. The focus group moderator will answer any questions posed by the participants about consent or privacy.

In addition to specific security procedures for the various data collection activities, two approaches cut across the entire study. First, all contractor employees will sign a pledge that data will be kept private to the extent allowed by law and respondent identity, and breaking that pledge is grounds for immediate dismissal and possible legal action. Second, HRSA/MCHB sought IRB clearance (Attachment K) through the CDC’s National Center for Health Statistics and is waiting for approval.

11. Justification for sensitive questions

Redesigned 3P’s Information Form. The 3P’s Form is designed to provide data on individual-level socio-demographics, service needs, services received, and follow-up visits and enables HRSA/MCHB to understand the HS population and to track outcomes and progress at the participant level. The form will help HRSA/MCHB assess how participation in Healthy Start may be associated with positive perinatal outcomes and reduce disparities in perinatal outcomes. The form also facilitates aggregate or crude benchmarking and comparison with national databases on various health behaviors, health services received, and perinatal outcomes. A number of items in the form refer to personal behaviors and circumstances that may be of a sensitive nature for respondents. Examples of potentially sensitive health behavior questions include those related to smoking, alcohol, and drug use; screening for HIV/AIDS and sexually transmitted infections; breastfeeding; use of family planning methods; pregnancy loss or infant death; and race/ethnicity. However, it is necessary to collect information from women on these topics because research has linked such behaviors to birth outcomes, and Healthy Start provides services to promote relevant healthy behaviors, links participants to needed services, and aims to reduce disparities in outcomes. HRSA/MCHB has minimized the number of sensitive questions to those necessary for the purposes of monitoring and evaluation; the form includes questions that are directly relevant to assess outcomes and progress toward goals of the program. Finally, women will be assured that they do not have to respond to any questions that they do not want to answer.

Healthy Start Focus Groups. Similar to the redesigned 3P’s Information Form, the Healthy Start focus groups will ask participating women to discuss their experiences with the Healthy Start program. Topics that may come up during the focus groups include potentially sensitive ones, such as smoking, alcohol, and drug use; breastfeeding; use of family planning methods; pregnancy experiences, and family support. Qualitative information collected from women on these topics is important to understanding the contribution of Healthy Start and to provide context to outcomes. Training of the focus group moderators will emphasize the importance of discussing topics that involve sensitive issues in a professional and nonjudgmental manner and facilitating a supportive environment to promote constructive conversation and sharing. Finally, women will be assured that they do not have to talk about any topics that they are not comfortable discussing.

National Healthy Start Survey, Community Action Network Survey, Healthy Start Site Visits. There are no questions of a sensitive nature for these instruments.

12. Estimates of annualized hour and cost burden

Section 12A:

The Healthy Start data collection will include instruments, in both electronic and paper-based formats, and a focus group with Healthy Start participants; a web-based program survey and site visit protocol for Healthy Start project directors and staff; and a web-based survey for Community Action Network (CAN) board members. The annualized frequency of the data collection will include one response per respondent for each data collection form. The estimated annualize burden hours is 92,156. Table A.4 presents the annual burden hour estimates for this data collection.

Table A.4. Estimates of annualized hours

Type of Respondent	Form Name	No. of Respondents	No. of Responses per Respondent	Total Responses	Average Burden per Response (in hours)	Total Burden Hours
Healthy Start Participants	3Ps Information Form: 1. Demographic Intake Form	40,675*+	1	40,675	0.08	3,254
Healthy Start Participants	2. Pregnancy History/Status	40,675	1	40675	0.17	6,915
Healthy Start Participants	3. Preconception	20,337*+	1	20337	1	20,337
Healthy Start Participants	4. Prenatal	20,337	1	20337	1	20,337
Healthy Start Participants	5. Postpartum	20,337	1	20337	1	20,337
Healthy Start Participants	6. Interconception/ Parenting	20,337	1	20337	1	20,337
Healthy Start Project Directors/Staff	National Healthy Start Program Web Survey	100+	1	100	2	200
Community Action Network (CAN) Board Members	CAN member Web Survey	225+	1	225	0.75	169
Healthy Start Project Directors/Staff	Healthy Start Site Visit Protocol	15+	1	15	6	90
Healthy Start Participants	Healthy Start Participant Focus Group Protocol	180+	1	180	1	180
Total		61,532		61,532		92,156

*The same individuals (40,675) complete the Demographic Intake and Pregnancy Status/History forms, and a subset of these same individuals (20,337) also complete the Preconception, Prenatal, Postpartum, and Interconception/Parenting forms for total of 61,532 respondents and responses.

⁺ These are the numbers included in the total respondent count.

Section 12B:

For each data collection effort, we use dollar per hour estimates to generate the estimated annualized burden costs. We used the median wage ($17.40) for Healthy Start participants hourly rate estimated by the Department of Labor, Bureau of Labor Statistics (BLS), Occupational Employment Statistics for all occupations in 2015 (http://www.bls.gov/oes/current/oes_nat.htm). In addition, for the Healthy Start site staff that will be involved in recruiting respondents and administering the instruments, the annualized hour and cost burden is estimated at $22.21 per hour, based on BLS’s median hourly wage for Counselors, Social Workers and Other Community and Social Service Specialist. Healthy Start project directors annualized hour and cost burden was estimated to be $46.99, based on BLS’s median hourly wage for General and Operations manager positions. The Community Action Network (CAN) Board Members annualized hour cost burden was estimated to be $33.38, based on BLS’s median hourly wage for Social and Community Service manager positions. The total hour cost was calculated by multiplying the total burden hours by the hourly wage rate.

Table A.5. Estimates of annualized burden costs

Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Cost
Healthy Start Participants	91,697	$17.40	$1,595,528
Healthy Start Project Directors/Staff	290	$22.21 - $46.99	$6,441 - $13,627*
Community Action Network (CAN) Board Members	169	$33.38	$5,641
Total			$1,607,610 - $1,614,796

*HS program participants - $17.40; Healthy Start Outreach Staff - $22.21; Healthy Start Project Director - $46.99. The range in hourly wage rates for the 3P’s Form and Healthy Start Participant Focus Group protocol is reflective of the various staff and respondents involved in administering/completing the survey or conduct/attend the focus group.

13. Estimates of other total annual cost burden to respondents or record keepers/capital costs

There are no capital and start-up cost to respondents associated with this data collection.

14. Annualized cost to federal government

The approximate annualized cost to the government for this data collection effort is $376,663. These costs are comprised of: federal employee salaries, contractor staff salaries, and operational expenses (e.g., equipment, printing, and postage). Table A.6 below provides the cost breakdown for the annualized cost to the federal government.

Table A.6. Estimates of annualized cost to the Federal Government

Item	Grade/Salary	Percent Effort	Annualized Cost
HRSA/MCHB Healthy Start Project Staff/Oversight	GS-14-5 ($123,406)	50%	$61,703
HRSA/MCHB Evaluation Officer	GS-14-5 ($123,406)	25%	$30,852
Contractor Staff* (Project Director)	$105,060	28%	$29,417
Contractor Staff (Survey Lead/Statistician)	$157,068	22%	$34,555
Contractor Staff (Research Analyst)	$108,118	27%	$29,192
Contractor Staff** (Database Development & Maintenance)
Project Manager	$196,707	5%	$9,835
Architect	$172,900	2%	$3,458
Junior Analyst	$86,450	21%	$18,155
Senior Analyst	$94,430	2%	$1,889
Lead Software (SW) Engineer	$294,424	20%	$58,885
Mid SW Engineer	$248,539	18%	$44,737
Junior SW Engineer	$167,315	15%	$25,097
Configuration Management Analyst	$248,539	3%	$7,456
SW Tester	$204,318	10%	$20,432
Operational Costs for Data Collection Activities (e.g., printing, postage, equipment), non-labor			$1,000
Total			$376,663

*Contractor salaries are loaded and include fringe benefits (e.g., costs for health insurance, travel, paid vacation). The fringe rate is 38% for full-time staff.

**The fringe rate for contractors associated with Database Development & Maintenance is 33%.

15. Explanation for program changes or adjustments

The requested burden of 92,156 has increased from the currently approved burden of 20,953  because the number of data collection instruments increased from one instrument with the original 3Ps Information Form to six forms in the redesigned 3Ps Information Form (1. Demographic Intake; 2) Pregnancy Status/History; 3) Preconception; 4) Prenatal; 5) Postpartum; 6) Interconception/Parenting.  The number of respondents and burden is calculated for each form thus increasing the overall burden.  While this represents an increase in number of respondents and annualized hour burden, it reduces the burden to individual participants that will not need to respond to all the questions as with the original 3Ps Information Form.  Healthy Start participants are administered the forms based on their perinatal stage.  Four of the six reformatted forms are only for participants that become pregnant while in the Healthy Start program.

This is a revision to the Preconception, Pregnancy and Parenting (3Ps) Information Form, which has been redesigned from one form into six forms. The six forms include: (1) Demographic Intake Form; (2) Pregnancy Status/History; (3) Preconception; (4) Prenatal; (5) Postpartum; and (6) Interconception/Parenting. The purpose of this redesign is to enhance the 3Ps Information Form to ensure collected data is meaningful for monitoring and evaluation, as well as screening and care coordination, and to streamline previously separate data systems. The 3Ps Information Form was also redesigned to allow questions to be administered in accordance with the participant’s enrollment/service delivery status and perinatal period. In addition to redesigning the 3Ps Information Form, HRSA deleted questions that are neither critical for evaluation nor programmatic purposes. HRSA also added questions to the 3Ps Information Form to allow the Form to be used as an all-inclusive data collection instrument for MCHB and Healthy Start grantees. The additional questions extend and refine previously approved content, allowing for the collection of more granular and/or in-depth information on existing topics. Adding these questions allows Healthy Start grantees to better assess risk, identify needed services, provide appropriate follow-up activities to program participants, and improve overall service delivery and quality. The remaining data collection instruments - National Healthy Start Program Survey; Community Action Network Survey; Healthy Start Site Visit Protocol; Healthy Start Participant Focus Group Protocol – have not been modified.

16. Plans for tabulation, publication, and Project time schedule

Analysis plan

Although information from the various data collection efforts will be combined to answer the evaluation questions, the analyses of data will vary based on the specific questions. Analyses using the redesigned 3P’s Form, the NHSPS, CAN survey, Healthy Start Site Visits, and Healthy Start Focus Groups are described in Table A.7 by evaluation question. The outputs and outcomes assessed are those shown in the logic model (Figure A.1 in Section A.1, with prioritized outcomes shown in Attachment B).

Table A.7. Analytic approach and methods for each Healthy Start evaluation question

Evaluation Question		Data Source(s)	Analytic Approaches & Methods
1	What components (e.g., activities, services, interventions) did grantees implement in the transformed HS program and to what extent did the components align with the five HS approaches? (Implementation Evaluation)	NHSPS	Descriptive analysis will involve the development of metrics to evaluate implementation and performance, such as progress towards goals set by MCHB/HRSA, implementation of evidence-based models, and other MCHB/HRSA guidance. Implementation goals may include the number and types of people served and the types of services provided as outlined in the funding opportunity announcement.
2	What factors (e.g., program and organizational) help explain effective implementation of the transformed HS program? (Implementation Evaluation)	3P’s Information Form, NHSPS, and CAN Survey	The descriptive analysis will also test the statistical significance of bivariate associations between program and organization level factors and indicator(s) of effective implementation. Program-level factors may include the number of participants served during the preconception, pregnancy and postpartum periods; outreach strategies employed; number and types of referrals provided; case management models utilized; caseloads maintained; the number and types of screenings provided; promotion of male involvement, among others. Organization level factors will likely include the type of program (urban, rural, border); the HS program level (1, 2 or 3); the lead agency type; age of the program; staffing characteristics; and the type of approaches and services provided, among others.
3	How many women and infants participated in the transformed HS program? (Health Services Utilization Evaluation)	3P’s Information Form	Descriptive analyses will include a summary of HS participants in terms of a number of individual characteristics, including socio-demographic indicators (such as age, race/ethnicity, income, education, insurance type, geographic area), health behaviors (such as smoking, alcohol use, drug use, breastfeeding) and health outcomes (such as low birth weight, preterm birth, infant mortality, maternal morbidity).
4	To what extent were services delivered to the highest risk target populations (women and infants), as intended? (Health Services Utilization Evaluation)	3P’s Information Form and NHSPS,	Bivariate analyses will test for statistically significant differences in health behaviors, health service utilization patterns, and health outcomes between HS and non-HS participants and among HS participants, by level of utilization of HS services.
5	What factors (e.g., personal, program, and organization level) help explain the volume of services used? (Health Services Utilization)	3P’s Information Form and NHSPS	Bivariate analyses will test for statistically significant associations between various program and organization level factors and level of utilization of HS services. Program level factors may include the number of participants served during the preconception, pregnancy and postpartum periods; the outreach strategies employed; the number and types of referrals provided; the case management models utilized; the caseloads maintained; the number and types of screenings provided; if male involvement is promoted, among others. Organization level factors will likely include the type of program (urban, rural, border); the HS program level (1, 2 or 3); the lead agency type; age of the program; staffing characteristics; and the type of approaches and services provided, among others.
6	What impact did the transformed HS program have on HS participants when compared to non-HS controls? (Outcome Evaluation)	3Ps Information Form linked to Vital Records data (for 100 grantees) and to PRAMS as well (for 15 grantees)	The outcome evaluation analysis will estimate the effect of program participation by comparing outcomes of HS participants and non-participants using multivariable techniques. Individual-level propensity score matching will ensure that outcome comparisons between participants and non-participants are balanced with respect to observed characteristics. Given that there are likely to be many more non-HS participants in vital records than HS participants, the analysis could be statistically strengthened by a 1:N (3, 4) match. Multiple comparison groups, including internal references among program participants, will be used to test the sensitivity of results and promote causal inference (e.g. postpartum versus prenatal enrollees, dose-response effects).
7	What factors (program/organizational) of the transformed HS program are associated with improved participant behaviors, health services utilization, and health outcomes? (Outcome Evaluation)	NHSPS and 3 Ps linked to Vital Records data (for 100 grantees) and to PRAMS (for 15 grantees)	Analyses will also examine variation in effects by program and organizational characteristics to identify critical practices that can be spread and scaled to maximize impact across grantees.

3P’s Information Form = Preconception, Pregnancy, and Parenting Information Form

NHSPS = National Healthy Start Program Survey

CAN Survey = Community Action Network Survey

Reports

Results from monitoring will be synthesized semiannually to assess trends and changes in implementation and outcomes—allowing for corrections throughout the grant period. Results from the evaluation will be summarized at two points: December 2017 (Phase I) and December 2019 (Phase II). Analyses of program effects spanning five years of program implementation will allow MCHB to examine the program effects on changes in short-, medium-, and long-term outcomes as the program matures throughout the grant cycle; these reports will discuss the results from these analyses at two points in time. It is important to assess the program effects on outcomes at multiple points in time to identify when the changes in outcomes occur and link the changes to the maturity of the program, information that can be used in program improvement and replication. In addition, it is important to assess the program effects on outcomes over a relatively long period of time (in this case, five years) to give the program time to affect long-term outcomes, which are typically difficult to change, and observe at the population level in a short period of time. Study briefings will be held with key HRSA/MCHB staff, the Secretary’s Advisory Committee on Infant Mortality, grantees, and other program stakeholders. Additional publications may include peer-reviewed journal articles and issue briefs to disseminate results to the broader community of maternal and child health (MCH) policymakers and practitioners.

Schedule

Funding for the Healthy Start grantees began in September 2014 and November 2014, and will end in May 2019. After the receipt of funding, grantees began providing services and approximately a year later a data system to collect information was under development by HRSA/MCHB. The estimated schedule for the project is presented in Table A.6 for key data collection, analysis, and reporting tasks relevant to this request for OMB approval. The maximum three years of clearance is requested with the intent that an extension for OMB clearance will be requested to continue data collection if needed.

Table A.8. Estimated time schedule for data collection, analysis, and reports

Task	Time Schedule
Develop data collection tools	October 2015 – August 2016
Receive OMB approval	October 2016
Develop data collection systems	September 2015 – January 2017
Administer 3P’s Information Form
Train staff on data collection	October – November 2016
Collect individual-level data for monitoring (Healthy Start grantees)	October 2016–May 2019
Field National Healthy Start Program Survey
Collect program-level data (Round 1)	March 2016–April 2016
Collect program-level data (Round 2)	March 2018–April 2018
Field Community Action Network Survey
Collect program-level data (Round 1)	April 2017–May 2017
Collect program-level data (Round 2)	April 2019–May 2019
Conduct Site Visits	January 2019–April 2019
Conduct Focus Groups	January 2019–April 2019
Conduct Analysis and Reporting
Analyze and synthesize data (Phase I)	June 2017–December 2017
Develop Phase I report	September 2017–December 2017
Interim study briefing	December 2017
Analyze and synthesize data (Phase II)	June 2019–December 2019
Develop Phase II report	September 2019–December 2019
Final study briefing	December 2019

17. Reason(s) display of OMB expiration date is inappropriate

There are no exceptions to the certification; the expiration date will be displayed. To continue data collection in the last two years of the grant, an extension or revision to this package will be submitted for OMB clearance.

18. Exceptions to certification for paperwork reduction act submissions

There are no exceptions to the certification.

1 The nine previous Healthy Start components included five service and four systems components. Service components included direct outreach services and client recruitment, case management, health education services, screening and referral for perinatal depression, and interconception continuity of care through the infant’s second year of life. Systems components included utilization of community consortia and provider councils to mobilize key stakeholders and advise local grantees, development of a local health system action plan, collaboration and coordination with Title V services, and development of a sustainability plan for continuation of services and project work beyond the grant period.

2 OMB numbers for the two previous national evaluations are 0915-0287 and 0915-0300 for the first national evaluation and 0915–0338 for the second national evaluation.

3 Social capital can be defined as the networks of relationships among organizations and people in a community that encourage mutually beneficial cooperation and enable the community to function effectively.

4 Best practices in this case are those shown to be effective across organizations based on research. In contrast, promising practices are those shown effective in a particular situation or under a specific circumstance and hold promise for adoption by other organizations.

5 HHS regulations at 45 CFR 46.402(a) define children as “persons who have not attained the legal age for consent to treatments or procedures involved in the research, under the applicable law of the jurisdiction in which the research will be conducted.” If research on a specific treatment involves solely treatments or procedures for which minors can give consent outside the research context (under applicable state and local laws, for example, research on sexually transmitted diseases or pregnancy), such individuals would not meet the definition of children as defined at 45 CFR 46.402(a). Under these circumstances, minors may provide informed consent without parental permission. Thus, HS grantees will tailor consent for minors depending on participating sites’ state laws related to pregnancy, family planning, and treatment for minors.

2