Supporting Statement A

Supporting Statement Part A for Transformed Healthy Start

Program and Evaluation: Justification

OMB Supporting Statement Part A for Transformed Healthy Start Program and Evaluation

A. Justification

1. Circumstances making the collection of information necessary

This statement from the Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) requests Office of Management and Budget (OMB) approval for data collection to support monitoring and evaluation of the transformed Healthy Start program. The information collection is a revision to OMB# 0915–0338 and is authorized under the Healthy Start Reauthorization Act 2007 (Public Health Law No. 110-339), which includes appropriations for the Healthy Start initiative and its evaluation through fiscal year 2013 (Attachment A). The revision includes a modification of the National Healthy Start Program Survey to align with the transformed Healthy Start program model and the addition of four other data collection efforts including a Preconception, Pregnancy, and Parenting (3P’s) Information Form; Healthy Start Community Action Network (CAN) Survey; Healthy Start Site Visits; and Healthy Start Participant Focus Groups.

The purpose of the monitoring and evaluation is to assess the implementation of the program; measure the effect of the program on individual-, organizational-, and community-level outcomes; and identify best and promising practices for the program. Results from monitoring and evaluation efforts will provide actionable evidence to support the improvement, sustainability, replication, and dissemination of the program. In addition, monitoring and evaluation of the transformed Healthy Start program is consistent with the needs HRSA’s MCHB to meet its Government Performance and Results Act requirements.

The data collection effort to support monitoring and evaluation is of interest to MCHB as the federal agency for promoting and improving the health of women and children. HRSA/MCHB will use the results of the monitoring and evaluation to improve interventions for reproductive age women, their children, and families that will help reduce health disparities, decrease infant mortality, and improve perinatal health outcomes.

Background of Healthy Start

The national Healthy Start program aims to reduce disparities in infant mortality and adverse perinatal outcomes. The program began as a demonstration project with 15 grantees in 1991 and expanded over the past two decades to 105 grantees serving 196 communities across 39 states. Today, Healthy Start has evolved from a program framework of nine service and systems core components to five approaches: (1) improving women’s health, (2) promoting quality services, (3) strengthening family resilience, (4) achieving collective impact, and (5) increasing accountability through quality assessment, performance monitoring, and evaluation.¹

Need for monitoring and evaluation of the Transformed Healthy Start program

The transformation of Healthy Start, based on recent science, innovations, and legislation, will necessitate revised methods for monitoring and a reassessment of the program. Information from a strong monitoring and evaluation effort will contribute to the program’s continued evolution and transformation by shaping key programmatic decisions, identifying successful implementation strategies, and strengthening the evidence base for the program model. In addition, results from monitoring and evaluation can be used to meet Government Performance and Results Act requirements.

Healthy Start benefits from more than two decades of experience that included an evaluation of its demonstration and two previous national evaluations. This experience has influenced the design and priorities for the monitoring and evaluation of the transformed program.² Specifically, previous monitoring and evaluations revealed important information about Healthy Start’s implementation and contribution to improvements in birth outcomes. However, the monitoring and evaluations were fundamentally limited by a lack of consistently collected and high quality data on outcomes to assess the association between program components and outcomes. Although grantees collected administrative data on all of their clients at the individual level, their data collection was not standardized and only reported in the aggregate to MCHB. In addition, the lack of a comparison group has made it challenging to develop an assessment of the program’s effect.

Considering the lessons learned from the previous funding cycles of the Healthy Start program and its evaluations, HRSA/MCHB seeks to conduct uniform individual-level data collection across grantees for programmatic monitoring purposes and a mixed-methods evaluation of the transformed Healthy Start program that includes the following four design components: (1) quasi-experimental design using an external comparison group to assess program impact on individual-level outcomes and differences in community-level outcomes, (2) multilevel design employing hierarchical linear modeling to assess the associations among specific program components and outcomes, (3) network analysis to assess collaboration and coordination within communities, and (4) an implementation study to assess program operations.

Underlying the monitoring and evaluation of Healthy Start is the program logic model (Figure A.1). This framework was used to identify the data elements for collection related to program implementation, outputs, and outcomes. The longer-term outcomes—such as improved birth outcomes and decreased maternal and infant morbidity and mortality—are unlikely to be observed during the five-year study period. However, the logic model identifies the short- and intermediate outcomes that are known to be associated with the longer-term outcomes. The individual-, organizational-, and community-level outcomes prioritized for study in the evaluation are specified in Attachment B.

Figure A.1. Transformed Healthy Start program logic model

Data collection activities under the monitoring and evaluation of the Transformed Healthy Start program

To support uniform data collection and the mixed-methods evaluation design, five types of data collection activities will be implemented: 3P’s Information Form, National Healthy Start Program Survey, CAN Survey, Healthy Start Site Visits, and Healthy Start Participant Focus Groups (Figure A.2). Below, a description of each of these data collection activities is provided.

Figure A.2. Summary of the Transformed Healthy Start Program Data Collection Activities

MONITORING

All 88 Healthy Start grantees

Preconception, Pregnancy, and Parenting (3P’s) Information Form

40,000 participants

Community Action Network Survey

225 CAN board members and committee chairs

3P’s Information Form

Sample of at least 675 participants 4–7 months postpartum during a four-month study period

Site Visits

90 key informants

Focus Groups

180 participants

3P’s Information Form

All participants 4–7 months postpartum during a four-month study period

3P’s Information Form

Sample of 675 matched participants 4–7 months postpartum

during a four-month study period

National Healthy Start Program Survey

88 project directors/staff

E

V

A

L

U

A

T

I

O

N

15 Selected Matched Comparison Sites

All 88 Healthy Start Grantees

15 Selected Healthy Start Grantees

• The Preconception, Pregnancy and Parenting (3P’s) Information Form (Attachment C) will collect uniform information at the individual level about women eligible for Healthy Start and their children (up to 2 years of age) and families for monitoring and evaluation purposes. These data have traditionally been collected by Healthy Start grantees at the individual level within their own administrative data systems; however, they have not been collected in a standardized format and have only been reported to MCHB in the aggregate. Under this grant cycle, MCHB is making an effort to improve the quality of information already being collected by grantees by supporting standardization of key program data elements, which will also support assessments of program performance and evaluation. The data elements on the 3P’s Information Form are limited to those considered necessary to describe the reach of the program and the services provided, and to develop measures as specified in the funding opportunity announcement (Attachment B). The women eligible for Healthy Start include those of reproductive age (ages 15–44 years) living in communities with the poorest perinatal outcomes in the nation, including low birth weight and infant mortality.

• The National Healthy Start Program Survey (NHSPS) (Attachment D) will collect information about implementation of the program across the five key approaches of Healthy Start. All (approximately 88) Healthy Start projects will be asked to complete this survey. Project directors may delegate completion of sections of the survey to other Healthy Start staff. The survey is designed for self-administration through a web-based application that will allow the respondent to stop in the middle and resume the survey at another time. Healthy Start projects will be asked to complete the survey three times—at the end of the first, third, and fifth grant years.

• The Community Action Network Survey (Attachment E) will collect information about the health networks and social networks that support maternal and child health and social capital within the community.³ Approximately 10 to 15 active CAN board members and committee chairs for 15 Healthy Start projects that are selected for in-depth data collection will be asked to complete the survey. CAN members include representatives of organizations and agencies in the community that range from state and local government to community-based organizations. The CAN may also include individual consumers and community leaders. However, the instrument is designed to assess the relationships between agencies and organizations in the community to address maternal and child health, and as a result, individuals without relevant organizational/agency ties will not be included among respondents. Consumer and community leader involvement will be captured through the NHSPS and site visits. The CAN survey is designed for self-administration through a web-based application. Active CAN members will be asked to complete the survey three times—during the first, third, and fifth grant years. Methods for selecting Healthy Start sites for in-depth study are described in Supporting Statement B.

• Healthy Start Site Visits (Attachment F) will collect in-depth qualitative information about program implementation and achievements. The information collected will also give context to quantitative outcomes and help identify best practices. Site visits will be conducted at the 15 Healthy Start projects selected for in-depth data collection. During the site visits, in-person interviews will be conducted with four types of informants: Healthy Start project directors and administrative staff, Healthy Start core service staff, health care providers, and Healthy Start CAN members. Site visits will be conducted once during the fifth grant year. Methods for selecting Healthy Start sites for in-depth study are described in Supporting Statement B.

• Healthy Start Focus Groups (Attachment G) will collect participants’ perspectives on program implementation, individual-level networks, and social capital within the community. The focus groups will be conducted in the communities of the 15 Healthy Start projects selected for in-depth data collection. Similar to the site visits, they will provide context to quantitative outcomes. Each focus group will include 10 to 12 Healthy Start participants. The focus groups will be conducted in English and Spanish at a minimum; other languages will be determined by the populations served. These focus groups will occur during the fifth grant year at the same time as the site visits. Methods for selecting Healthy Start sites for in-depth study are described in Supporting Statement B.

Information collected through these five activities will be used together to monitor implementation and evaluate the effectiveness of the Healthy Start program in improving perinatal health among disadvantaged populations. The mixed-modes data collection approach will capture both quantitative measures of program activities, outputs, and outcomes as well as qualitative impressions of program implementation and lessons learned. This data collection approach will generate results useful to policymakers and practitioners, informing them about the implementation and value of Healthy Start as an intervention working at multiple levels to reduce infant mortality.

2. Purpose and use of information collection

The purposes of the monitoring and evaluation are aligned with Healthy Start program needs and goals for accountability, programmatic decision making, and ongoing quality assessment at the grantee and national levels. The monitoring and evaluation of the transformed Healthy Start Program are focused around the following goals:

• Provide real-time information to assess implementation of the program and enable identification of issues at earliest possible stages for midcourse corrections among individual grantees and for the program as a whole

• Provide credible and rigorous evidence of program effect on outcomes at multiple levels and across the life course of participants

• Assess the relationship between program components and outcomes to identify the relative contribution of components to desired outcomes for programmatic decision making

• Identify best and promising practices in implementation for replication and dissemination of the program⁴

• Strengthen the evidence base for multipronged initiatives to improve maternal and child health

To reach these goals, the monitoring and evaluation will address five key research questions. Aligned with the purposes stated in Section A.1, each question has a monitoring and evaluation design component associated with it in parentheses.

1. What is the program’s progress over time, and what are areas of program success and areas for improvement? (program monitoring)

2. How does the program perform on individual-, organizational-, and community-level outcomes? (outcomes study)

3. What are the relative contributions of the five Healthy Start approaches to individual- and community-level outcomes? (multilevel study)

4. To what extent is the program fostering community action and facilitating coordination among services for women, children, and their families to increase social capital and collective impact? (network study)

5. To what extent are the Healthy Start program components implemented with fidelity, and how might implementation influence individual-, organizational-, and community-level outcomes? (implementation study)

   1. What are Healthy Start best and promising practices in program implementation?

   2. Which Healthy Start practices can be replicated and under which circumstances?

The strength of the data collected for the monitoring and evaluation will be critical in the development of credible results. Table A.1 summarizes each data collection method and the monitoring and evaluation components into which they will feed.

Table A.1. Data collection efforts and design component

Data Collection Effort	Program Monitoring	Outcomes Study	Multilevel Study	Network Study	Implementation Study
3P’s Information Form					
National Healthy Start Program Survey					
Community Action Network Survey					
Site Visits					
Focus Groups					

Below, we discuss the specific use of the information collected under each method.

• The 3P’s Information Form (Attachment C) is designed to collect information about Healthy Start participants across all Healthy Start grantees and women at 15 comparison sites. The flow of the form mirrors the life course of preconception to pregnancy to delivery to postpartum; the instrument ends with factors that will influence the woman’s trajectory through the life course, including her community and the services she receives from Healthy Start. The form has 10 sections: (A) enrollment and demographic information, (B) pregnancy status, (C) client health/risk information, (D) previous pregnancy information, (E) birth outcomes and postpartum information, (F) child and parenting information, (G) health education, (H) health service utilization, (I) the client’s perspective on her community, and (J) receipt of Healthy Start services (Healthy Start participants only). These data are the source of individual-level demographic, services, and outcomes data essential for monitoring the program, developing the benchmarks as shown in Attachment B, and conducting the outcomes and multilevel studies of the evaluation. The variables provided through the form are also essential to facilitate effective matching of women at comparison sites to Healthy Start participants, which is crucial in measuring the effect of the transformed Healthy Start program. In addition, some elements related to access to services, service receipt, and community supports will be used in the implementation and network studies. HRSA/MCHB will also use these data elements for purposes of program monitoring, such as Healthy Start services use and risk profile and outcomes for the population served. Most of these data, except for Section I on client’s perspective on her community, were collected by grantees in a nonstandardized format in previous grant cycles.

• The National Healthy Start Program Survey (Attachment D) is designed to provide high quality information about the implementation of the Healthy Start program across its five key approaches. Accordingly, after the first section of the survey asking for general program information, the five subsequent sections correspond to each of the five approaches, and the final section ends with questions about program achievements. The sections of the survey are (1) overview of services, staffing, outreach, and retention; (2) improve women’s health; (3) promote quality services; (4) strengthen family resilience; (5) achieve collective impact; (6) increase accountability through quality, performance monitoring, and evaluation; and (7) Healthy Start project achievements. These data will be used for conducting outcomes and multilevel studies to provide variables related to program components and intervention models that may explain outcomes. For the network study, the data will provide information about the nature and extent of Healthy Start projects’ collaboration and linkages in the community. For the implementation study, the information will be used to assess services offered and provided, intervention models used by projects, aggregated outcomes for the population served, and achievements at the grantee and national levels.

• The Community Action Network Survey (Attachment E) is designed to collect information about implementation of the Healthy Start program as related to the health and social networks to support maternal and child health, and social capital within the community. The sections of the survey are (1) organizational information, (2) CAN participation, (3) infrastructure for collaboration, (4) quality of collaboration, (5) progress toward achieving goals, and (6) perspectives on the community. Information from the CAN survey will be used mainly in the network study to quantify the relationships between organizations and agencies within the 15 selected Healthy Start communities. It will also be used in the multilevel study and implementation study to provide variables to describe aspects of program implementation as related to partnerships and resources in the community.

• Healthy Start Site Visits (Attachment F) will include key informant interviews that will cover several aspects of program activities, including staffing, services provided, populations served, partnerships, networks, and reflections on challenges and successes. Qualitative information from the site visits will be used mainly to assess program implementation and identify and describe best and promising practices. By providing information about the nuances of program implementation, it may provide context to quantitative outcomes.

• Healthy Start Focus Groups (Attachment G) are designed to capture the participants’ perspectives on program implementation, individual-level networks, and social capital within the community. The focus group protocols include the following areas for discussion: outreach and participation, services received, case management and service coordination, home visiting, counseling and support, health education/promotion, medical home, and perspectives about the community. Similar to the site visits, the focus groups will provide information about program implementation, specifically about outreach, populations served, and services provided, but from the participant perspective. It may also provide context to interpret quantitative outcomes.

3. Use of improved information technology and burden reduction

3P’s Information Form, National Healthy Start Program Survey, and Community Action Network Survey. These three data collection efforts will comply with the Government Paperwork Elimination Act (Public Law 105-277, Title XVII) by employing technology efficiently in an effort to reduce burden on respondents. HRSA/MCHB will use an online, web-based application to obtain information from respondents. The application will include automated range checks and branching and will enforce consistency among critical questions to optimize resources and facilitate collection of high quality data. The programming will allow the collection of information specific to each respondent by skipping respondents out of questions not pertinent to them, thereby eliminating undue time burden on respondents. The application will also allow respondents to stop and return to the instrument so that they can complete it at their convenience. The instruments solicit only information that corresponds to the specific research items discussed in Section A.2, above. No superfluous or unnecessary information is being requested of respondents.

Healthy Start Site Visits and Focus Groups. As these are qualitative data collection efforts, HRSA will not use information technology to collect information from 90 staff and stakeholders during site visits (four key informant interviews at each of the selected 15 Healthy Start sites) and the approximately 180 Healthy Start participants during the focus groups (12 participants at each of the 15 selected sites). The data collection is qualitative in nature and requires information from a relatively small number of individuals; therefore, it is neither appropriate nor practical nor cost-beneficial to build electronic instruments to collect the information. All information will be collected orally in person using discussion guides, supported by digital recordings. Site visit and focus group transcripts will be analyzed using Atlas.ti, a software system used for the qualitative analysis of large amounts of data collected in text format.

4. Efforts to identify duplication and use of similar information

There are no current HRSA/MCHB data collection activities for monitoring and evaluating the transformed Healthy Start Program. The information that we are requesting to collect described in this OMB package is not available elsewhere.

5. Impact on small businesses or other small entities

This activity does not impact small entities.

6. Consequences of collecting the information less frequently

Table A.2 summarizes the data collection efforts, including the frequency of information collection. After the table, we discuss the consequences of collecting the information less frequently for each data collection activity.

Table A.2. Summary of data collection efforts

Data Collection Method	Data Collected	Respondents	Administration	Rounds of Data Collection	Consequences of less frequent data collection
3P’s Information Form	Individual-level data	~40,000 Healthy Start participants annually across 88 Healthy Start projects ~675 comparison women during each round at 15 comparison sites	Web-based form, administered by Healthy Start staff to participants Comparison organization staff or MCHB data collection staff to participants	Ongoing for Healthy Start participants; 3 rounds for comparison women during first, third, and fifth grant years	Limit the ability to assess outcomes and overall program and individual grantee performance Limit the ability to assess changes as the program matures
National Healthy Start Program Survey	Program implementation and aggregate outcomes data	All Healthy Start projects each round (~88)	Web-based survey, self-administered by Healthy Start project director and staff	3 rounds during first, third, and fifth grant years	Limit the ability to link changes in outcomes to the implementation of program components and identify the best and promising practices associated with better outcomes.
Community Action Network (CAN) Survey	Organizational-level data	~600 CAN members across 15 selected Healthy Start projects	Web-based survey, self-administered by CAN members	3 rounds during first, third, and fifth grant years	Limit the ability to assess changes in community-level and systems outcomes of the program and link them to changes in individual-level outcomes.
Site Visits	Qualitative program implementation information	~90 key informants across 15 selected Healthy Start projects	Interviews with Healthy Start project director, core service staff, providers, and CAN members	1 round during the fifth grant year
Focus Groups	Participants’ perspectives of implementation	~180 participants across 15 selected Healthy Start projects	Group discussions led by national evaluation staff	1 round during the fifth grant year

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

This request fully complies with 5 CFR 1320.5. There are no special circumstances.

8. Comments in response to the federal register notice/outside consultation

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on January 28, 2014, Volume 79, Number 18, Page Numbers 4476–4477. No comments were received.

In an effort to consult with experts both inside and outside the Department of Health and Human Services, HRSA/MCHB presented a description of the planned evaluation of the transformed Healthy Start program during the Secretary’s Advisory Committee on Infant Mortality in April 2013. In addition, HRSA/MCHB staff reviewed the survey and provided feedback on the electronic versions of the instruments during several conference calls.

HRSA/MCHB pre-tested the 3P’s Form, the NHSPS, and the CAN survey across three grantees during the period of 1/28/2014 – 2/14/2014. All pre-tests were conducted using a paper version. The results of the pre-test and recommendations for finalizing the instruments are presented in Attachment I. The pre-test allowed us to validate the length of the instruments and, thus, reduce the public burden. The pre-test also allowed us to refine and clarify the instructions and language; responses collected during the pre-test were not and will not be analyzed.

The instruments were revised based on results of the pre-test and feedback from HRSA/MCHB staff. Contact information for the three grantees that participated in the pilot is provided in Table A.3:

Table A.3. Pre-test grantee contact information

NAME	CONTACT INFORMATION
Tiffany Wootson Majors, MBA, MHA Project Director, Baltimore Healthy Start	410-396-7318 ext. 232 [email protected]
Maria Lourdes F. Reyes, MD, MPH Director of California Programs California Border Healthy Start	Tel: (619) 791-2610 ext. 305 [email protected]
Virginia Berry White, LMSW Project Director, Low Country Healthy Start	803-531-8008 [email protected]

9. Explanation of any payment/gift to respondents

3P’s Information Form. Below, we describe our rationale for providing compensation for completion of the individual-level data collection form.

• Monitoring. Healthy Start participants will not be compensated for completing the 3P’s Form, as information will be collected as part of the enrollment and participation process and will be essential for providing, targeting, and improving services for these women.

• Evaluation. Data for Healthy Start participants will be drawn from the monitoring data collection activities for the evaluation. However, HRSA recognizes the time burden placed on respondents at the 15 selected comparison sites without Healthy Start in the community and, therefore, not receiving Healthy Start services. Respondents at comparison sites will receive nominal compensation in the form of a $25 gift card for responding to the 30-minute survey. Such an incentive will (1) encourage eligible postpartum women to agree to participate in the study, (2) minimize the burden for recruitment, and (3) improve response rates, particularly among hard-to-reach populations. Based on evidence from previous research, we expect that offering incentives to the women will increase their willingness to participate and decrease the amount of effort needed to recruit them, thus imposing less recruitment burden on and requiring less interviewing resources by the organizations. Moreover, a higher response rate will improve the reliability and validity of the data for analytic purposes.⁵

One of the classic references that provides support for paying respondents for survey participation is Singer and Kulka (2001), who conducted research on the effect of changes in welfare policy on low-income people (similar to the target population for Healthy Start).⁶ They noted that obtaining “a representative sample can be problematic because members of low-income groups may not be highly motivated to participate in surveys. Incentives—especially monetary incentives—are particularly useful in countering this difficulty, as a supplement or complement to other efforts at persuasion.” This study also found a differential effect of incentives on minority populations. They cited research based on the 1996 panel of the Survey on Income and Program Participation, showing that the use of a $20 incentive was much more effective in recruiting and retaining black and poor households, compared to nonblack and nonpoor households; Healthy Start serves low-income, minority populations.

Markesich and Kovac (2003) reported on the effects of an incentive experiment on response rates and efficiency in a telephone survey with a low-income population.⁷ First, they found that incentives had a positive impact on survey participation: an incentive of $20 yielded a response rate of 68.7 percent, and an incentive of $35 yielded a response rate of 73.3 percent. Second, they found that it took the lower-incentive group 2 1/2 weeks longer to reach the same response rate as the higher incentive group. A higher response in a shorter period of time results in cost savings to the survey. They also commented that, “While there is no gold standard on how much incentive to offer a survey respondent, the OMB has approved use of monetary incentives in the range of $20 to $30 with specific target populations similar to those of interest here.” The referenced study populations were recipients of Temporary Assistance for Needy Families—a low-income population similar to the population of interest for Healthy Start.

HRSA/MCHB will provide a postpaid gift card (customized to the comparison site) worth $25, upon completion of each round of the form. Though we anticipate that most women would not meet eligibility criteria during all three rounds of data collection, respondents to all rounds of data collection could receive a total of $75 across the three rounds. The $25 incentive will be made in the form of a gift card, because those benefits are easier and more convenient to redeem than checks, especially for participants who may not have bank accounts. HRSA/MCHB has used Target/Walmart cards for the participant survey component of the previous Evaluation of the National Healthy Start Program (OMB #0915-0300).

Healthy Start Focus Groups. To encourage attendance, focus group participants will be given an incentive of a $25 gift card when they attend the focus group. The incentive will compensate focus group participants for the burden they incur. In addition to each focus group taking 90 minutes, participants must travel and potentially hire somebody for child care.

National Healthy Start Survey, Community Action Network Survey, and Healthy Start Site Visits. Healthy Start staff and stakeholders will not receive incentive payments because most are participating as part of their professional positions.

10. Assurance of confidentiality provided to respondents

3P’s Form. HRSA/MCHB has embedded protections for privacy in the study design. The information collection will fully comply with all aspects of the Privacy Act. Individuals and agencies will be assured of the privacy of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). All participants will be told during the consent process that the data they provide will be treated in a confidential manner to the extent allowed by law.⁸ They also will be informed that participation is voluntary, that they may refuse to answer any question, and that they can stop at any time without risk to their receipt of services outside of Healthy Start. In addition, their name will not be provided to the federal government. A unique ID will be assigned to each participating woman, and grantees/organizations at comparison sites will keep a separate list of names with unique IDs; this list will not be shared with MCHB.

At Healthy Start sites, individuals must have a form record in the data collection system to be considered a program participant. Similarly, at comparison sites, individuals must have a form record in the data collection system to be considered a study participant. The form can be incomplete if the individual refuses to provide all information requested. However, without a form in the data collection system, grantees and comparison sites will not have a means for accounting for individuals recruited (and Healthy Start services provided) to HRSA/MCHB. Organizations will ask program/study participants to sign an informed consent form to authorize participation in the program/study. A staff member will read the consent elements and record the sample member’s response in the data collection system. HRSA/MCHB will work with grantees and organizations at comparison sites to customize consent procedures so they are acceptable to HRSA/MCHB, the participating organizations, and the Institutional Review Board.

National Healthy Start Program Survey, Community Action Network Survey, Healthy Start Site Visits. As part of establishing communication for the remaining data collection efforts, potential participants will be sent information about the study and what is required for participation. The elements of consent will be explained in these communications. In addition, we will develop consent forms and procedures for participants to sign at the time of data collection. We will develop consent forms as part of the web-based instrument for the NHSPS and CAN surveys that will request electronic signatures from respondents and paper-based consent forms for key informants to sign during site visits. No personally identifiable data will be collected from these data collection methods. The requested information is at the aggregate or organizational level.

Healthy Start Focus Groups. When Healthy Start participants arrive at the focus group location, they will be given a consent form to read, sign, and return to the moderator. The focus group moderator will answer any questions posed by the participants about consent or privacy.

In addition to specific security procedures for the various data collection activities, two approaches cut across the entire study. First, all contractor employees will sign a pledge that data will be kept private to the extent allowed by law and respondent identity, and breaking that pledge is grounds for immediate dismissal and possible legal action. Second, HRSA will seek Institutional Review Board clearance; the clearance will be the responsibility of the evaluation contractor with which HRSA engages to implement the data collection.

11. Justification for sensitive questions

3P’s Form. The 3P’s Form is designed to describe the health care and other social service support utilization, health knowledge, health behaviors, and perinatal outcomes of women participating in Healthy Start and those that would be eligible for Healthy Start at comparison sites. The form will help HRSA/MCHB assess how participation in Healthy Start may be associated with positive perinatal outcomes and reduce disparities in perinatal outcomes. A number of items in the form refer to personal behaviors and circumstances that may be of a sensitive nature for respondents. Examples of potentially sensitive health behavior questions include those related to smoking, alcohol, and drug use; screening for HIV/AIDS and sexually transmitted infections; breastfeeding; use of family planning methods; pregnancy loss or infant death; and race/ethnicity. However, it is necessary to collect information from women on these topics because research has linked such behaviors to birth outcomes, and Healthy Start provides services to promote relevant healthy behaviors, links participants to needed services, and aims to reduce disparities in outcomes. HRSA/MCHB has minimized the number of sensitive questions to those necessary for the purposes of monitoring and evaluation; the form includes questions that are directly relevant to assess outcomes and progress toward goals of the program. In addition, training of Healthy Start and data collection staff at comparison sites will stress the importance of asking all questions that involve sensitive issues in a professional and nonjudgmental manner. Finally, women will be assured that they do not have to respond to any questions that they do not want to answer.

Healthy Start Focus Groups. Similar to the 3P’s Information Form, the Healthy Start focus groups will ask participating women to discuss their experiences with the Healthy Start program. Topics that may come up during the focus groups include potentially sensitive ones, such as smoking, alcohol, and drug use; breastfeeding; use of family planning methods; pregnancy experiences, and family support. Qualitative information collected from women on these topics is important to understanding the contribution of Healthy Start and to provide context to outcomes. Training of the focus group moderators will emphasize the importance of discussing topics that involve sensitive issues in a professional and nonjudgmental manner and facilitating a supportive environment to promote constructive conversation and sharing. Finally, women will be assured that they do not have to talk about any topics that they are not comfortable discussing.

National Healthy Start Survey, Community Action Network Survey, Healthy Start Site Visits. There are no questions of a sensitive nature for these instruments.

12. Estimates of annualized hour and cost burden

The total annual burden hours estimated is summarized in Table A4. For each data collection effort, we use dollar per hour estimates to generate the estimated annualized burden hours.

We use the median wage ($16.71) for program participant hourly rate estimated by the Department of Labor, Bureau of Labor Statistics (BLS), Occupational Employment Statistics for all occupations in 2012. In addition, for the Healthy Start and comparison site staff that will be involved in recruiting respondents and administering the instrument, the annualized hour and cost burden is estimated at $23.96 per hour, based on BLS’s median hourly wage for health care social workers. Healthy Start administrative staff annualized hour and cost burden was estimated to be $45.15, based on BLS’s median hourly wage for all managerial positions. The total hour cost was calculated by multiplying the total burden hours by the hourly wage rate.

Table A.4. Estimates of annualized hour and cost burden

Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours	Hourly Wage Rate*	Total Hour Cost
Preconception, Pregnancy, and Parenting Information (3P’s) Form	40,675	1	0.50	20,338	$16.71-$45.15	$32,894
National Healthy Start Program Web Survey	88	1	2.00	176	$45.15	$7,946
CAN member Web Survey	225	1	0.75	169	$45.15	$20,318
Healthy Start Site Visit Protocol	15	1	6.00	90	$45.15	$3,688
Healthy Start Participant Focus Group Protocol	180	1	1.00	180	$23.96-$45.15	4,512
Total	41,183			20,953		$69,358

*HS program participants - $16.71; Healthy Start Outreach Staff - $23.96; Healthy Start Project Director - $45.15. The range in hourly wage rates for the 3P’s Form and Healthy Start Participant Focus Group protocol is reflective of the various staff and respondents involved in administering/completing the survey or conduct/attend the focus group.

13. Estimates of other total annual cost burden to respondents or record keepers/capital costs

There are no capital and start-up cost to respondents associated with this data collection.

14. Annualized cost to federal government

The evaluation will take place over a three-year period. The annualized cost to federal government for evaluation, which includes the data collection portion of the evaluation to be awarded to a contractor and HRSA staff time/resources estimated at 20% of GS 14 Step 5 annual rate ($120,429), which will not exceed 1 percent of the fiscal year Healthy Start program budget. The total estimated cost is $1,024,086.00 annually; approximately $3 million for the three year contract.

15. Explanation for program changes or adjustments

This is a revision to the existing data collection activity for Healthy Start. The purpose of the changes/adjustments was to decrease burden, standardize data collection, and capture individual level data. These changes will improve the quality of data that can be used to monitor and assess the association between program components and outcomes.

The National Healthy Start Program Survey was modified to decrease the amount of time it takes grantees to complete the survey and to improve question clarity.

The 3Ps form was developed to capture individual-level or program participant information that will allow grantees to perform real-time internal analysis and the Maternal and Child Health Bureau (MCHB), Division of Healthy Start Program Services (DHSPS) to pool data and get a snapshot of implementation and outcomes at both the national and project levels.

The Healthy Start Community Action Network (CAN) Survey was developed and designed to collect data needed to accurately measure the size and strength of the organizational networks in the Healthy Start community. In turn, this will provide important information on the extent to which Healthy Start and other community organizations are working together to achieve common goals.

16. Plans for tabulation, publication, and Project time schedule

Analysis plan

Although information from the various data collection efforts will be combined to answer the evaluation questions, the analyses of data will vary based on the specific questions. Analyses using the 3P’s Form, the NHSPS, the CAN survey, Healthy Start Site Visits, and Healthy Start Focus Groups are described in Table A.5 by evaluation question. The outputs and outcomes assessed are those shown in the logic model (Figure A.1 in Section A.1, with prioritized outcomes shown in Attachment B).

Table A.5. Analytic approach and methods for each Healthy Start evaluation question

Evaluation Question		Data Source(s)	Analytic Approaches & Methods
1	What is the program’s progress over time, and what are areas of program success and areas for improvement?	3P’s Information Form	Descriptive analysis assessing participant demographic characteristics, services received, and outcomes at a point in time and over time. Comparison of outcomes to benchmarks as provided in Attachment B. Analysis of changes in outcomes over time at the individual and aggregate level.
2	How does the program perform on individual-, organizational-, and community-level outcomes? (monitoring and outcomes study)	3P’s Information Form, NHSPS, and CAN Survey	Descriptive analysis comparing demographic characteristics, outputs, and outcomes of Healthy Start participants and nonparticipants: comparisons of means for bivariate and continuous variables (t-tests) and comparisons of distributions for categorical variables (chi-square tests). Descriptive analysis of changes in program and community-level outcomes over time. Multivariate analyses comparing outputs and outcomes of Healthy Start participants to nonparticipants while controlling for individual-, organizational- and community-level factors (using an indicator for individuals participating in Healthy Start and considering their length of participation): methods include ordinary least squares and logistic regression frameworks, regression discontinuity, and difference-in-difference estimators. Comparison to benchmarks: aggregate comparisons of participants to national benchmarks for women with comparable demographic characteristics (using the same methods as in the descriptive analysis). All analyses will incorporate sample weights to ensure that results generated using samples are representative of all participants and comparison group women.
3	What is the relative contribution of the five Healthy Start approaches to individual-, grantee-, and community-level outcomes? (multilevel study)	3P’s Information Form, NHSPS, and CAN Survey	Descriptive analysis examining outputs and outcomes of Healthy Start participants stratified by different individual-, organizational-, and community-level variables (e.g., program components and models): comparisons of means for bivariate and continuous variables (t-tests) and comparisons of distributions for categorical variables (chi-square tests). Multivariate analyses to assess the associations between individual-, organizational-, and community-level factors with output and outcomes of Healthy Start participants: methods include ordinary least squares and logistic regression frameworks and hierarchical linear modeling.
4	To what extent is the program fostering community action and facilitating coordination among services for women, children, and their families to increase social capital and collective impact? (network study)	3P’s Information Form, NHSPS, CAN Survey, Healthy Start Site Visits, and Healthy Start Focus Groups	Descriptive analysis of CAN participation, activities, and social capital: reporting of means, standard deviations, and distributions of key variables. Network analysis to assess the strengths and nature of relationships between organizations in Healthy Start communities. Qualitative analysis using Atlas.ti to organize the data and identify key themes in coordination, collaboration, and resources in the community.
5	To what extent are the Healthy Start program components implemented with fidelity, and how might implementation influence individual-, organizational-, and community-level outcomes? (implementation study)	3P’s Information Form, NHSPS, CAN Survey, Healthy Start Site Visits, and Healthy Start Focus Groups	Descriptive analysis of program components implemented, services provided, and other outputs: reporting of means, standard deviations, and distributions of key variables. Qualitative analysis using Atlas.ti to organize the data and identify key themes in implementation. Comparison of components implemented by grantees and Healthy Start guidance: reporting the percentages of grantees implementing specific aspects of the components and the methods in which they are implemented.

3P’s Information Form = Preconception, Pregnancy, and Parenting Information Form

NHSPS = National Healthy Start Program Survey

CAN Survey = Community Action Network Survey

Reports

Results from monitoring will be synthesized semiannually to assess trends and changes in implementation and outcomes—allowing for corrections throughout the grant period. Results from the evaluation will be summarized at two points: once halfway through the grant period (Phase I) and once at the end of the grant period (Phase II). Analyses of program effects spanning five years of program implementation will allow MCHB to examine the program effects on changes in short-, medium-, and long-term outcomes as the program matures throughout the grant cycle; these reports will discuss the results from these analyses at two points in time. It is important to assess the program effects on outcomes at multiple points in time to identify when the changes in outcomes occur and link the changes to the maturity of the program, information that can be used in program improvement and replication. In addition, it is important to assess the program effects on outcomes over a relatively long period of time (in this case, five years) to give the program time to affect long-term outcomes, which are typically difficult to change, and observe at the population level in a short period of time. Study briefings will be held with key HRSA/MCHB staff, the Secretary’s Advisory Committee on Infant Mortality, and grantees. Additional publications may include peer-reviewed journal articles and issue briefs to disseminate results to the broader community of maternal and child health (MCH) policymakers and practitioners.

Schedule

Funding for the Healthy Start grantees will begin in June 2014 and end in May 2019. After the receipt of funding in June, grantees are anticipated to begin providing services by September 2014, and a data system to collect information will be developed by HRSA/MCHB prior to the provision of services. The estimated schedule for the project is presented in Table A.6 for key data collection, analysis, and reporting tasks relevant to this request for OMB approval. The maximum three years of clearance is requested with the intent that an extension for OMB clearance will be requested to continue data collection for the remaining two years of the grant.

Table A.6. Estimated time schedule for data collection, analysis, and reports

Task	Time Schedule
Develop data collection tools	December 2013–January 2014
Receive OMB approval	Summer 2014
Develop data collection systems	June 2014–September 2014
Administer 3P’s Information Form
Train staff on data collection	August 2014
Collect individual-level data for monitoring (Healthy Start grantees)	September 2014–May 2019
Collect individual-level data (Comparison organizations—Round 1)	February 2015–May 2015
Collect individual-level data (Comparison organizations—Round 2)	February 2017–May 2017
Collect individual-level data (Comparison organizations—Round 3)	February 2019–May 2019
Field National Healthy Start Program Survey
Collect program-level data (Round 1)	April 2015–May 2015
Collect program-level data (Round 2)	April 2017–May 2017
Collect program-level data (Round 3)	April 2019–May 2019
Field Community Action Network Survey
Collect program-level data (Round 1)	April 2015–May 2015
Collect program-level data (Round 2)	April 2017–May 2017
Collect program-level data (Round 3)	April 2019–May 2019
Conduct Site Visits	January 2019–April 2019
Conduct Focus Groups	January 2019–April 2019
Conduct Analysis and Reporting
Analyze and synthesize data (Phase I)	June 2016–December 2016
Develop Phase I report	September 2016–December 2016
Interim study briefing	December 2016
Analyze and synthesize data (Phase II)	June 2019–December 2019
Develop Phase II report	September 2019–December 2019
Final study briefing	December 2019

17. Reason(s) display of OMB expiration date is inappropriate

There are no exceptions to the certification; the expiration date will be displayed. To continue data collection in the last two years of the grant, an extension or revision to this package will be submitted for OMB clearance.

18. Exceptions to certification for paperwork reduction act submissions

There are no exceptions to the certification.

1 The nine previous Healthy Start components included five service and four systems components. Service components included direct outreach services and client recruitment, case management, health education services, screening and referral for perinatal depression, and interconceptional continuity of care through the infant’s second year of life. Systems components included utilization of community consortia and provider councils to mobilize key stakeholders and advise local grantees, development of a local health system action plan, collaboration and coordination with Title V services, and development of a sustainability plan for continuation of services and project work beyond the grant period.

2 OMB numbers for the two previous national evaluations are 0915-0287 and 0915-0300 for the first national evaluation and 0915–0338 for the second national evaluation.

3 Social capital can be defined as the networks of relationships among organizations and people in a community that encourage mutually beneficial cooperation and enable the community to function effectively.

4 Best practices in this case are those shown to be effective across organizations based on research. In contrast, promising practices are those shown effective in a particular situation or under a specific circumstance and hold promise for adoption by other organizations.

5 OMB highlighted the importance of response rates in surveys in its January 20, 2006, memo “Guidance on Agency Survey and Statistical Information Collections” (page 56):

“A survey’s response rate is a valuable data quality and field performance indicator, and is probably the most widely cited single number associated with the generalizability of a survey’s results. A high response rate increases the likelihood that the survey results reflect the views and characteristics of the target population. Conversely, a low response rate can be an indicator of potential nonresponse bias, which would be detrimental to the accuracy of the results of a study in a variety of ways, including:

Survey estimates may be biased if those who choose to participate (respondents) differ substantially and systematically in some way from those who choose not to participate (nonrespondents). If these differences are related to critical information from the survey or the census, the results may be misleading or even erroneous.

The standard errors of the survey estimates may also be biased because an incomplete sample may fail to capture the true variability that would be observed in a complete sample.”

The same memo addresses the use of incentives noting, “Incentives are most appropriately used in Federal statistical surveys with hard-to-find populations or respondents whose failure to participate would jeopardize the quality of the survey data” (page 68). The memo also concludes, “Research has consistently shown that monetary incentives are more effective in increasing survey response than nonmonetary incentives” (page 69).

6 Singer, Eleanor, and Richard A. Kulka. “Paying Respondents for Survey Participation.” In Studies of Welfare Populations, Data Collection and Research Issues, Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, Michele Ver Ploeg, Robert A. Moffitt, and Constance F. Citro, Editors, Committee on National Statistics, National Research Council. Washington, DC: The National Academies Press, 2001. Singer and Kulka review research on the use of incentives, particularly with regard to low-income populations. They find that incentives significantly reduce survey nonresponse, and are cost effective, lowering the overall cost and burden for most surveys.

7 Markesich, Jason, and Martha D. Kovac. “The Effects of Differential Incentives on Completion Rates: A Telephone Survey Experiment with Low-Income Respondents.” Presented at the Annual Conference of the American Association of Public Opinion Research, Nashville, TN, May 16, 2003.

8 HHS regulations at 45 CFR 46.402(a) define children as “persons who have not attained the legal age for consent to treatments or procedures involved in the research, under the applicable law of the jurisdiction in which the research will be conducted.” If research on a specific treatment involves solely treatments or procedures for which minors can give consent outside the research context (under applicable state and local laws, for example, research on sexually transmitted diseases or pregnancy), such individuals would not meet the definition of children as defined at 45 CFR 46.402(a). Under these circumstances, minors may provide informed consent without parental permission. Thus, we will tailor consent for minors depending on participating sites’ state laws related to pregnancy, family planning, and treatment for minors.

1