29_Response to FRN Comments

Attachment 29: Response to Federal Register Notice

The government received 3 letters in response to the Federal Register Notice (Volume 76, Number 238, pages 77236-77237 on December 12, 2011). Each letter had multiple comments that are individually addressed below. The original letters follow this summary.

COMMENT 1: One comment about the survey of other community resources noted that it is brief and appropriate. (no response)

COMMENT 2: One comment was received that is supportive of the following elements of the design: the inclusion of model specific estimates, the attention to the experience and suitability of program sites for evaluation, and the inclusion of multiple home visiting models within a state to control for state-level influences. (no response)

COMMENT 3: One comment noted that the reliance on self-report measures of outcomes was problematic.

RESPONSE 3: The information collection package under consideration is for baseline instruments and implementation data collection only. A subsequent information collection package will focus on the outcome measurement. We appreciate the recommendations around specific measures for outcomes and will keep the recommendations in mind as the outcome data collection package is constructed.

COMMENT 4: The MIHOPE design includes numerous outcomes with limited sample size to detect effects, particular on specific subgroups (women who smoke during pregnancy, children born to women with fewer psychological resources). The comment recommended reducing the number of outcomes, deepening measurement on specific outcomes and increasing the sample size.

RESPONSE 4: The design of MIHOPE reflects HHS’s goal to be responsive to the legislative language while balancing resource constraints. The legislative language mandated that the MIHOPE evaluation must examine all participant outcomes and benchmark domains discussed in the legislation. The research team has attempted to streamline data collection while still remaining responsive to the Congressional requirement.

COMMENT 5: The implementation study is broad and burdensome on home visitors, program administrators and participating families.

RESPONSE 5: We appreciate the concern and have attempted to streamline many of the data collection elements within the implementation study. However, implementation science has shown that high quality program implementation is important to obtain expected impacts. HHS believes the implementation of home visiting programs is a critical element to understanding program impacts.

COMMENT 6: One comment noted concern about the lack of alternate services within the community to which the comparison group is referred.

RESPONSE 6: HHS recognizes that access to service is important for the families eligible for MIECHV. For ethical reasons the study will only include communities that have greater demand for home visiting than is available. Therefore, whether the MIHOPE study is in that community or not, some families would not have been able to receive services anyway.

COMMENT 7: One comment advised the team to ensure that efforts around outreach or recruitment of participants do not violate state contract specifications around contractually specified target population.

RESPONSE 7: We appreciate the comment and will ensure participating in any activities related to MIHOPE do not violate state contractual requirements.

COMMENT 8: One comment noted some state activities around home visiting may interfere with the evaluation.

RESPONSE 8: The MIHOPE recruitment team will be gathering information from states and local implementing agencies to determine the feasibility of evaluation participation on a host of variables, including the context.

COMMENT 9: One comment noted that local sites already have substantial data collection responsibilities to meet the benchmark requirements.

RESPONSE 9: MIHOPE recognizes the burden local sites have in collecting benchmark data. The implementing agency staff will not be responsible for data collection efforts in MIHOPE.

COMMENT 10: Some questions in the interviews of program managers and state administrators are redundant with information that is available from other sources.

RESPONSE 10: The evaluation is built on a multi-level conceptual framework of the determinants of home visiting service delivery and impact.  By design, some constructs are measured across instruments targeted to different levels of the framework, such as state administrators, program managers, supervisors, and home visitors. The intent is to assess consistency and explain differences in perceptions of stakeholders across these levels, and to assess how consistency and differences relate to service delivery and impacts. The study uses both quantitative and qualitative approaches.  Quantitative approaches will provide standardized descriptive data on factors for service delivery and actual service delivery.  Qualitative approaches will provide unique contextual information and insight into critical implementation processes.

COMMENT 11: One comment asked about state IRB involvement in the evaluation.

RESPONSE 11: MIHOPE will work with each state that is chosen for the evaluation to ensure appropriate involvement by states’ IRBs in the evaluation procedures.

COMMENT 12: Three comments requested clarification on the time it will take to complete the weekly logs by local staff, and one expressed concern over time burden for the logs to be maintained by home visitors and supervisors, especially with multiple families per home visitor or multiple home visitors per supervisor.

RESPONSE 12: The logs have been revised to be more streamlined. It is anticipated that not all families in a home visitors’ caseload and not all home visitors under a supervisor will be in the study. In addition, the home visitor may not see every family each week. Therefore it is now estimated the weekly burden per home visitor or supervisor will be approximately 12 minutes per week across all cases participating in the evaluation.

COMMENT 13: One comment requested clarification on what is meant by state administrative records.

RESPONSE 13: MIHOPE hopes to match MIECHV participants to, at a minimum, birth records and child protection records.

COMMENT 14: One comment asked if data collected will be shared with the state and local programs.

RESPONSE 14: The data collected will only be reported in the aggregate, not at the individual or site level.

COMMENT 15: One comment requested clarification on the relationship between the MIECHV funding and the eligibility of families to be included in the study.

RESPONSE 15: MIHOPE may include families that are not specifically MIECHV funded but are enrolled in programs receiving MIECHV funding, since the participants who are not MIECHV funded are expected to have similar programmatic experiences to those families funded by MIECHV.

COMMENT 16: Two comments expressed concern that the burden estimates for some of the instruments was underestimated. In addition, one comment expressed concern about respondent fatigue for elements such as the video snippets at the end of the home visitor survey. Finally, one commenter felt the video snippets may feel like a test and induce stress on the home visitors.

RESPONSE 16: The majority of the surveys have been significantly refined and streamlined from the draft version. The team intendeds to conduct pretesting to ensure the survey can be completed within the estimated burden. The study team will provide instructions indicating that it is not a test and that responses will be used only for research purposes.

COMMENT 17: Two comments expressed concern that the survey collects much data that would easily be available from the basic information already collected by each of the models.

RESPONSE 17: Because the study design will be examining efficacy both within models but also across models, the data collected about programs must be consistent across models. The amount, quality and operational definition of most of the data required has been found to vary widely across home visiting models. Therefore HHS believes the collection of this information is critical to the success of the study.

COMMENT 18: There was a commenter who felt that the intrusiveness of a number of questions warrants great concern. In this regard, these were thought to pose threats to the establishment of the trusting relationship between the home visitor and the client, as well as obvious challenges to the validity of responses that are given to a relative stranger concerning such issues as drug use, child abuse, criminal history, parenthood with multiple partners, and citizenship.

RESPONSE 18: The research team has successfully administered similar sensitive questions to both families and home visitors in past research without sacrificing response rates. Families are reminded that participation in the study is voluntary and that they can refuse to answer any specific question that makes them uncomfortable.

Sensitive questions such as those regarding drug and alcohol use, depression, partner relationships, and intimate partner violence come from previously used surveys, such as Building Strong Families, Supporting Healthy Marriage, Baby FACES, and Early Learning Initiative, as well as the CDC’s Pregnancy Risk Assessment Monitoring Survey (PRAMS). Questions about citizenship have likewise been asked by a number of surveys of low-income immigrant populations conducted by the study team, such as the Children’s Health Initiative studies.  The study team has not encountered issues with lower response rates, break offs, or item non-response with any of these questions or measures. 

Analysis of data from the study team's previous experience with home visiting programs shows no difference in family engagement rates for families participating in the research as compared to families enrolling in the participating home visiting programs before and following the study recruitment period. Analysis of the responses to interview questions in sensitive areas such as psychosocial risks for poor parenting shows a high rate of agreement with assessments of these risks by service providers. 

COMMENT 19: Two comments raised concerns about the number of questions around drug use.

RESPONSE 19: We appreciate the concern and the survey has been modified to include a reduced set of questions on substance abuse from the CDC’s Pregnancy Risk Assessment Monitoring System (PRAMS) questionnaire.

COMMENT 20: One comment suggested that the questions regarding arrests, convictions and incarcerations are potentially demeaning and threaten the home visitor-client relationship.

RESPONSE 20: The survey has significantly reduced the number of questions, retaining only one. The legislative language requiring the MIHOPE evaluation specifies crime as an important participant outcome. Therefore HHS feels some data should be collected on this domain.

COMMENT 21: One comment expressed concern that asking a client if she has “ever worked for one employer for six consecutive months?” will be seen by many as implying that clients in these programs would not normally be up to doing so.

RESPONSE 21: The survey has been modified to ask how many months the individual has worked in the past three years. This survey question is one the research team has used successfully in past evaluations.

COMMENT 22: One comment noted that asking how many children the client and the father have had with other partners is extremely insensitive.

RESPONSE 22: This question has been removed from the survey.

COMMENT 23: One comment raised the issue that in the current political climate, asking the client if she is a citizen seems both unnecessary and catering to a “hot-button” issue.

RESPONSE 23: This question has been removed from the survey.

COMMENT 24: Two comments requested that questions concerning health insurance and service use be condensed.

RESPONSE 24: The number of questions regarding health insurance and service use has been reduced.

COMMENT 25: One comment expressed concern that the use of videotapes during home visits would compromise client privacy.

RESPONSE 25: As with other data collection activities, family participation in the video-recording of selected visits is strictly voluntary. Each family in the subsample selected for video-recording of visits will be asked to provide signed informed consent specifically for this data collection activity, and any adults in the home who might be video-recorded will also be asked to provide consent at the time of the recording.  Study procedures will adhere to strict guidelines to maintain confidentiality of the video-recordings. Families and home visitors will be able to request that the camera be stopped at any time. During diapering or breastfeeding, the camera will be adjusted so that only faces are being recorded.

COMMENT 26: One comment requested avoiding negatively phrased items and to ensure questions have both a positive and negative frame.

RESPONSE 26: We appreciate the comment and we have reviewed all of the instruments to balance positive and negative language.

COMMENT 27: Two comments expressed concern that asking multiple questions regarding home visitor psychological well-being is excessive and intrusive. A final comment suggested that the questions regarding psychological wellbeing may raise liability concerns for employers.

RESPONSE 27: The home visitors will be administered informed consent. The consent will clearly indicate participation in the evaluation will be completely voluntary and is not a condition of their employment. The home visitors may refuse to answer any individual questions they feel uncomfortable answering. . The instruments include, the CES-D and Attachment Style Questionnaire, are well standardized and widely used instruments. Because the constructs of home visitor depression and attachment style have been shown in past research to be important moderators of the relationship between the home visitor and client, the measures have been retained. The study team will develop a protocol to address any adverse psychological reactions arising from home visitors’ participation in the survey, should they occur.

COMMENT 28: Two comments noted there were some measures not included in the draft instrument and therefore the commenters could not judge the burden imposed by this instrument. RESPONSE 28: There was a comment that two measures were not included in the public documents: 1.) The Organizational Social Climate instrument is a proprietary measure and therefore cannot be included in the public documents. The authors of the instrument estimate it takes approximately 15-20 minutes to complete. The measure captures important constructs related to organizational context, culture and climate and individual-level work attitudes and burnout. 2.) The SF-12 is a proprietary measure that captures health status. According to the instrument developer, the SF-12 takes 2-3 minutes to administer.

COMMENT 29: One commenter included a number of recommendations about the “Self-completed questionnaires by home visitors during selected home visits.”

RESPONSE 29: This questionnaire has been deleted in its entirety.

COMMENT 30: One comment recommended that the information on personnel included in the program site manager surveys could be streamlined and restructured.

RESPONSE 30: The study team has removed most of these items as similar information will be collected directly from the home visitors and supervisors.

COMMENT 31: One comment noted concerns that some sections of the program site manager survey are long and complex.

RESPONSE 31: The research team has reduced, reorganized and streamlined the noted sections to improve the validity of the responses and reduce burden.

COMMENT 32: One comment noted that the reading level for many of the survey questions is not appropriate for the at-risk population that home visiting programs serve.

RESPONSE 32: We appreciate the recommendation. The study team is conducting pretests to make sure the families involved in home visiting programs understand the questions.

COMMENT 33: One commenter asked for a definition of the term “focal child,” and for clarification about whether program participants with children ages three through five will be included in the survey.

RESPONSE 33: The study will include only families with a child under six months old. Most families will have only one such child, which will be the “focal child.” For families with two or more children in that age group, one child will be chosen at random by the study team at the time of the baseline survey.

COMMENT 34: One comment noted that the survey format includes skip logic, which may be confusing for clients who take the survey with paper and pencil. In addition, if the survey will be online, clients will need to be provided with computers with Internet access and support in using a computer-based survey.

RESPONSE 34: The survey will be administered by an interviewer over the phone.

COMMENT 35: One comment noted the family baseline survey does not address whether a caregiver other than the child’s mother is required to participate in the survey, pointing out that most of the questions are focused on mothers and are written with that focus by, for example, using the pronoun “she.”

RESPONSE 35: The survey is intended to be limited to the caregiver who is eligible for the MIECHV funding. Computer-assisted interviewing will ensure that appropriate pronouns are used. Based on prior studies, the study team thinks it will be rare that caregivers other than mothers will be included in MIECHV programs. However, the survey will be administered in an appropriate fashion to those other caregivers as needed.

COMMENT 36: One comment noted that the survey does not ask about the current age of the child, which home visiting program the child is enrolled in, or the age of the child at enrollment.

RESPONSE 36: The study team expects to receive this information about the child before administering the baseline survey as part of the procedures for determining eligibility for participation in the survey.

COMMENT 37: One comment mentioned that many of the questions are sensitive. Please insert assurances guaranteeing family anonymity or confidentiality.

RESPONSES 37: All participants will be provided informed consent which explains the privacy and confidentiality provisions in the survey.

COMMENT 38: One comment noted that the survey does not allow the respondent to select “non applicable” or “declined to respond.”

RESPONSE 38: These categories have been added to the appropriate questions.

COMMENT 39: One comment requested omission of the second question that asks about the child’s birth weight since the first question will allow the analyst to determine if the weight was normal, low or very low.

RESPONSE 39: The second question will only be used if the respondent cannot provide the specific birth weight.

COMMENT 40: One comment requested that we drop the sensitive questions regarding child maltreatment that may lead to reporting.

RESPONSE 40: This question has been dropped.

COMMENT 41: One comment requested that we revise the questions about fathering relationships to refer to a romantic relationship or specify if questions reference the biological father or the current partner.

RESPONSE 41: This clarification has been added. Whenever a question is asked, the language will be clear whether the biological father or the current partner is the referent.

COMMENT 42: One comment noted that a 7 point scale may provide too many options for respondents and recommended using a 5 point scale.

RESPONSE 42: The seven categories will not be read to respondents. Respondents will instead be asked to provide a number from 1 to 7. The text is provided so that the interviewer can clarify what a particular number means. This question was used successfully in past research with similar populations.

COMMENT 43: One comment recommended separating “every day” and “almost every day” as responses in questions regarding father involvement.

RESPONSE 43: Past research has indicated that separating these two choices does not yield additional information. Therefore we would like to leave the categories as written.

COMMENT 44: One comment recommended that we focus the survey on the degree to which the respondent receives a given type of social support, rather than the names of individuals providing support. The degree of social support would allow for a comparison across the two time periods, whereas names do not reveal the degree of support received and would make an analysis difficult.

RESPONSE 44: This questionnaire has been dropped from the survey

COMMENT 45: One comment noted that the questions on interpersonal violence (IPV) use the past tense, but do not define the time period about which the respondent is being asked. Similarly, the questions do not specify whether the respondent should consider only a current partner, if the respondent has one, or a previous partner.

RESPONSE 45: These questions have been modified to include information on the number of times incidents occurred in the previous year. The questions were taken from the Conflict Tactics Scale, a commonly used instrument to measure IPV.

COMMENT 46: One comment requested changes in the questions on family economic self-sufficiency section: specifically, including the number of income earners in the households that share resources with the child; consolidating questions regarding employment and education; and consolidating questions on in-kind benefits.

RESPONSE 46: The survey has been modified to include information on earnings from other family members and the total household income to assess the available financial resources. Questions have also been added to ascertain whether the biological father shares resources with the family and focal child. The questions on employment and education have been reduced and streamlined. The survey now restricts questions on in-kind benefits to TANF, SNAP, WIC and SSI.

COMMENT 47: One comment stated the purpose of the state administrator survey is unclear, and asked for clarification about whether this information will be shared with others and if questions can be streamlined to utilize existing resources such as the state needs assessments.

RESPONSE 47: The state administrators’ responses will be private and will not be shared on an individual basis with the federal government. The data will only be shared in the aggregate across all participating states. The research team will attempt to access other data resources prior to the interviews with states to obtain relevant information.

COMMENT 48: One comment noted that the surveys of program site managers and home visitors are difficult to follow.

RESPONSE 48: We appreciate the comment. These surveys have been substantially streamlined and edited to improve readability and ease of completion.

COMMENT 49: One comment noted that because home visitors typically do not track hours spent on specific tasks, such as home visiting, manual paperwork, and data entry, it will be difficult to indicate the number of hours worked in a typical week on each listed topic. The commenter recommended using a percentage of time instead of a percentage of hours.

RESPONSE 49: Understanding how home visitors spend their time is an important goal of the study. This data is critical to the cost study that is mandated by the law. The study team is conducting cognitive interviews with home visitors to determine whether and how to ask these items to assure that home visitors provide valid and reliable responses.

COMMENT 50: One comment noted the survey questions about referrals to community resources and home visiting program goals should be asked of the program managers only and not the home visitors.

RESPONSE 50: It is important to understand home visitors’ knowledge, attitudes, and past experience regarding referrals to community services. It is also important to assess the home visitors understanding of the goals of the home visiting program and how that influences service delivery. This section has been reduced in burden for the home visitors. Home visitors are now only asked a smaller number of questions about referrals to and coordination with a smaller set of services.

COMMENT 51: One comment requested that since the qualifier words “excellent” and “highly” will lead to false negative responses, these words should be removed and that the questions should read, “I have had training to carry out this activity” and “I am skilled in carrying out this activity.”

RESPONSE 51: In this section, the study team aims to assess home visitors’ perceptions of the adequacy of their training and their skills. In the cognitive interviewing that is part of our pretesting, the study team will determine whether and how to modify these items to assure valid and reliable ratings of training adequacy and skill.

COMMENT 52: One comment requested clarification about whether the referrals of interest in the evaluation are focused on whether families are referred into the home visiting program or if they are focused on referrals participating families are receiving from the home visiting program to other community resources.

RESPONSE 52: The evaluation is focused on understanding those referrals participating families receive from the home visiting program to other community resources. The study team is pretesting the relevant items and will add clarifying language if the items are not clear.

COMMENT 53: One comment expressed concerns regarding the log maintained by Supervisors on Supervisory Activities; specifically, that filling out the proposed log may be time consuming for supervisors who typically oversee 12 or more visitors. The commenter recommended that to reduce the burden on supervisors, it may be more appropriate for the log information to be recorded once or twice a month rather than recording this detailed information weekly. To simplify the proposed individual supervision chart, the commenter recommends allowing the supervisor to indicate the number of visitors who received each type of supervision topic, instead of completing the individual supervision chart for each individual visitor. Additionally, if training information is collected in the log maintained by supervisors on supervisory activities, then the same information should not be collected in the other proposed MIECHV evaluation surveys because the log would provide the most accurate information.

RESPONSE 53: The study team has revised the instrument so that supervisors will complete the section on training once a month. Supervisors will only complete the section about supervisory activities related to specific home visitors for those home visitors who are providing services to families in the study. The study team anticipates that this will be for no more than 5 home visitors at a time. In addition, the study team has built in skip patterns for instances in which supervision was not provided to a specific home visitor. The study team has revised the other implementation instruments so that the logs will be the only data source that tracks training activities.

COMMENT 54: Two comments suggested that HHS has underestimated the time it will take to complete the self-completed questionnaires by parents and by home visitors.

RESPONSE 54: These questionnaires have been eliminated.

COMMENT 55: One commenter was unclear how information on program costs is relevant to the evaluation goals.

RESPONSE 55: The evaluation is interested in two aspects of program costs. First, understanding which agency(s) fund the program is vital to understanding the influential organizations that may shape the program or implementation system. Second, the legislative language included the requirement for MIECHV, if scaled broadly, to reduce costs. To fulfill this requirement, the evaluation includes a cost analysis of the program.

COMMENT 56: One commenter recommended eliminating the draft protocol for group and one-on-one interviews with home visitors because all relevant questions are addressed in the other proposed MIECHV evaluation tools. In addition, the commenter recommended eliminating the draft protocol for group and one-on-one interviews with home visiting supervisors because all relevant questions are addressed in the other proposed MIECHV evaluation tools. The commenter suggested that questions could be added to the other MIECHV evaluation tools only if they are necessary to achieve the stated evaluation goals.

RESPONSE 56: The study uses both quantitative and qualitative approaches.  Quantitative approaches will provide standardized descriptive data on factors for service delivery and actual service delivery.  Qualitative approaches will provide unique contextual information and insight into critical implementation processes. The research team believes that group and one-on-one interviews may be critical for understanding important processes that cannot be captured with the baseline survey or logs. The one-on-one supervisor interview has been eliminated.

DATE: January 11, 2012

TO: Administration for Children and Families

Office of Planning, Research and Evaluation

370 L’Enfant Promenade SW

Washington, DC 20447

Attn: OPRE Reports Clearance Officer

[email protected]

FROM: Nancy Peeler, MIECHV Project Administrator

Michigan Department of Community Health

109 W. Michigan Ave. FL 4

Lansing, MI 48913

[email protected]

517 335-9230

RE: Submission of Public Comments on Estimated Total Annual Burden

Mother and Infant Home Visiting Program Evaluation (MIHOPE)

Site Recruitment

a. We agree that the collection of information will be necessary in order to select sites for the study.

b. The estimate doesn’t take into account the time it will take the State Administrator/staff to collect additional information that is listed, but was not required in our MIECHV applications and which we may not have. For example, details about local evaluations such as sample size, etc.

c. Some of the information to be collected is not clear – definitions will need to be provided. Examples include:

In the calls with the State Administrator, Number of Families. If you are seeking information about the expanded slots generated through MIECHV funding (e.g. ‘new’), we have that information. If you are seeking information about total new program enrollment in a particular timeframe, regardless of fund source (existing or new from MIECHV), we would need to collect that information from our local programs.

Likewise, any information regarding military families would be new information that we would have to collect. Local sites may not have this information, as it may not be required by their models.

Also listed for the calls with State Administrators, what are you including under your definition of local evaluation? Does that included data collection for reporting or for CQI or monitoring purposes, in the absence of a planned evaluation?

d. The burden can be minimized by clearly defining terms ahead of time, and clearly defining what information is being sought from whom.

Regarding the study itself, we have several questions/comments:

1. Will you be seeking additional information/data from the states regarding the analysis of our Needs Assessments?

2. What is the incentive for a ‘comparison group’ family to participate in the study? It is not wise to assume that there are alternate services within the community to which a family can be referred in the event that they are not selected in the lottery to receive services. Also, what are you including in your definition of ‘alternate services’?

3. The study will need to be careful that in the course of outreach/recruitment and enrollment related to the study, they do not cause the local programs to violate state contract specifications around target population.

4. One of our infrastructure components is around coordinated local home visiting enrollment, and centralized outreach, intake, enrollment, etc. Participation in the study would impact this, given that the study wants to be involved in assigning who will enroll. Pilot sites in our ‘central intake’ project may not be able to participate in the study due to the potential conflict.

5. How will the study use the data that states are collecting to measure their benchmarks? Local sites already have a significant data collection responsibility under the MIECHV requirements; can some of what is being collected already be used, in lieu of a whole new set of data collection requirements?

6. Given that the study is happening within the context of grants given to states, will we need to send the study protocol through our state IRB process?

7. How much time will it take for local staff to complete the proposed weekly ‘webbased activity logs’ for the study? That has the potential to compromise service provision, and could reduce the number of families any one staff member can see, and put local programs out of compliance with their state contracts to provide services to a certain number of families.

8. There is reference made to accessing ‘state administrative records’; what records or information are included within that term?

9. Will study data collected in a given state be shared with the State, and not just the local program?

10. The study needs to clarify whether the children that will be enrolled in the study must be receiving services under ‘MIECHV’ funds, or if it can be any ‘new’ child enrolled in the program, regardless of fund source. For our state, if it is only MIECHV funded slots, for several programs, there are < 30 slots being funded, which potentially eliminates sites from your list. Also, some of our programs are using blended and braided funds, so there aren’t any truly ‘MIECHV’ funded slots.

11. It may be especially difficult for local sites to recruit enough families for the study if they are using a home visiting model that serves families with children up to five years of age (e.g., Healthy Families America), since families may remain in the program for a longer period of time and fewer slots will become available (e.g. fewer ‘new’ children coming into the program).

Dear Dr. Goldstein and Colleagues:

Attached please find Nurse-Family Partnership’s comments to the Maternal, Infant and Early Childhood Home Visiting Evaluation: Baseline Survey Data Collection, which was published in the Federal Register, Vol. 76, No. 238 on December 12, 2011.  We are grateful for the opportunity to offer our recommendations to this critical phase of the national evaluation and appreciate your continued willingness to engage us in discussions regarding the direction of the national evaluation. 

As you will see from our recommendations, we urge you to revise several important components of the evaluation to strengthen its reliability and sharpen its focus to determine the effects of home visiting models on important child and family outcomes of public health, educational and economic importance.  For example, we strongly recommend that administrative, medical, Medicaid and other reliable records be used to validate measurements as opposed to self-reporting by families participating in the study.  We also recommend that you significantly limit the number of outcomes measured and devote resources to reliably measuring a small set of primary outcomes that the models were found to impact in their respective trials and research.  We offer a number of other outcomes, including significantly curtailing the implementation study, which we believe poses a substantial burden on home visitors, programs and families served by home visiting that may have a negative impact on the delivery of services and willingness of families to participate in home visiting. 

As we were completing our recommendations regarding the baseline survey data collection, we learned that the new Strong Start for Mothers and Newborns Initiative recently announced by the Centers for Medicare and Medicaid Innovation may modify or augment the national evaluation for the MIECHV Program, possibly to validate outcomes through Medicaid and other reliable data, among other changes.  We are pleased to hear reports that sister agencies, the Administration for Children and Families, the Health Resources and Services Administration, the Centers for Medicare and Medicaid Services, and the Centers for Diseases Control and Prevention, are collaboratively working on the national evaluation.  We are anxious to learn more about this collaboration and specifically whether anticipated changes to the evaluation will address the recommendations we have offered.  We are anxious to partner with you to ensure that the national evaluation is well designed and executed to determine important outcomes for children and families. 

We are therefore requesting to meet with you and all relevant partners from your sister agencies who are now involved in shaping the national evaluation.  David Olds and various members of my staff are available to meet in the coming weeks.  In anticipation of the meeting, we would appreciate obtaining information regarding the manner in which the national evaluation will change as a result of the Strong Start Initiative.  I look forward to hearing from you at your earliest convenience. 

 

With warm regards,

Tom

Tom Jenkins | President & CEO
Nurse-Family Partnership | National Service Office
1900 Grant St, Ste 400, Denver CO 80203
Direct: 303.327.4274 | Toll free: 866.864.5226
Mobile: 303.552.6018 | e-Fax: 303.951.3874
Links: Website | Facebook | Twitter

February 10, 2012

Naomi Goldstein

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

370 L’Enfant Promenade SW

Washington, DC 20447

Re: Maternal, Infant and Early Childhood Home Visiting Evaluation: Baseline Survey Data Collection

Federal Register, Vol. 76, No. 238

Dear Dr. Goldstein:

We appreciate your commitment to ensuring that the Mother and Infant Home Visiting Program Evaluation (MIHOPE) is effectively designed and executed. Because MIHOPE seeks to answer whether evidence-based home visiting models improved the health and well-being of children and families, we are extremely pleased that it includes model-specific assessments of important child health and development outcomes. We are also grateful that MIHOPE includes a consideration of the experience and suitability of program sites engaged in the evaluation. In addition, we appreciate MIHOPE’s focus on states that are implementing several home visiting models simultaneously as a smart approach to control for state-level influences on programs’ abilities to implement home visiting programs well.

While we applaud many components of MIHOPE, we note that the current structure of the evaluation is unlikely to lead to valid estimates of the impact of home visiting models on specific child and family outcomes found in previous trials of these home visiting models. As a result, the evaluation is likely to be of limited use to the field and policymakers in determining whether the Maternal, Infant and Early Childhood Home Visiting Program (MIECHV Program) is a worthy investment of scarce public resources. Our key concerns and recommendations are briefly summarized below and discussed in more detail in this correspondence.

MIHOPE’s nearly exclusive reliance on self-reporting to measure outcomes, rather than a review of medical and administrative records and assessments of relevant bio-markers, is likely to lead to invalid measurements and an under-estimate of the real effects of the home visiting programs. Based on Nurse-Family Partnership’s trials, self-reporting is fraught with bias due to the tendency of control group participants to provide social desirable responses as opposed to more accurate responses provided by nurse-visited participants. Invalid measurements undermine the utility of the national evaluation to inform the home visiting field or justify future investment in the MIECHV Program. We recommend limiting the total number of outcomes and devoting resources to rigorous measurement of a small set of primary outcomes.

MIHOPE’s design, which evaluates numerous outcomes with limited sample sizes, lacks statistical power to detect effects that prior research of the Nurse-Family Partnership model predicts will be found only in important sub-populations of children and families. The inability to detect impacts on sub-populations, such as pregnant women who smoke, or children born to women with fewer psychological resources, will lead to the conclusion that Nurse-Family Partnership has weak or no impact, because it failed to examine impacts on important sub-populations of children and families that constitute major public health risk groups. We recommend reducing the number of outcomes studied and increasing the sample size and depth of measurement.

The scope of the implementation study is overly broad and unduly burdensome on home visitors, program administrators and participating families, resulting in the following negative consequences:

o Based on a review of the data collection instruments, the implementation study, which seeks to evaluate factors that influence the delivery of home visiting services, will consume significant resources which should be used to increase the sample size and conduct deep measurement of primary outcomes. We recommend that the implementation study evaluate core elements of each model that must be in place to produce strong outcomes and use those core elements as the focus of the implementation study. Recommendations on the core elements are set forth in Section III (A) of this correspondence.

o The implementation study substantially underestimates the burden placed on home visitors, administrators, and families, possibly by as much as 50%. Overly burdensome data collection will impair the ability of models to achieve outcomes and have a chilling effect on participation rates by children and families in need of home visiting services. Recommendations regarding the surveys, questionnaires and interviews are detailed in Appendix I.

o The MIECE Annual Survey of Program Site Home Visitors, which poses questions regarding the psychological functioning of home visitors, raises ethical concerns. All home visitors will become research subjects and will have to sign informed consent. This raises concerns about visitors feeling coerced into participation as an implicit condition of employment. In addition, state and federal law, including the Americans with Disabilities Act, prescribes the types of psychological tests employers can require of employees. To the extent that the questionnaire constitutes psychological testing, employing agencies and states may face liability for requiring home visitors to complete the questionnaire and/or consent to be subjects. We recommend deleting this line of inquiry and substantially curtailing the implementation study.

o The baseline surveys, questionnaires and interviews are missing key information needed to assess the utility, clarity and necessity of the information to be gathered.

I. MIHOPE Should be Limited to Primary Outcomes Measured by Reliable Sources, Rather than Self-Reporting

MIHOPE relies almost exclusively on self-reporting of family members to measure child and family health, educational and economic sufficiency outcomes. Self-reporting introduces social desirability bias into the evaluation, leading to potentially invalid measurements of outcomes. Previous research on the Nurse-Family Partnership demonstrated that nurse-visited women were more accurate in their self-reporting of undesirable behaviors than were women in control groups. Overall, parents know what is unacceptable behavior (e.g., prenatal tobacco use, abusive or neglectful behavior) and are unlikely to fully admit such behaviors, but once involved in a therapeutic relationship, are more likely to reveal those vulnerabilities. This leads to treatment differences in levels of disclosure, which undermine the detection of real differences in maternal and child health. In trials of the Nurse-Family Partnership, for example, nurse-visited women, compared to control-group counterparts, were more accurate reporters of the numbers of cigarettes they smoked per day at the end of pregnancy and of their children’s specific intellectual skills at age three (the control group inflated their children’s abilities).

In previous comments, we recommended that the national evaluation concentrate scarce resources on valid measurements of replicated outcomes found in previous trials of each of the home visiting models to determine whether those same outcomes are being reproduced in the MIECHV Program. In order to strengthen the validity of measurements of outcomes, we again recommend that you limit the long list of proposed outcomes and focus the evaluation on primary outcomes of clear public health, educational and economic importance found in previous research on each of the respective models. Limiting the number of outcomes in this fashion will allow for a thorough evaluation of each model’s respective outcomes using more objective administrative records, medical records, vital records, and Medicaid encounter and claims data records, among other reliable sources.

For the Nurse-Family Partnership, this approach would allow for thorough measurements of the following primary outcomes replicated in its multiple trials:

Prenatal Cigarette Smoking: Prenatal smoking is the most significant, modifiable contributor to fetal growth restriction and contributes to preterm delivery. Preterm delivery and low birth weight are the leading contributors to infant mortality. In addition to contributing to preterm birth, fetal growth restriction and infant mortality, prenatal tobacco exposure increases the risk of a host of health and developmental problems in children that are not simply mediated by low birth weight. Randomized controlled trials of the Nurse-Family Partnership have demonstrated consistent reductions in tobacco use among pregnant women, resulting in reductions in prematurity and low birth weight births among women who smoke. Given the importance of preventing and/or reducing prenatal tobacco usage, we strongly recommend that MIHOPE include prenatal cigarette smoking as a primary outcome measured with urine cotinine assays. Because pregnant women tend not to report cigarette usage accurately and reporting among nurse-visited women become more accurate over the course of pregnancy, cotinine tests are the most reliable way to measure this important outcome.1

1 Nurse-Family Partnership is not recommending, nor does it currently conduct cotinine testing of pregnant women and mothers enrolled in its program. However, Nurse-Family Partnership recommends urine continine assays at registration and the end of pregnancy for participants who are subjects of the national evaluation to obtain the most reliable measurements of changes in cigarette smoking.

2 See The Economic Burden of Child Maltreatment in the United States and Implications for Prevention available at http://www.sciencedirect.com/science/journal/aip/01452134.

Preterm Birth and Low Birth Weight: As indicated above, preterm birth is the leading cause of infant mortality and a significant contributor to childhood morbidity. In clinical trials, the Nurse-Family Partnership model demonstrated reductions in preterm birth and low birth weight among women who smoked cigarettes. The public health significance of this outcome is unquestioned. We therefore recommend that non-elected preterm birth and low birth weight be examined as primary outcomes using medical records as the source of data, with a particular focus on women identified as cigarette smokers at program registration. We recommend that the sample size of the trial be increased to ascertain model-specific impacts on this set of outcomes and that the sample sizes be increased to ensure adequate statistical power to detect moderate program impacts (for example, a 20% reduction in preterm birth or low birth weight).

Inter-Birth Intervals: Closely-spaced subsequent births increase the risk for low birth weight, infant mortality, and compromised development in children, including increased risk of autism. In addition, closely-spaced births hinder the ability of parents, particularly mothers, to obtain gainful employment due to child care and other challenges. As a result, closely-spaced births increase parental dependence on public assistance and the costs associated with increased health care under Medicaid. In each of the randomized controlled trials of the Nurse-Family Partnership, there were significant reductions in closely spaced pregnancies and births among-nurse-visited mothers compared to control-group mothers. This public health outcome should be a primary outcome measured as part of MIHOPE. Nurse-Family Partnership recommends that MIHOPE include spacing between births as a primary outcome measured with reviews of birth certificates, the outcomes of those subsequent pregnancies (spontaneous and therapeutic abortions, low birth weight and preterm newborns) using both interviews and birth certificate data, and between index and subsequent pregnancies measured with interviews.

Childhood Injuries and Child Abuse and Neglect: Injuries are the leading cause of mortality among children age 1 and up; and injuries experienced in the first year of life are frequently caused by child abuse and neglect. A recent report released by the Centers for Disease Control and Prevention estimated the total lifetime costs associated with one year of confirmed cases of child abuse and neglect at $124 billion.2 Nurse-Family Partnership’s clinical trials demonstrate significant reductions in state-verified reports of child abuse and neglect in its first trial (where county-wide rates of reported and substantiated cases of maltreatment were the highest in New York State) as well as reductions in injuries revealed in the medical records of children in both the first and second trials of the Nurse-Family Partnership. This outcome of public health importance should be a primary outcome measured in MIHOPE through medical reports.

Based on decades of experience, Nurse-Family Partnership strongly cautions against using maternal self-reporting of outcomes in this domain and Child Protective Services (CPS) records, which are fraught with surveillance bias. This problem is compounded by the fact that only a small fraction of real maltreatment is identified through the formal CPS system and detection in CPS records varies by states and localities. If the evaluators employ CPS records, they should have confidence that they can rule out two major sources of surveillance bias: 1) the home visitors themselves, who are mandated reporters; and 2) other family members who, at least in the Nurse-Family Partnership and probably other programs, are systematically drawn into the program and whose commitment to the mother and child are increased as a result of the Nurse-Family Partnership intervention, and who are likely to report parents when abuse or neglect are suspected. The involvement of other family members and friends in support of the mother and child are explicit objectives of the Nurse-Family Partnership. Nurse-Family Partnership’s experience from its Elmira trial is that nurse-visited children were identified at lower thresholds of severity for maltreatment than were children identified as maltreated in the control group, and we have extensive anecdotal reports of nurse-visited family members reporting mothers for maltreatment when they observed them neglecting or abusing their children. CPS records are crude instruments for measuring something so important. We strongly urge careful consideration of this outcome measure.

Language Development: If additional resources are secured for longer-term follow-up, we strongly recommend that MIHOPE conduct direct assessments of children’s language development, which is a major predictor of school success. Clinical trials of the Nurse-Family Partnership demonstrated improvements in children’s language development among those born to mothers with low psychological resources. We recommend that this outcome be included in MIHOPE and reliably measured with the Preschool Language Scales at age 2, or other well validated direct assessments.

II. MIHOPE Should be Modified to Hypothesize and Examine Moderated Program Effects

The current design of MIHOPE, which includes an evaluation of numerous outcomes with limited sample sizes, does not permit an assessment of effects that prior research of the Nurse-Family Partnership predicts will be found only in important sub-populations of children and families. For example, clinical trials of the Nurse-Family Partnership demonstrate impacts on birth weight and length of gestation were concentrated on infants born to women who smoked cigarettes at program registration. Likewise, Nurse-Family Partnership’s replicated outcomes on reductions in injuries and improvements in language development were concentrated among children born to women who had low psychological resources, that is, those mothers with higher rates of depression and anxiety, and lower levels of intellectual functioning and mastery. The current design of MIHOPE cannot detect these important findings because of insufficient sample size to detect such moderated effects and because the design of the evaluation does not make specific hypotheses about moderated program impact. As a consequence, Nurse-Family Partnership’s moderated effects will not be detected by the evaluation, leading some to conclude that the effects from the trials were not replicated. The impact of the Nurse-Family Partnership should be estimated with the specific populations found to benefit from the original Nurse-Family Partnership trials, particularly given that children and families with significant risk factors are the very populations most in need of home visiting services. MIHOPE’s exclusion of a focus on specifics about conditional effects undermines the scientific and policy relevance of the investigation.

For these reasons, we recommend that the overall number of outcomes evaluated be reduced and that the evaluation focus on those of primary public health, educational and economic importance, and the sample size of the evaluation be significantly increased to allow for a thorough, reliable analysis of the primary outcomes and hypothesized moderated effects.

III. The Scope of Implementation Study and Corresponding Data Collection Should be Significantly Reduced to Save Resources to Focus on Outcomes of Primary Importance and to Reduce Burden on Home Visitors and Families

A. Implementation Study is Overly Broad and Unduly Burdensome and Should be Reduced to Save Resources for a Valid Measurement of Primary Outcomes

Although Nurse-Family Partnership agrees with the importance of studying factors that influence the delivery of home visiting services, we are concerned that the implementation study as proposed will require significantly more time and resources on the part of nurses and administrators than is currently reflected in the burden estimates. We believe that an over-emphasis on the implementation study will reduce resources for examining whether the outcomes attributable to the original home visiting models were replicated in the MIECHV Program. Compared to the substantial literature on maternal and child health outcomes affected by the NFP, there is little prior research to justify or guide such a broad scope of inquiry on program implementation. We therefore recommend that you curtail the implementation study in order to focus resources on valid measurement of the primary outcomes attributable to home visiting models as outlined in Section I of this correspondence.

We recommend that the implementation study focus on the following questions:

Do the home visitors and supervisors employed meet the basic qualifications for their roles as described by the program model being implemented?

Have all of the home visitors and supervisors in the program met the education/training requirements of the model they are implementing?

Is the population enrolled consistent with the target population identified by the model as appropriate and likely to benefit based on previous scientific findings?

Were the program-engagement, completed-visit, and attrition rates for each program better, the same, or worse than the parameters set by each program model for implementation excellence?

Was reflective supervision provided to home visitors, and was it provided at the frequency recommended by the model?

Are there organizational policies in place to support implementation of the model(s)?

B. The Scope of the Implementation Study Unduly Burdens Home Visitors and Families

Based on our review of the instrument tools, the burden estimates for the home visitors, program administrators and participating families appears to be drastically underestimated by about 50%, raising serious concerns about the impact of the data collection on the home visitor’s ability to deliver services, as well as the families willingness to participate. We also are concerned with the substance of certain questions and the prosecutorial style in which some appear to be presented. We offer detailed comments regarding the questionnaires in the attached Appendix I to this correspondence.

C. Questions Regarding the Mental State of Home Visitors in the Implementation Study Raise Ethical and Legal Concerns

The MIECE Annual Survey of Program Site Home Visitors examines the mental state of home visitors to measure the quality of home visiting services. The questionnaire includes questions, such as the following:

“If you’ve got a job to do, you should do it no matter who gets hurt.”

“I wonder why people would want to be involved with me.”

“I wonder how I would cope without someone to love me.”

“Sometimes I think that I am no good at all.”

These types of questions typify the exploratory nature of much of the implementation evaluation, which does not begin to approach the level of scientific foundation found in the primary outcomes identified in Section I of this Comment. Further, this type of inquiry raises the following questions:

(1) What is the impact of making the visitors subjects of research on their ability to perform the home-visiting roles?

(2) To what extent will they consent to this line of inquiry?

(3) To what degree will home visitors feel coerced into consenting as a condition of employment?

(4) Are employing agencies subject to liability for mental/psychological testing of home visitors under applicable state and federal law?

(5) What is the cost of this line of inquiry and how does it relate to reduced resources in the outcome evaluation for adequate sample size and deep measurement of primary outcomes?

We believe that these issues can be avoided by significantly scaling back the scope of the implementation study as recommended above.

IV. Baseline Surveys, Questionnaires and Interviews are Missing Information

Finally, we note that certain information and questionnaires appear to be missing from the information provided to respond to this Comment Request, including Section D, Organizational Social Context Measure in the MIECE Annual Survey of Program Site Home Visitors, as well as in the MIECE Annual Survey of Home Visiting Site Supervisors. We look forward to receiving the missing information.

V. Conclusion

We appreciate the opportunity to offer recommendations. We are also grateful for the Department’s leadership and willingness to engage us in meaningful dialogue regarding the implementation and evaluation of this important program. Together, we are making progress toward the successful evaluation of the MIECHV Program. We look forward to discussing these issues with you in greater detail.

Sincerely,

Thomas R. Jenkins, Jr.

President & CEO

Cc: Terry Adirim, Office of Special Health Affairs, Health Resources and Services Administration

Mayra Alvarez, Director of Public Health Policy, HHS Office of Health Reform

Martha Coven, Associate Director for Education, Income Maintenance, and Labor, OMB

Sherry Glied, Assistant Secretary for Planning and Evaluation

Joan Lombardi, Deputy Assistant Secretary & Inter-Department Liaison for Early Childhood Development, Administration for Children and Families

Michael Lu, Associate Administrator for Maternal and Child Health, Health Resources and Services Administration

Cindy Mann, Director, Center for Medicaid and State Operations, Centers for Medicare and Medicaid Services

Bryan Samuels, Commissioner, Administration on Children, Youth and Families

Mary Wakefield, Administrator, Health Resources and Services Administration

Appendix I.

Review of Proposed MIECE Data Collection Instruments

MIECE Family Baseline Survey

General comments

Numerous questions are missing from the document and only referenced by topic. This prevents full disclosure for the reviewer. For a survey of more than 390 questions, the estimate of 1.0 hour for average burden per respondent appears to be a substantial underestimation.

The survey collects much data that would easily be available from the basic information already collected by each of the models. Beyond issues of unnecessary burden, the intrusiveness of a number of questions also warrants great concern. In this regard, these pose threats to the establishment of the trusting relationship between the home visitor and the client, as well as obvious challenges to the validity of responses that are given to a relative stranger concerning such issues as drug use, child abuse, criminal history, parenthood with multiple partners, and citizenship.

Specific comments

P 14. There are far too many questions on drug use, resulting in a tone that suggests a legal prosecution. Many clients will find this insulting.

P 18. Prosecutor-like questions concerning arrests, convictions, and incarcerations of the client and child’s father are demeaning to the client at a critical period in the formation of the client-home visitor relationship.

P 21. Asking a client if she has “ever worked for one employer for six consecutive months?” will be seen by many as implying that clients in these programs would not normally be up to doing so. The phrasing is potentially demeaning.

P 27. Asking how many children the client and the father have had with other partners is extremely insensitive. To many it will be seen as insulting and inflammatory.

P 28. In the current political climate, asking the client if she is a citizen seems both unnecessary and catering to a hot-button issue.

P 29. Questions concerning health insurance should be condensed.

P 30-31. There are far too many questions concerning service use. These should be condensed to focus on the most common types.

MIECE Self-completed Questionnaires by Parents during Selected Home Visits

General comments

The survey consists of 38 questions and is estimated as requiring 0.2 hours to complete. This estimate of burden appears to be accurate. The general nature of these questions appears to be appropriate. However, the use of videotaping during home visits raises concern regarding the client’s privacy.

Specific comments

P 3. Of the eight items under question 5, the first six are negatively phrased. These questions should be better mixed in positive and negative terms, in order prevent the establishment of a biased perspective.

MIECE Annual Survey of Program Site Home Visitors

General comments

A substantial section consisting of over 100 questions is missing from the document and only referenced by title and topic. This prevents full disclosure for the reviewer. For a survey of more than 600 questions, the estimate of 1.25 hours for average burden per respondent appears to be greatly underestimated. The overall burden of this survey needs to be significantly reduced.

Specific comments

P 7-10. Two instruments, consisting of 45 questions on psychological well-being is excessive.

P 11. This section represents 105 questions that are missing from the survey. This is a substantial omission that clearly affects the reviewer’s ability to fairly judge the overall burden imposed by the survey.

P 38. One can only imagine the respondent fatigue developed by the time Section P is reached. To expect the respondent to thoughtfully analyze a video clip at this point reflects a lack of such awareness.

MIECE Self-completed Questionnaires by Home Visitors During Selected Home Visits

General comments

The survey consists of 18 questions and is estimated as requiring 0.2 hours to complete. This estimate of burden appears to be accurate. The general nature of these questions appears to be appropriate.

Specific comments

P 2-3. Reference to “caregivers” may be confusing, depending on the circumstances at the time of the home visit. It would be more appropriate to refer to these individuals as clients.

P 2. A 7-pt Likert scale implies more discrimination than what is reasonably warranted. This should be reduced to no more than a 5-pt scale.

Log Maintained by Home Visitors on Service Delivery

General comments

This log is not fully described, but appears to require responses to over 100 items. Further, these logs are expected to be completed each week on each client. The average burden is estimated as 0.5 hours. However, when considered across the home visitor’s caseload, this burden becomes very substantial, and particularly so when one considers that the home visitor may also be completing similar reporting as part of normal duties required by the model. Given these concerns, the very substantial and unnecessarily duplicative burden imposed by this log requires some very thoughtful reconsideration.

Questionnaire for Participants in Home Visitor and Home Visitor Supervisor Interviews

General comments

Nine questions (primarily demographic) are asked No concerns are raised in regards to these.

Draft Protocol for Group and One-to-One Interviews with Home Visitors

General comments

Over 140 questions are asked for an interview protocol that is estimated to require 75 minutes for completion. Clearly this underestimates the length of thoughtful responses to these questions. Questions concerning the length of home visits, the nature of initial training, the nature of meetings with supervisors, and the types of data that are collected would be available from other sources and are therefore redundant.

MIECE Annual Survey of Home Visiting Program Site Supervisors

General comments

A substantial section consisting of over 100 questions is missing from the document and only referenced by title and topic. This prevents full disclosure for the reviewer. For a survey of approximately 500 questions, the estimate of 1.25 hours for average burden per respondent appears to be underestimated. The overall burden of this survey needs to be significantly reduced.

Specific comments

Section C. Two instruments, consisting of 45 questions on psychological well-being is excessive.

Section D. This section represents 105 questions that are missing from the survey. This is a substantial omission that clearly affects the reviewer’s ability to fairly judge the overall burden imposed by the survey.

Log Maintained by Supervisors on Supervisory Activities

General comments

This log requires responses to over 100 items and it is expected to be completed each week on each home visitor. The average burden is estimated as 0.5 hours. Depending on the number of home visitors being supervised, this may become burdensome and particularly so when one considers that the supervisor may also be completing similar reporting as part of normal duties required by the model and/or the implementing agency Given these concerns, the duplicative burden imposed by this log may require some reconsideration.

Draft Protocol for Group and One-to-One Interviews with Home Visiting Supervisors

General comments

Over 130 questions are asked for an interview protocol that is estimated to require 75 minutes for completion. Clearly this underestimates the length of thoughtful responses to these questions. Questions concerning the nature of initial training, the nature of supervisory meetings with home visitors, and the types of data that are collected would be available from other sources and are therefore redundant.

MIECE Annual Surveys of Home Visiting Program Site Managers

General comments

This survey requires the development of 40 or more documented responses, as well as answers to more than 170 additional questions. To assume that this would be completed within the estimate of 3.0 hours would likely be far below what would actually be required. This is an exceptionally burdensome instrument. The content and structure of this survey requires a full reconsideration to focus only on essential information collected in the most expeditious manner from the respondent.

Specific comments

Survey #1, Sections D-E. The substance of these personnel characteristics could easily be restructured into checklists that provide the essential information.

Survey #2, Sections C-E. These lists of variables for which responses are given will be far too long. Beyond the most essential items in each table, the validity of responses will come into question as clients consider the supposed priorities of multiple entities across an increasingly daunting number of variables.

Survey #2, Section V. This becomes a very tedious section that should be condensed and restructured into a short series of checkboxes for each service, providing only the most essential information.

Draft Protocol for Interviews with Program Managers

General comments

Over 90 questions are asked for an interview protocol that is estimated to require 75 minutes for completion. Again, as with other estimates for interviews, this underestimates the length of thoughtful responses to these questions. Questions concerning the nature of program goals, the availability of community services, the presence of other home visiting programs, and data collection procedures are available from other sources and are therefore redundant.

MIECE Annual Semi-structured Interview with State MIECHV Administrators

General comments

90 questions are asked for an interview protocol that is estimated to require 2.0 hours for completion. This estimate of burden may be accurate, but it leaves little time for detailed answers. The overall focus of the questions appears to be highly appropriate.

MIECE Annual Surveys of Administrators of Community Resources that Provide Services Relevant to Home Visited Families

1. Survey of Community Service Providers Nominated by Home Visiting Program

General comments

17 questions are asked that provide a brief and appropriate overview of the service provider.

2. Survey of Other Home Visiting Programs Nominated by Home Visiting Program

General comments

10 questions are asked that provide a brief and appropriate overview of the service provider.

To Whom It May Concern:

The Texas Health and Human Services Commission (HHSC) appreciates the opportunity to comment on the proposed information collection activity for Maternal, Infant and Early

Childhood Home Visiting Evaluation (MIECE), Baseline Survey Data Collection as requested in the Federal Register, Vol. 76, No. 238, issued on December 12, 2011 (file code 4184–22–M). The Federal Register indicates that the purpose of the national evaluation is to inform the federal government about the effectiveness of the Maternal, Infant and Early Childhood Home Visiting (MIECHV) program and to provide information to states to help them develop and strengthen home visiting programs in the future.

HHSC feels that the number of questions asked in these surveys and interviews is very excessive.

Many of the questions listed on the proposed survey and interview forms seek information that exceeds the stated goals. Without a significant reduction in the size of and amount of detail required in the interviews and surveys, HHSC is very concerned that this evaluation will place a heavy burden on the State, contractors, and local communities and will be impossible to administer.

HHSC also believes that because of the length of the proposed surveys/interview questionnaires and the number of each to be conducted, administering them will significantly burden States, home visiting contractors, and clients. Consequently, administration may require additional resources. Although Texas’ home visitation programs are funded through federal grants, the grant funding is already allocated and maximized. Changes to the home visitation program’s budget for additional costs, would require federal approval. Moreover, HHSC is concerned that additional resources will not be available in the current fiscal environment for the proposed MIECE. HHSC is concerned specifically that:

 Current capacity likely is insufficient to complete the proposed surveys/interviews, which may require increasing contract hours with home visiting contractors to complete the proposed home visitation surveys and interviews.

 Costs may be incurred for translating/interpreting new client survey materials.

 Incentives may be needed to motivate clients to complete the proposed surveys/interviews.

 Any survey made available online only would require resources for completing the survey by phone or in a face-to-face meeting, since not all clients have access to computers with Internet access.

 Because home visiting programs are already required to collect data for over 30 benchmark measures, as a condition of receiving federal grant funding, it may be difficult to collect data through the surveys and interviews in addition to the benchmark measures.

HHSC urges the Administration for Children and Families (ACF) to reconsider the size and the number of questions in the proposed interviews and surveys. In its current form, this evaluation will likely jeopardize the success of the MIECHV program.

To alleviate the burden and allow States to work within allotted resources while still accomplishing the stated goals, Texas recommends that ACF reduce state burden by changing the proposed MIECE in the following ways:

 Eliminate the duplicative questions in the proposed surveys/interview questionnaires.

 Collect the requested information through alternative data sources rather than through multiple surveys and interviews. For example, the MIECHV program grant administrator already collects certain information requested in proposed survey/interview questions.

 Examine how evidence-based home visiting program models may already collect supervision and training information in order to alleviate a portion of the data collection burden through the proposed surveys/interviews.

 Reduce the number of questions in the proposed surveys/interview by focusing only on questions that meet the evaluation goals.

 Clarify who should complete each survey, as the roles of Site Manager and Home Visiting Supervisor may be filled by the same person in certain home visiting programs.

The following includes Texas’ comments, suggestions, and questions regarding each of the specific proposed surveys and interview documents.

A. Maternal, Infant and Early Childhood Home Visiting Evaluation (MIECE) Family Baseline

Survey

HHSC has concerns about the Family Baseline Survey generally, as well as concerns about specific survey questions.

HHSC generally is concerned that:

 HHS underestimates the amount of time necessary to complete a 400 item survey. In

HHSC’s view, a client would need approximately five hours to complete the entire survey.

 The reading level for many of the survey questions is not appropriate for the at-risk population that home visiting programs serve. Texas recommends lowering the reading level.

 Texas recommends defining the term “focal child.”

 The baseline survey questions primarily focus on mothers with children under 24 months and do not identify when participants enter the program. Because the MIECHV program grant allows for multiple program models, some participants may enter a home visiting program serving prenatal clients (e.g., Nurse-Family Partnership) or a home visiting program serving children ages three through five (e.g., Home Instruction for Parents of Preschool Youngsters). It is unclear whether the program participants with children ages three through five are excluded from the baseline survey.

 The survey format includes skip logic, which may be confusing for clients who take the survey with paper and pencil.

 If the MIECE family baseline survey is an online survey, clients will need to be provided computers with Internet access and support on using a computer-based survey. Requiring States to provide clients with Internet access and support for online survey completion may require additional resources including increased costs for program contractors.

 The MIECE family baseline survey does not address whether a caregiver other than the child’s mother is required to participate in the survey. Most of the questions are focused on mothers and are written with that focus by, for example, using the pronoun “she.” Texas recommends that the survey clarify how other caregivers (e.g., father, grandmother, foster parent) should respond to the survey.

 The survey does not ask about the current age of the child, which home visiting program the children are enrolled in, or the age of the child when he/she enrolled in the home visiting program.

 Many of the questions are sensitive. Texas recommends inserting assurances

guaranteeing families’ anonymity or confidentiality.

 The survey does not allow a client to respond “not applicable” or to decline to respond.

Texas recommends that the survey include response options for questions that are “not

applicable” or to permit a client to choose not to respond.

With respect to specific questions, Texas has the following concerns and recommendations:

1. Domain: Child Health and Development, Construct: Newborn Health-Birth Weight

 HHSC recommends omitting the second question, “Was child’s birth weight…” The

first question will allow the analyst to determine if the weight was in the normal, low,

or very low range.

2. Domain: Parenting, Construct: Child Maltreatment

 This question requests information that respondents may consider private or difficult

to answer without repercussions. Texas recommends adding options to allow clients

to refuse to answer.

3. Domain: Parenting, Construct: Fathering-Relationship Happiness

 Texas recommends revising the question to clarify that the questions refer to a

romantic relationship.

 It is not clear that the respondent is to discuss a relationship with an adult, nor is it

clear if the survey wants a response only if it is the child’s biological father.

 The survey does not ask whether the respondent is currently in a relationship. Texas

recommends revising the survey to ask the respondent to whom the respondent refers

in this answer.

 The seven-point scale may provide too many options to the respondent; therefore, a

five-point scale may be less confusing.

4. Domain: Parenting, Construct: Fathering-Paternity, Father Involvement

 Texas recommends separating “every day” and “almost every day” into two response

choices.

5. Domain: Parenting, Construct: Maternal Health-Physical Health/Illness

 As written, the survey references to items drawn from the SF-12 Health Survey

Scoring Demonstration. Texas cannot comment on these items because they are not

included.

6. Domain: Parenting, Construct: Attachment Style

 Too many of the questions asked in this survey appear duplicative or to require

distinctions too fine to be necessary. Texas recommends abbreviating the survey to

eliminate the duplications and near-duplications.

7. Domain: Parenting, Construct: Maternal Substance Use-Substance Use

 Texas recommends clarifying the question to clarify that it is addressing prescription

drugs, illicit drugs, or both.

8. Domain: Parenting, Construct: Social Support

4

 HHSC recommends focusing the survey on the degree to which the respondent

receives a given type of social support, rather than the names of individuals providing

support. The degree of social support would allow for a comparison across the two

time periods, whereas names do not reveal the degree of support received and would

make an analysis difficult.

9. Domain: Intimate Partner Violence, Construct: Intimate Partner Violence-Psychological,

Emotional Violence

 The questions use the past tense, but do not define the time period about which the

respondent is being asked. Similarly, the questions do not specify whether the

respondent should consider only a current partner, if the respondent has one, or a

previous partner. HHSC recommends clarifying both of these points.

10. Domain: Family Self Sufficiency

 Texas recommends amending this section to request details on the number of income

earners in the household or whether the income earners share their resources with the

focal child to gain an understanding of the resources that are available to the child.

11. Domain: Family Self Sufficiency, Construct: Maternal and Paternal Employment

o The construct is too detailed and many questions may not be relevant to the goals

of the evaluation.

o Texas recommends eliminating the following questions: “For whom do you

usually work the most hours?” and wage questions, as they are covered in the

section “Maternal and paternal employment.”

o Texas recommends combining these questions: “Would you describe your job

as…” and “How many hours per week do you usually work at this job?”

12. Domain: Family Self Sufficiency, Construct: Maternal and Parental Employment-

Earnings

 For clarity, Texas recommends that the question specify “total annual earnings”

instead of “total earnings.”

13. Domain: Family Self Sufficiency, Construct: Education

 Texas recommends adding a response choice (with respect to both maternal and

paternal education) for some college, but no degree completion.

 Texas recommends clarifying the question about the “father’s educational attainment”

to specify the “child’s father’s educational attainment.” HHSC suggests replacing the

word “attainment” with a less formal word to lower the reading level.

14. Domain: Family Self Sufficiency, Construct: Public Assistance

 Combine with Income because both questions focus on benefits and support from

non-employment sources.

15. Domain: Family Self Sufficiency, Construct: The Assistance from Nonresidential Fathers

section

 To compare the amount of child support owed with the amount actually collected, the

survey also should ask for the amount of child support owed based on the legal

agreement.

16. Domain: Demographics

 Reorganizing the questions in a more logical order would help to spot duplicative

questions. For example, the survey asks multiple repetitive questions on whether the

respondent is Hispanic or Latino(a).

5

17. Domain: Demographics, Construct: Housing and Household Composition-Ages and

Relations of Other Members

 Asking the respondent to identify the number of children they have is unnecessary if

the respondent answered the same question previously in this survey.

18. Domain: Demographics, Construct: Housing and Household Composition-Mobility,

Languages Spoken at Home, Acculturation

 Texas recommends that the survey define the term “RAD.”

19. Domain: Demographics, Construct: Demographic of Index Child’s Parents-Other Living

Children

 These questions are repetitive of questions already asked. Texas recommends

deleting this domain.

20. Domain: Actual Services, Construct: Mother-Related Screenings, Referral, Coordination,

and Service Use

 The first section includes too many questions; Texas recommends selecting questions

that most closely align with the goals of the evaluation.

21. Domain: Actual Services, Construct: Social Services

 The question refers to AFDC, an outdated term. The AFDC references should be

changed to refer to Temporary Assistance for Needy Families (TANF).

 The questions discuss “Emergency Food,” which can come from a variety of sources.

Texas recommends that the question specify the food source of interest.

 The questions on alcohol and drug treatment are duplicative of questions in “Maternal

Substance Use- Alcohol Use.” Texas recommends deleting them.

B. MIECE Annual Semi-Structured Interview with State MIECHV Administrators

In general, Texas requests clarification on the goal of the “MIECE Annual Semi-Structure

Interview with State MIECHV Administrators” and whether the survey results will be tied to federal funding.

Texas also requests clarification on whether the survey questions are intended to change each year. Many of the questions posed, for example, do not make sense for an annual survey. For example, question 1 asks about the structure of the program, but program structure will not change from year to year. Consequently, the question does not need to be asked on an annual basis.

In addition, requiring state administrators to provide background information on the program is unnecessarily burdensome. Most of the background information requested in this survey can be found in State grant applications and does not need to be collected during a structured interview.

C. MIECE Annual Surveys of Home Visiting Program Site Managers

In general, the current format of this survey, including the tables and footnotes, are difficult to follow. Texas recommends changing this if the survey is put into a web-based format. Many of the survey questions are unnecessary. First, many of the questions deal with policies and procedures dictated by either the State MIECHV program or the evidence-based program model. By asking if the respondent has made any modifications to the evidence based program model, rather than asking about specific details, the survey could be streamlined and less burdensome. Second, questions related to training seek information already being captured in the site manager logs that are being completed as a requirement for home visitations programs. These questions should be deleted.

Texas’ specific concerns and recommendations regarding the MIECE Annual Survey of

Home Visiting Program Site Managers are below. Texas feels that making the following changes will provide more options for the responder and make the survey easier and faster to complete.

1. In survey #2, Section J

 The value of questions on hiring interviewing techniques is unclear in questions 3, 11,

12, 11 (the second one), and 12 (the second one).

2. In survey #2, Section M, Question 1

 Recommend changing question to read, “How often does your program site conduct

staff meetings?”

 Include skip logic if “No team meetings” is selected.

3. In survey #2, Section M, Question 2

 Provide an answer choice of “other.”

4. In survey #2, Section M, Question 4

 Recommend changing question to read, “How often do the meetings include

presentation by outside speakers?”

5. In survey #2, Section N, Question 1

 Recommend changing question to read, “How often does your program site conduct

case conferences?”

 Include skip logic if “No case conferences” is selected response.

6. In survey #2, Section N, Question 3

 Recommend changing question to read, “How often do the case conferences include

outside experts?”

D. MIECE Annual Survey of Home Visiting Program Site Supervisors and MIECE Annual

Survey of Program Site Home Visitors (Note: Texas’ comments are combined for both surveys due to the similarity of the two documents.)

The MIECE Annual Survey of Home Visiting Program Site Supervisors and the MIECE Annual Survey of Program Site Home Visitors are lengthy and complex in their current form. It appears that the goal of many of the survey questions does not align with the stated goals of this evaluation.

Texas’ specific concerns and recommendations regarding the MIECE Annual Survey of Home Visiting Program Site Supervisors and the MIECE Annual Survey of Program Site Home Visitors are below:

1. Section A, Question 1

7

 List all evidence-based home visiting models recognized by the Health Resources and

Services Administration (HRSA).

2. Section A, Question 3

 Recommend using census classifications for race and ethnicity.

3. Section B, Question 3

 Because home visitors typically do not track hours spent on specific tasks, such as

home visiting, manual paperwork, and data entry, it will be difficult to indicate the

number of hours worked in a typical week on each listed topic.

 Recommend using a percentage of time instead of a percentage of hours.

4. Section B, Question 10

 Does not appear to be appropriate for this survey and may not yield accurate results.

5. Section B, Question 11

 Recommend deleting because the question is identical to Question 10.

6. Section B, Question 12

 Recommend providing a range for response to make answering the questions easier

for respondents.

7. Section B, Question 12b

 Because most training provides a range for time that was spent on didactic/lecture,

demonstration by trainer, discussion and or question/answer, and/or role

playing/practice/behavioral rehearsal, this question may be difficult to answer.

Moreover, the question’s relevance to the evaluation’s goals is not evident. Texas

recommends deleting the question.

8. Section B, Question 14

 Recommend deleting question 14 if Question 15 is left in the survey.

9. Section B, Question 16

 Define “home visit plan” and include skip logic if home visit plans are not used.

10. Section B, Questions 20 and 21

 Recommend including a timeframe for these questions, e.g., “Have you ever?” or

“How often do you?”

11. Section C

 Questions in this section appear to exceed the scope of the stated evaluation goals and

raise privacy concerns. Texas recommends deleting Section C.

12. Section D

 Recommend publishing this section so states can review and comment.

13. Section E

 Recommend deleting Section E because the content is addressed in the program manager survey.

14. Section F

 Texas believes Section F should be included in the program manager survey and not

completed by the home visitors. Because these questions more broadly address the

MIECHV program, program managers are more likely to know this information, and

this will help reduce the burden on home visitors.

15. Sections G through N

 Because the qualifier words “excellent” and “highly” will lead to false negative responses, Texas recommends removing these words. The questions should read, “I have had training to carry out this activity” and “I am skilled in carrying out this activity.”

 Recommend amending the skip logic for page ii of Section G through N because the

skip logic leads respondents to questions that do not exist (e.g., Question 6 and

Question 7).

16. Section J

 Acronyms should be spelled out. For example, the survey should define “MH” as

“mental health.”

17. Section K

 Acronyms should be spelled out. For example, the survey should define “DV as

“domestic violence.”

18. Section O

 Recommend deleting Section O for two reasons:

o First, the section does not apply to all home visiting programs, such as home visiting programs serving older children, because the questions address care for babies.

o Second, Section O reads more like a test rather than a survey and may be stress inducing for visitors.

19. Section P

 Recommend deleting Section P for three reasons:

o First, the purpose of the section is unclear, and therefore, it seems irrelevant to the stated evaluation goals.

o Second, it may be difficult to play the two-minute video if participants do not have the technical capability. It may be easier to read a scenario to the participant rather than asking them to watch the video.

o Third, the section reads more like a test than a survey and may be stress inducing for visitors.

D. MIECE Annual Surveys of Administrators of Community Resources that Provide Services

Relevant to Home Visited Families

Texas suggests the following changes:

1. Question 3

 Rephrase the question to clarify whether the question is asking for referrals into home

visiting programs or referrals out of home visiting programs.

2. Question 9

 Recommend including options for “None” and “Other” to give the responder a more

complete range of options.

E. Log Maintained by Supervisors on Supervisory Activities

Texas has concerns regarding the log maintained by Supervisors on Supervisory Activities.

Specifically, filling out the proposed log may be time consuming for supervisors who typically oversee 12 or more visitors. To reduce the burden on supervisors, it may be more appropriate for the log information to be recorded once or twice a month rather than recording this detailed information weekly.

To simplify the proposed individual supervision chart, Texas recommends allowing the

supervisor to indicate the number of visitors who received each type of supervision topic, instead of completing the individual supervision chart for each individual visitor.

Additionally, if training information is collected in the log maintained by supervisors on

supervisory activities, then the same information should not be collected in the other proposed MIECHV evaluation surveys because the log would provide the most accurate information.

F. Log Maintained by Home Visitors on Service Delivery

The proposed log maintained by home visitors on service delivery also appears to be burdensome and time-consuming for home visitors. For example, a home visitor may see 12 families in one week and spend approximately five to ten minutes filling out the proposed home visitors service delivery log for each family. Cumulatively, this would require approximately 1 to 2 hours to complete each week. Therefore, adding new categories to the log will increase the time necessary to complete the log. Texas thus recommends adding sections to the home visitors log only if the information that is being collected directly relates to the evaluation goals. If training information is collected in the log maintained by home visitors, then the same information should not be collected in the other proposed MIECHV evaluation surveys because the log would provide the most accurate information.

Texas has the following additional specific concerns and suggestions regarding the log

maintained by home visitors on service delivery:

1. Question 2G in the “Contact with Client”

 Recommend adding a response option for “None.”

2. Table 1

 Will the visitor be able to check multiple boxes?

G. MIECE Self-completed Questionnaires by Parents during Selected Home Visits

Texas believes that HHS has underestimated the amount of time a family will need to complete the MIECE Self-completed questionnaire during selected home visits because the reading level on the scale is higher than it should be for at-risk parents/caregivers. Texas estimates that it will take closer to 30 minutes for a family to complete the questionnaires, rather than 12 minutes as stated in the Federal Register notice. Overall, Texas accordingly recommends that the reading level for the scale be lowered.

Texas has the following additional specific concerns and suggestions:

1. Section A, Question 1 asks how well the program has done, but the scale is based on

expectations.

10

 Measuring program performance based on expectations does not seem logical

because parent expectations may not align with program activities or goals (e.g., the

parent expected job training, but the home visiting programs do not provide that).

This section is more a measure of how well programs communicated their services

rather than how well they are performing. Texas recommends that the question be

revised either with pre-populated responses based on the goals of home visiting

programs or that the interviewer select just those expectations provided by the parent

during the baseline survey that align with the goals of home visiting programs.

 The scale for expectations currently includes two negative responses, one neutral

response, and one positive response. Texas recommends balancing the scale for

expectations to include the same number of negative, neutral, and positive responses.

2. Section B, Question 2

 Recommend including a response for “Never.”

3. Section D, Question 9

 Recommend using the same scale used in the other questions (Never, Sometimes,

Often, Always) rather than including the response “Often.” Using the same scale will

reduce confusion.

4. Section F

 Recommend using a five-point scale, instead of the seven-point scale, because the

seven point scale may provide too much variability in responses.

 Only two items (Questions 19 and 25) are negatively worded, which may lead to a

high error rate. Recommend changing wording of questions 19 and 25 to a positive

format for consistency.

H. MIECE Self-completed Questionnaires by Home Visitors during Selected Home Visits/

Home Visitor Questionnaire at Videotaped Visits

The title of this survey suggests that home visits will be videotaped. Texas requests clarification

on the relevance of videotaped visits to the stated evaluation goals.

Additionally, the questionnaire language in this survey is focused on a female caregiver and

needs to be changed to accommodate for male caregivers. For example, Question 12 states, “The

caregiver believes the way we are working toward her goals is correct.”

Texas has the following additional concerns and suggestions:

1. Table 1 would be more effective with a five-point scale; the seven-point scale may give

too much variability.

 Only two items (Questions 4 and 10) are negatively worded, which may lead to a high

error rate. Either more items should be worded negatively or those items should be

worded positively.

2. Question 15

 The response “often” may confuse the respondent. Texas recommends instead using

the scale “Never, Sometimes, Often, Always” to provide consistency with other

questions. Changing the scale also would eliminate the need for Question 14, which

asks “Do you ever have trouble understanding the caregiver?”

11

3. Question 17

 The response “often” may confuse the respondent. Texas recommends instead using

the scale “Never, Sometimes, Often, Always” to provide consistency with other

questions. Changing the scale also would eliminate the need for Question 16, which

asks “Do you think that the caregiver ever has trouble understanding you?”

I. Questionnaire for Participants in Home Visitor and Home Visitor Supervisor Interviews

1. Recommend using census classifications for race or ethnicities in this survey.

J. Draft Protocol for Interviews with Program Managers

1. It is unclear how information on the program costs needs is relevant to the stated

evaluation goals.

2. Recommend addressing questions 15, 16, and 17 in a survey or interview with the State

MIECHV administrator because the administrator is more likely to have knowledge of

state activities.

K. Draft Protocol for Group and One-on-One Interviews with Home Visitors

Texas recommends eliminating the draft protocol for group and one-on-one interviews with

home visitors because all relevant questions are addressed in the other proposed MIECHV

evaluation tools. Questions could be added to the other MIECHV evaluation tools only if they

are necessary to achieve the stated evaluation goals.

L. Draft Protocol for Group and One-on-One Interviews with Home Visiting Supervisors

Texas recommends eliminating the draft protocol for group and one-on-one interviews with

home visiting supervisors because all relevant questions are addressed in the other proposed

MIECHV evaluation tools. Questions could be added to the other MIECHV evaluation tools if

they are necessary to achieve the stated evaluation goals.

Texas appreciates the opportunity to provide comments on the proposed information collection

activity for MIECE, Baseline Survey Data Collection.