Project LAUNCH Cross-Site Evaluation
Supporting Statement Part A
For OMB Approval
The Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services is requesting Office of Management and Budget (OMB) clearance for instruments to be used in data collection for a cross-site evaluation of a new initiative called Project LAUNCH (Linking Actions for Unmet Needs in Children’s Health). The Substance Abuse and Mental Health Services Administration (SAMHSA) awarded six five-year cooperative agreements under Project LAUNCH in September 2008 and anticipates awarding 12 additional cooperative agreements in September 2009. Through an intra-agency agreement between SAMHSA and ACF, the Office of Planning, Research and Evaluation within ACF is overseeing the development and implementation of a cross-site evaluation of Project LAUNCH. The current submission requests clearance for a three-year period of data collection from 18 Project LAUNCH grantees. ACF will submit an extension request for the continuation of evaluation activities during the final years of the grants.
The purpose of Project LAUNCH is to promote healthy development and wellness in children birth to 8 years of age. Project LAUNCH is intended to address issues in the child service system by enhancing systems coordination, integrating child behavioral health services with other health services, and incorporating evidence-based programs to address children’s healthy development. Each Project LAUNCH grantee is expected to focus on systems-level development at the State or Tribal level and in one designated community. Additionally, within the designated or “local” community, grantees are also required to implement promotion and prevention activities within each of the five categories: 1) mental health consultation; 2) developmental assessments across service settings; 3) integration of behavioral health into primary care; 4) family strengthening and parenting skills training; and, 5) home visitation.
Project LAUNCH is authorized under Section 520A of the Public Health Service Act (42 U.S.C. 290bb–32) and addresses Healthy People 2010 focus area 18 (Mental Health and Mental Disorders). In order to provide SAMHSA, Congress, and the public with information on how this new and comprehensive initiative is implemented and on its outcomes, ACF has contracted with Abt Associates Inc. to conduct a cross-site evaluation. The cross-site evaluation will collect information from Project LAUNCH grantees related to State, Tribal, and local systems development; implementation of evidence-based services in local communities; and service system outcomes for children, families and providers.
The cross-site evaluation will describe how Project LAUNCH efforts affected State and Tribal policies pertaining to children and families, led to expansion of existing services or implementation of new services, and increased service coordination. Although this is not an impact evaluation, findings will also document outcomes for children, families, and providers touched by Project LAUNCH.
Data collection methods for the cross-site evaluation will consist of annual site visits to Project LAUNCH grantees and a web-based data reporting system related to systems development and services implementation and outcomes.
During annual site visits, evaluators will conduct interviews with Project LAUNCH administrators, collaborators and service providers in State and Tribal areas and local communities of focus. The site visits have two primary objectives. First, the cross-site evaluation will collect data about Project LAUNCH activities aimed at infrastructure development and systems change at both the State/Tribal and community levels. We will seek to understand how these activities are implemented and document system change outcomes. Second, the cross-site evaluation seeks to understand grantees’ progress in program implementation and service delivery. During site visits, members of the evaluation team will collect information about the implementation of services within each of the five service categories of interest to Project LAUNCH. Data collected during site visits will expand and aid in explaining data reported by grantees in the web-based data reporting system.
Through the web-based data collection system, grantees will be asked to report semi-annually on systems development (at the State, Tribal, and local levels) and on services (including implementation of services or trainings for children, families and providers; services delivered; demographics of children, families and providers receiving Project LAUNCH services; and changes in service delivery settings and provider practices).
Grantee-specific child and family outcomes will be reported by the grantees as part of their local evaluations as per SAMHSA grant requirements and therefore not calculated in the burden requested in this clearance. However, findings reported by grantees using experimental or quasi-experimental designs to assess outcomes of Project LAUNCH services may be utilized in reports from the cross-site evaluation.
Evaluation findings will provide ACF and SAMHSA with ongoing information about program operations, apparent strengths and challenges, and areas where grantees would benefit from technical assistance and support. Accordingly, the evaluation will help guide policy development and program improvement at all levels. Evaluation findings will be routinely shared with Project LAUNCH grantees so that they can learn from each other, and can benefit from innovative strategies they are using in their project designs as well as how they are overcoming challenges to reaching their goals. Finally, cross-site evaluation results will support and enrich program implementation among new cohorts of grantees if Project LAUNCH is expanded.
The proposed data collection will use a web-based data entry system, previously developed for the Center for Mental Health Services (Decision Support 2000+ platform), to collect information in a uniform manner across grantees. This is a NET-based suite of online applications that provides the capacity for rapidly producing secure data capture and project collaboration portals tailored to the needs of SAMHSA evaluation and grantee monitoring. It is user-friendly and facilitates accurate and expeditious data collection.
This on-line data capture system offers the following major features that will support both data collection and program management:
A performance measurement site with state-of-the-art data capture and upload/download features, real-time data quality reports, trend reports, and program comparisons; and
Project management tools providing technical assistance materials, helpdesk access, and a space for clients and project staff to collaborate on documents, share schedules, contact information, and project task lists.
To minimize burden in subsequent data reporting periods, information entered at the first data collection time point will be pre-loaded for each grantee. Grantees will designate individuals needing access to the system.
Because Project LAUNCH is a new initiative, there have not been any similar cross-site data collection efforts. Although SAMHSA requires grantees to conduct local evaluations of their Project LAUNCH programs, the cross-site evaluation is designed to provide a more comprehensive examination of Project LAUNCH activities and outcomes across the communities served. To reduce the possibility of duplication of efforts between local evaluations and the national evaluation, the cross-site evaluation is designed to build upon and complement local data collection plans specified by Project LAUNCH grantees in their grant applications.
No small businesses are impacted by the data collection for this project.
A.6 Consequences of Collecting the Information Less Frequently
To address the study’s research objectives about how Project LAUNCH is implemented over time and how service provision is affected, site visits with grantees will be conducted annually, while data about services provided, numbers of families served, system change efforts at the State, Tribal, and local levels, and service system outcomes for children, families and providers will be required semi-annually to correspond with SAMHSA’s semi-annual progress-reporting periods.
There are no special circumstances that might require deviation from the guidelines of 5 CFR 1320.5.
The first Federal Register notice for the Cross-Site Evaluation of Project LAUNCH was published in the Federal Register Vol. 74, No. 48, Page 10919 on March 13, 2009. During the notice and comment period, one public comment was received but did not relate to the instruments or data collection. A copy of the 60-day notice is included in Appendix A.
Many individuals and organizations, including the Project LAUNCH Expert Consultant Group, were contacted for advice on aspects of the evaluation design and data collection instruments. Their feedback was obtained through in-person meetings and telephone conversations. Members of the Project LAUNCH Expert Consultant Group are listed in Table A.1.
tABLE A.1
Membership of Project LAUNCH EXPERT CONSULTANT Group
Individual |
Affiliation |
Beulah Allen, MD |
National Indian Youth Leadership Project |
David M. Chavis, PhD |
Community Science |
Paul Florin, PhD |
University of Rhode Island |
Stephanie M. Jones, PhD |
Harvard University |
Milton Kotelchuck, PhD |
Boston University |
Deborah Leong, PhD |
Metropolitan State College of Denver |
Judith S. Palfrey, MD |
Harvard Medical School |
Michelle Christensen Sarche, PhD |
American Indian Alaska Native Programs |
Ruth E.K. Stein, MD |
Albert Einstein College of Medicine |
Joseph Trimble, PhD |
Western Washington University |
Abraham H. Wandersman, PhD |
University of South Carolina |
In addition to the Expert Consultant Group, guidance on the evaluation design was sought from Russell Glasgow, Ph.D., the developer of the evaluation framework that will be used to guide the cross-site evaluation. This framework, called RE-AIM, is described more fully in section A.16.
No payments or gifts will be provided to respondents as part of data collection. Grant applicants were informed that they would be expected to take part in a cross-site evaluation of the program when they applied for and received Project LAUNCH funding. Data about individual families and children who receive Project LAUNCH services will be reported by grantees in the aggregate. To the extent possible, the cross-site evaluation will incorporate data already collected through grantees’ local evaluations. Accordingly, it will not be necessary to provide individuals with separate compensation for participation in the cross-site evaluation.
For each of the data collection methods, procedures will be in place to provide assurance of privacy to respondents to the fullest extent of the law.
Annual site visits
During annual site visits to each grantee, we will conduct interviews with a variety of State, Tribal, and local project staff. Each respondent will be asked to sign a consent form to participant in an interview, and the consent form will explain the study procedures for assuring that the answers provided by the respondent will remain private. The consent form will explain that (a) participation in the interviews is voluntary and there are no penalties for refusing to participate at any time during the interviews; (b) the respondent can refuse to discuss any topic; (c) data will be stored in de-identified files and (d) no names will be used in any evaluation reports. At the time of the interviews, respondents will be asked to sign two copies of the consent form. One will be retained by the cross-site evaluator, and the second copy will be given to the respondent for his/her files. A copy of the consent form is provided in Appendix B.
Although we will not indicate the names and titles of individuals at Project LAUNCH sites who provide information during the annual site visits, it is possible that we will be drawing comparisons between sites and identifying grantees by name. We will inform all study participants about this during our consent process prior to initiating the interview.
Semi-annual web-based data reporting
The evaluation will collect information using an on-line data collection system: DS2000+. This system has passed internal audits and tests conducted by the Abt Associates Inc. Security Officer and have been reviewed on multiple occasions by SAMHSA IT staff members. The DS2000+ data collection and analysis modules include https certificate-based authentication and transaction encryption processes approved for HIPAA data transactions on Federal healthcare projects. The DS2000+ platform provides a modular permission system allowing data segregation by role, individual, and system object. Accordingly, it will not be possible for any individuals who are not part of the research team to access these data.
A large portion of the data grantees will report through the web-based reporting system is about service delivery and will involve data already being collected by grantees for their local evaluations. All information on services will be reported to the cross-site evaluation in the aggregate; thus no information collected can be linked to individual families or individual agencies. To the extent that the local evaluations involve collecting information on services provided to individual families and children, the cross-site evaluation team will work closely with local grantees and their regional and State Institutional Review Boards (IRBs) to ensure that the necessary IRB approvals are obtained, and that human subject protections are assured. No respondent identifiers will be contained in public use files made available from the cross-site evaluation.
The cross-site evaluation will be requesting information in the aggregate about the demographics of children and families served. Although these data will be reported to the cross-site evaluation only in the aggregate through the web-based reporting system, local program staff will be collecting it on an individual basis from parents as they are enrolled in services and upon completion or termination of services. As part of this data collection, local project staff will collect written consent from each parent for participation in the evaluation. The consent form will describe the procedures to assure privacy for the respondents, and the interviewers will be knowledgeable about privacy procedures and will be prepared to describe them in detail or to answer any related questions raised by respondents. Necessary consent forms and privacy protections will be developed and administered by each grantee. Each grantee will obtain IRB approval from their regional and/or State IRB, as well as the institutional IRB overseeing the local evaluation, and therefore will be subject to the terms and guidelines of their specific IRB. The cross-site evaluation team will provide technical assistance in drafting these documents and responding to IRB concerns, if requested by the grantee. Copies of the consent forms from individual families and children will be stored at the local sites, under lock and key.
The cross-site evaluation will be requesting information on providers in local sites who participated in Project LAUNCH-related activities. These include trainings, mental health consultations services and integration of behavioral health and primary care. Specific questions will ask about how these services have changed their practices in caring for children. Although these data will be reported to the cross-site evaluation only in the aggregate through the web-based reporting system, local program staff will be collecting it on an individual basis from providers as part of their local evaluations. Local project staff will collect written consent from each provider for participation in the evaluation. The consent form will describe the procedures to assure privacy for the respondents, and the interviewers will be knowledgeable about privacy procedures and will be prepared to describe them in detail or to answer any related questions raised by respondents. Necessary consent forms and privacy protections will be developed and administered by each grantee. Each grantee will obtain IRB approval from their regional and/or State IRB, as well as the institutional IRB overseeing the local evaluation, and therefore will be subject to the terms and guidelines of their specific IRB. The cross-site evaluation team will provide technical assistance in drafting these documents and responding to IRB concerns, if requested by the grantee. Copies of the consent forms from individual providers will be stored at the local sites, under lock and key.
All data collected via the web-based portal will be stored electronically through Abt’s password protected secure network system. Project directories and databases are protected at Abt Associates by assigned group memberships, passwords and other techniques (e.g. ACLs), which prohibit access by unauthorized users. In addition to the issue of protection of privacy, data security encompasses backup procedures and other file management techniques to ensure that files are not inadvertently lost or damaged. Project data files are backed up to tape, using fast dump/restore software (backup Exec version 8.5) and DLT-4 tapes that hold up to 70GB of compressed data. The procedures currently utilized at Abt Associates ensure the privacy and security of many various Abt Associates research databases.
To achieve the goal of describing the families participating in local Project LAUNCH services, local program staff will be collecting information from parents that will include some sensitive questions about family and child risk factors. Although this information is sensitive, it is necessary to accurately describe families who receive Project LAUNCH services. The questions employed are from standardized measures or have been used extensively in prior studies with no evidence of harm (for example, in the Fragile Families Study and in the Early Head Start Research Evaluation Project).
The data on individual families and children being collected by the local Project LAUNCH grantees and their local evaluators will be dictated by the guidelines set forth by their IRBs and HIPAA policies to ensure the confidentiality of sensitive information. As part of the consent process, participating parents will be informed that they might find some questions sensitive, and will be asked to sign a consent form to participate, acknowledging that their participation is voluntary. All respondents will be informed that their identity will be kept private and that they do not have to answer questions that make them uncomfortable.
All information collected at the individual child/family level will be de-identified and reported in the aggregate to the cross-site evaluation.
The proposed data collection does not impose a financial burden on respondents nor will respondents incur any expense other than the time spent completing the interviews.
The estimated annual burden for study respondents—State, Tribal, and local Project LAUNCH program staff, evaluators, providers and stakeholders—is identified in Table A.2. These estimates were calculated based on an assumption of 18 Project LAUNCH grantees, and, for annual site visits, 12 respondents per site.
To compute the total estimated annual cost, the total burden hours were multiplied by the average hourly wage for each adult participant, according to the Bureau of Labor Statistics, Current Population Survey, 2008. We assumed that most respondents to the site visit interviews and web-based data reports (state and local systems data; local services data) would be program directors and program evaluators, and accordingly, used the mean hourly rate for Social Scientists and Related Workers, All Other ($34.49 per hour) to assess the cost of their providing this information.
TABLE A.2
ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST
Instrument |
Annual Number of Respondents |
Number of Responses per Respondent |
Average Burden Hours per Response |
Estimated Annual Burden Hours
|
Average Hourly Wage |
Total Annual Cost
|
Site Visit Interview Guide |
216 |
1 |
1.25
|
270 |
$34.49 |
$9,312.30 |
Electronic Data Reporting: Systems Measures |
18 |
2 |
4
|
144 |
$34.49 |
$4,966.56 |
Electronic Data Reporting: Services Measures |
18 |
2 |
8
|
288 |
$34.49 |
$9,933.12 |
Estimated Annual Burden Hours: |
702 |
|
$24,211.98 |
|||
Not applicable.
The total cost to the federal government for the three years of data collection and reporting under the contract with Abt Associates is estimated to be $3,300,000. Annual costs to the government are $1,100,000.
A.15 Explanation for Program Changes or Adjustments
This submission to OMB is a new request.
The cross-site evaluation of Project LAUNCH is intended to address the following eleven over-arching questions:
Q1: What was the child and family service system at the State/Tribal and community level when Project LAUNCH began?
Q2: What infrastructure, legislation and other policies have been developed or enhanced at the State/Tribal and community/local level through Project LAUNCH?
Q3: What are the outcomes of the changes in infrastructure, legislation and policies that have been initiated/enhanced/expanded at the State/Tribal and community level through Project LAUNCH?
Q4: What are the funding and sustainability or maintenance initiatives that have begun or been enhanced at the State/Tribal and community level through Project LAUNCH and what has resulted?
Q5: What are the child- or family-related programs and services that have been initiated/enhanced/expanded under Project LAUNCH?
Q6: What is the target population that Project LAUNCH is trying to serve with each of the service system components that has been initiated/enhanced/expanded?
Q7: What are the outcomes for the service system as a result of the programs/activities that have been initiated/enhanced/expanded under Project LAUNCH?
Q8: What changes in primary care, early childhood and other providers have occurred through Project LAUNCH?
Q9: What changes in knowledge, attitudes and behaviors related to parent ability to promote healthy child development have occurred among parents receiving individual LAUNCH services?
Q10: What changes in the physical, cognitive, social, emotional, and behavioral health have occurred for children birth to age 8 who have received or whose parents have participated in individual LAUNCH services?
Q11: What is the community-wide effect of Project LAUNCH on the physical, cognitive, social, emotional, and behavioral health of young children birth to age 8?
To address these questions, the cross-site evaluation will utilize two basic data collection components – site visits and web reporting on systems and services. Table A.3 shows which data collection components will be used to address each of the evaluation questions. In annual site visits, evaluation staff will interview State, Tribal, and local staff to obtain information on the community context, service coordination and systems infrastructure that support services. The web-based reporting system will have two components. One component will focus on State, Tribal, and local systems-level development. A second component will track aggregate numbers of services and trainings delivered to children, families, and providers. The services component includes aggregate demographics collected on families who receive services and information about provider and provider setting changes.
The data on service delivery will be used to produce a profile for each of the evidence-based practices being implemented under Project LAUNCH. The profile will be based on the RE-AIM framework. RE-AIM conceptualizes the ultimate public health benefit of an intervention as being the combined effect of five “evaluative dimensions”: Reach (proportion of the population that participated in the intervention), Effectiveness (outcomes), Adoption (proportion of settings that adopt the intervention), Implementation (extent to which the intervention is implemented as intended in the real world), and Maintenance (extent to which the intervention is sustained over time). According to its developers, the approach is consistent with systems-based,1,2 community-based, and public health interventions.2 Using RE-AIM, the public health impact is expressed as a single indicator, derived as the multiplicative combination of the five dimensions. The index can be used to evaluate the overall public health impact of an intervention3 or to compare two or more public health interventions.
In the case of the evaluation of Project LAUNCH, each service will be scored on three of the RE-AIM dimensions: Reach, Implementation, and Adoption. (Scores will be on a scale of 0 – 100%.) These scores will be used to characterize service delivery within each of the five service types that Project LAUNCH grantees are expected to implement: home visiting, family strengthening, developmental assessments, mental health consultation, and integration of mental health into primary care. If a site is implementing multiple services within a category, the scores for Reach, Implementation and Adoption will be averaged across those services.
TABLE A.3
Evaluation Questions and Data SOURCES
Evaluation Question |
Data Source |
||
Annual Site Visit Interviews |
Semi-annual Web-based Reporting |
Grantee Local Evaluations |
|
Community Context (at the State, Tribal, and Local Levels) |
|||
1. Child and family service system before LAUNCH |
X |
|
|
2. Contextual factors affecting LAUNCH |
X |
|
|
Infrastructure/System Improvement (at the State, Tribal, and Local Levels) |
|||
3. Changes in infrastructure, legislation and other policies |
X |
X |
|
Funding/Sustainability (at the State, Tribal, and Local Levels) |
|||
4. Funding/sustainability/ maintenance initiatives |
X |
X |
|
Service Delivery (at the Local Level) |
|||
5. Child/family-related programs and services under LAUNCH |
|
X |
|
6. Target populations for child/family-related programs and services under LAUNCH |
|
X |
|
7. Service system changes
|
X |
X |
|
Provider Outcomes (at the Local Level) |
|||
8. Changes in provider practices |
|
X |
|
Child and Family Outcomes (at the Local Level) |
|||
9. Changes in family outcomes |
|
|
X
|
10. Changes in child outcomes |
|
|
X
|
Community-Wide Effects |
|||
11. Changes in community |
|
|
X
|
Analyses will employ a variety of methods, including descriptive statistics (means, percentages) and simple tests of differences across subgroups and over time (t-tests, chi-square tests). Most of the evaluation questions call for descriptive analyses, which can be answered by calculating averages and percentages of families, children, providers participating in services; average scores on service delivery outcomes; and comparisons of these averages across grantees and across time. Cross-tabulations of program characteristics and family characteristics will also provide important information about differences across programs in the kinds of families they serve and how different families are served.
The time schedule for fielding, analyzing, and reporting the data findings for the cross-site evaluation of Project LAUNCH is as follows:
Annual Site Visits Fall 2010, 2011, 2012
Semi-Annual Web-based Reporting Spring/Fall 2010 – 2012
Data Analysis Spring 2010 – Fall 2012
Data Reports Annual
To date, there have been no publications from the study. However, regular reports will be produced. Sixty days prior to the end of each contract year, we will submit a draft annual report for review. This report will be revised and finalized 30 days before the end of each contract year. The annual report will include data collection description; summary of findings; summary of technical assistance provided to grantees, and, issues encountered and the resolutions. A final cross-site evaluation report will be produced at the end of the contract period. Additionally, up to two times per year we will develop and disseminate evaluation-related information and materials, which may include brief reports for the project website, brochures or fact sheets for providers and/or families, issue briefs, or peer-reviewed journal articles.
Not applicable. The OMB number and expiration date will be displayed on the top of the first page of each data collection instrument and consent form including the top of the first web page for the web-based data collection system.
No exceptions are necessary for this data collection.
REFERENCES
1 Stokols, D. Establishing and maintaining health environments: toward a social ecology of health promotion. Am Psych; 1992;47: 6-22.
2 Green, LW & Johnson, JL Dissemination and utilization of health promotion and disease prevention knowledge: theory, research and experience. Can J of Pub Health: 1996 (suppl 2); 11-17.
3Bopp, M. Wilcox, S Laken, M Hooker, SP Saunders, R Parra-Medina, D. Using the RE-AIM framework to evaluate a physical activity intervention in churches. Prev Chronic Dis: 2007; 4(4): 1-9.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Dawn Smith |
| Last Modified By | DHHS |
| File Modified | 2009-09-21 |
| File Created | 2009-09-21 |