PPW OMB supporting statement 9-1-2010

SUPPORTING STATEMENT FOR THE

EVALUATION OF PREGNANT AND POSTPARTUM WOMEN (PPW)

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT) requests approval from the Office of Management and Budget (OMB) for the Evaluation of Pregnant and Postpartum Women (PPW), hereinafter known as the Evaluation. CSAT requests approval for a total of 23 instruments for this new data collection. Of these 23 instruments, 18 instruments are client-level tools and 5 instruments are process-level tools. Not all client-level tools are asked of every participant or at every time point. These instruments include: 

Client-level tools (by respondent):

Mother (Attachment A-1)

1. Brief Infant Toddler Social and Emotional Assessment (A-1.1)

2. Child Data Collection Tool (A-1.2)

3. Parenting Relationship Questionnaire (A-1.3)

4. Parenting Stress Index (A-1.4)

5. Social Skills Improvement System (A-1.5)

6. Trauma Symptom Checklist for Young Children (A-1.6)

7. BASIS-24® (A-1.7)

8. Child Abuse Potential Inventory (A-1.8)

9. Family Support Scale (A-1.9)

10. (duplicate) Ferrans and Powers Quality of Life Index (A-1.10)

11. Items Administered to Women (A-1.11)

Child (Attachment B-1)

12. CRAFFT (B-1.1)

Partner/Father (Attachment C-1)

10. (duplicate) Ferrans and Powers Quality of Life Index (C-1.1)

Project Staff (Attachment D)

13. Children’s Discharge Tool (D-1)

14. Women’s Discharge Tool (D-2)

15. Staff Completed Women’s Items (D-3)

16. Staff Completed Child Items (D-4)

Medical Staff (Attachment E)

17. Newborn’s Medical Record Audit (E-1)

18. Staff Completed Newborn Items (E-2)

Process-level tools (by respondent):

Mother (Attachment A-1)

1. Site Visit Protocol-Client Focus Group (A-1.12)

Project Director (Attachment F)

2. Biannual Project Director Telephone Interview (F-1)

Clinical Director/Supervisor (Attachment G)

3. Site Visit Protocol-Clinical Director/Supervisor Interview (G-1)

Counselor (Attachment H)

4. Site Visit Protocol-Counselor Interview (H-1)

Program Director (Attachment I)

5. Site Visit Protocol-Program Director Interview (I-1)

Title V, Section 508, Residential Treatment Programs for Pregnant and Postpartum Women, of the Public Health Service Act, as amended, mandates the evaluation and dissemination of findings of residential treatment programs for pregnant and postpartum women. Though several Federal agencies have mandates to fund projects targeting substance abusing women and their minor children, evaluation data have shown that 10 percent or less of women treated in women-specific programs were pregnant or postpartum.¹

To address the needs of this underserved population, CSAT published a Request for Applications (RFA) in 2008, which can be found in Attachment J-1. Eleven grantee projects were funded as a result of this RFA. To comply with Section 508, this Evaluation has implemented a systematic process of assessing accomplishments toward meeting the goals of the RFA. The Evaluation appraises maternal and minor child health outcomes in 11 projects in which SAMHSA and RFA goals are to improve the quality and availability of treatment through accountability, capacity, and effectiveness.

2. Purpose and Use of Information

Although rates of illicit drug abuse are lower for females than for males ages 12 years and older, in recent years, rates have increased for women but have remained relatively stable for men.^2,3 Further, substance abuse remains a significant problem among PPW in the United States.⁴ Studies have consistently shown that prenatal substance use can have deleterious effects on maternal quality of life, birth outcomes, and child development.^5,6,7 A National Institute on Drug Abuse (NIDA) study showed that children exposed to alcohol and illicit drugs are at-risk for birth defects, mental retardation, and later behavioral and learning difficulties.⁸ Postnatal parental substance abuse can also negatively affect children by compromising the parent-child relationship,⁹ negatively influencing child development,¹⁰ contributing to child abuse and neglect,¹¹ increasing encounters with the child welfare system,¹² and leading to both behavioral and mental health problems.¹³ Children of substance abusing parents are also at higher risk for substance abuse themselves.¹⁴ Although the literature consistently suggests that prenatal and postnatal substance use negatively impact the mothers and their children in many ways, substance abuse among PPW remains undertreated.¹⁵ The following data suggest the national magnitude of the problem:

• According to estimates from the National Survey on Drug Use and Health (NSDUH), based on self-reported past-month use among pregnant women between the ages of 15 to 44 years, 16.4 percent reported smoking cigarettes, 10.6 percent used alcohol, 5.1 percent used illicit drugs, and 4.5 percent binge drank alcohol.¹⁶ In a large, multisite study (N =1,632), the incidence of substance use during pregnancy was as follows: tobacco (25%), alcohol (23%), marijuana (6%), methamphetamine (5%), and barbiturates (1%).¹⁷

CSAT is especially concerned about the high morbidity and mortality rates of African-American pregnant women and their infants. Combined 2006 and 2007 NSDUH estimates indicate that among pregnant women, illicit drug use was highest among White women (6.2% vs. 5.7% for African American and 2.9% for Hispanic women), but binge alcohol use was highest among African American women (7.8% vs. 3.4% for White and 2.5% for Hispanic women).¹⁸

• The effects of illicit drug use by women during the prenatal period are well documented in the literature and include problems such as inadequate prenatal care,¹⁹ fetal death, premature delivery, congenital anomalies,²⁰ and low birth weight infants.²¹ The National Center for Health Statistics reports persistent racial/ethnic disparities in infant mortality. From 1980-2000, the infant mortality rate for babies born to African-American mothers was 14.0 per 1,000 live births while the rate for babies born to Caucasian mothers were 5.7 per 1,000.²²

• Maternal alcohol use during pregnancy has been shown to have neurological and behavioral effects on minor children.²³ The prevalence of infants born with Fetal Alcohol Syndrome (FAS) has been estimated at 0.2 to 1.5 per 1,000 live births.²⁴ Evaluation data showed that only 10 percent or less of pregnant or postpartum women treated for substance abuse/addiction and its complications utilize women-specific services targeted toward them.²⁵

Use of the data to support performance measurements for SAMHSA. The primary purpose of the PPW Program is to provide cost-effective, comprehensive residential substance abuse treatment services for women and their minor children that can be sustained over time. To measure the achievement of this goal, CSAT is conducting this Evaluation and program monitoring to measure the outcomes of treatment at each project on maternal and child outcomes, including maternal substance abuse, quality of life, infant birth outcomes, and early childhood development. CSAT will use this information to document and report the extent to which the goals of the RFA were achieved, as mandated by Congress.

Outcome data reflect SAMHSA’s desire for consistency in data collected within the Agency. SAMHSA has implemented specific performance domains called National Outcome Measures (NOMs) to assess the accountability and performance of its discretionary and formula grant programs. These domains represent CSAT’s focus on the factors that contribute to the success of substance abuse treatment. The PPW program addresses the following performance domains:

• Abstinence from Drug / Alcohol Use

• Employment / Education

• Crime and Criminal Justice

• Family and Living Conditions

• Social Connectedness

• Access / Capacity

• Retention

• Perception of Care

• Use of Evidence-Based Practices

It is well known that the use of drugs, alcohol, and tobacco has a negative impact on maternal health and pregnancy outcomes. Outcome data are used to measure the success of clinical treatment and recovery support services. The results are used to assess the need to continue SAMHSA’s targeted PPW programs, to design programs, to coordinate systems of care, and to provide assistance that ensures such programs can contribute appropriately to treatment and prevention of substance abuse among pregnant women and the prevention of health and educational problems among the offspring of these women.

SAMHSA/CSAT has established core standardized Government Performance and Results Act (GPRA) Client/Participant Outcome Measures (OMB No. 0930-0208) for all grant programs to report in order to capture the following essential client/participant-level information. The PPW projects are required to collect the following GPRA data for each mother that enters treatment.

1. Whether, over the past year, there was an increase in the percentage of women that received services:

a) Who had no use of illegal drugs or misuse of prescription drugs during the past month.

b) Who had a permanent place to live in the community/residing in a stable living environment.

c) Who had reduced involvement with the criminal/juvenile justice system.

d) Who were currently employed or engaged in productive activities/attending school.

2. Retention in the program—the percentage of clients who completed the program or who left the program before completion and their status (discharge status).

3. Types of services received while in the program—to show the percentage of clients in the different types of treatment modalities.

4. Whether clients sought help from self-help groups to support their recovery.

PPW Program Background and Overview. In recent years, SAMHSA has awarded a number of three-year PPW grants: six in fiscal year (FY) 2003, eight in 2006, 16 in 2008, and 11 in 2009. The proposed Evaluation includes only those PPW grantees with FY 2009 awards. Participation in the Evaluation is mandatory.

Inclusion criteria: Women who are low-income and are pregnant and/or up to 12 months postpartum at intake are eligible for admission to PPW treatment programs. Grantees are expected to collect cross-site Evaluation data for all women admitted to their programs, their minor children (under 18 years) who receive services, and the children’s father(s) and/or woman’s partner. This data collection is in addition to the CSAT-required GPRA data collection.

Racial and ethnic minority women, especially African American women, have been identified as an important subpopulation by CSAT because they have been traditionally underserved and have had high morbidity and mortality rates. These women and children have limited access to quality health services. An addition in the Congressional budget language for FY 2006 recommends that SAMHSA explore ways to increase family-focused treatment capacity. Therefore, for the PPW Program, CSAT increased the target population to include the partners of the women and the fathers of the children, when deemed to be appropriate and beneficial, as well as extended family members of the women and children in treatment.

Length of residential treatment: CSAT recommends that the intensive residential treatment phase of the treatment process not exceed 12 months. In general, PPW projects select treatment phases ranging from 3 to 12 months as charted by the client’s individual service plan.

Program goals: The primary purpose of the PPW treatment program is to provide cost-effective, comprehensive residential substance abuse treatment services for women and their minor children that can be sustained over time. The five core goals of the PPW program are listed in the next subsection (Treatment outcomes) under evaluation question 1. The service system should address the individual needs of the target population and preserve and support the family unit, while creating a safe and healthy environment for family members.

Treatment outcomes: Treatment outcomes for the women include decreased alcohol and drug use; increased safe and healthy pregnancies; improved mental and physical health; improved family functioning; economic stability; improved quality of life; as well as decreased involvement in and exposure to crime, violence, sexual and physical abuse, and child abuse and neglect. Treatment program goals for the children are improved birth and developmental outcomes; prevention of and/or decreased use and abuse of alcohol and drugs; reduced effects related to maternal drug abuse; improved mental and physical health; and decreased involvement in and exposure to crime, violence, sexual and physical abuse, and child abuse and neglect.

The Evaluation design includes both client-level outcomes and process evaluation components. Attachment J-2 presents the logic model for the Evaluation data collection.

The primary evaluation objectives are addressed by three main evaluation questions:

1. To what extent are PPW grantees meeting the goals and objectives of the RFA?

This evaluation question (and primary focus of the study) examines the extent to which grantees achieve the following five core goals of the PPW program (as outlined in the PPW RFA):

1. Decrease the use and/or abuse of prescription drugs, alcohol, tobacco, illicit and other harmful drugs (e.g., inhalants) among pregnant and postpartum women;

2. Increase safe and healthy pregnancies; improve birth outcomes; and reduce related effects of maternal drug abuse on infants and children;

3. Improve the mental and physical health of the women and children;

4. Improve family functioning, economic stability, and quality of life; and

5. Decrease involvement in and exposure to crime, violence, sexual and physical abuse, and child abuse and neglect.

2. How do PPW clients compare to pregnant women in other CSAT programs on GPRA outcomes and national estimates of substance use and birth outcomes?

This evaluation question examines GPRA outcomes for both PPW clients and pregnant women in other CSAT funded substance abuse treatment programs to determine if PPW clients achieve better outcomes. Additionally, to answer this question, we will compare PPW client outcomes to national benchmark estimates of substance use and birth outcomes to examine how different (or similar) PPW clients are to others in the United States.

3. Do treatment implementation and contextual factors mediate or moderate PPW client outcomes?

This evaluation question examines the extent to which the goals and objectives of the RFA (i.e., client-level outcomes) are influenced by different factors such as specific grantee characteristics, strategies, and services, and the context within which the grantees’ treatment programs are implemented. Sub-questions include the following:

1. What are the characteristics of the services provided by PPW grantees to clients (e.g., type of services provided, location of services provided, extent to which the services provided match what was proposed, and degree to which services are gender-sensitive, family-focused, and comprehensive)?

2. What is the quality of the services provided by PPW grantees to clients (e.g., characteristics of staff who deliver services to clients, quantity of services delivered, extent to which the quality of services provided match what was proposed, and the degree to which evidence-based practices (EBPs) are implemented)?

3. What is the relationship between client-level outcomes and the characteristics and quality of client services received?

4. What is the relationship between client-level outcomes and the type and amount of technical assistance (TA) requested and received?

5. Does the context within which PPW grantees implement services influence client-level outcomes?

6. To what extent are PPW grantees implementing strategies to ensure the sustainability of their treatment program?

Information to be Collected and Frequency of Collection.

A crosswalk of evaluation questions, data sources, measured constructs and data collection time points is provided in the matrix in Attachment J-3. Most tools are maternal interviews, some are staff chart reviews, one interview is administered to fathers/partners. One interview is administered to children ages 11-17. Other child tools are based on child age; one is completed at each time point for each eligible child, based on maternal report and/or records review. Although some tools can be self-administered, staff will be trained to administer the tools in order to standardize administration method and accommodate limited literacy of some clients.

Data will be collected from women at four time points (intake, 6-months post-intake, discharge, and 6-months post-discharge), consistent with the GPRA data collection schedule. The schedule for collecting child data is similar to the mother’s with the addition of a 3-months post-intake time point with selected tools to account for the changes that occur more quickly in young children, particularly among those children who are reunified with their mothers during treatment. In addition, the “intake” time point for newborns is upon delivery for pregnant women entering the treatment program. For the purpose of the Evaluation, intake is based on the date the woman’s intake GPRA is completed and discharge is based on maternal discharge from residential treatment. The data collection schedule is provided in Table B-1 and includes information on respondents, data collectors, data collection methods and schedules for each tool.

Data collection instruments (client-level). A thorough instrumentation assessment was completed to create a streamlined battery of psychometrically sound tools that are potentially sensitive enough to detect changes and that will minimize administration burden in the treatment setting. The tools were assessed based on their utility in measuring the PPW constructs and outcome measures. Tools were also assessed for ease of scoring, reading level, staff qualifications necessary for administration, grantee burden, and availability of tools in Spanish. The selected tools are also potentially clinically beneficial to the client treatment. In addition to these tools, data collection is supplemented by adding a few questions (not full tools) to fill remaining gaps, including issues such as maternal tobacco use, additional birth outcomes, pregnancy outcomes, women’s violence exposure, and additional treatment and recovery supports for women. These additional items are included in the form of staff-completed newborn, child and women’s items, and items administered to women. Child tools are administered based on child age. A brief description of tools is provided here. See Attachments A-E for complete copies of each tool. The proprietary client-level tools include tools commonly in use in the field.

Women focused tools:

• BASIS-24®: The 24-item Behavior and Symptom Identification Scale (BASIS-24®), copyrighted by McLean Hospital in Belmont, Massachusetts, is a leading behavioral health survey that measures the change in self-reported psychological symptoms and problem difficulty experienced during the past week. It was developed to assess patient outcomes from residential and outpatient mental health treatment. The survey measures the degree of difficulty experienced by the client on a five-point scale ranging from no difficulty to extreme difficulty. The BASIS-24® yields an overall score and six subscale scores: depression/functioning, relationships, self-harm, emotional lability, psychosis, and substance abuse.

• Child Abuse Potential Inventory (CAP Inventory): This tool can be used to assist in screening for suspected physical child abuse cases. The CAP Inventory includes a physical child abuse scale as well as six subscales, three of which measure psychological difficulties (i.e., distress, rigidity, unhappiness) and three of which measure interactional problems experienced by the respondent (i.e., problems with child and self, problems with family, and problems from others).

• Ferrans and Powers Quality of Life Index (QLI): The QLI measures quality of life overall and satisfaction with and personal importance of four domains: health and functioning, social and economic, psychological and spiritual, and family. The website for this instrument is (http://www.uic.edu/orgs/qli/).

• Family Support Scale (FSS): This tool measures the helpfulness of sources of support available to families rearing a young child. Examples of sources of support include parents, spouse or partner’s parents, and co-workers.

• Women’s Discharge Tool: This tool measures a woman’s length of stay in treatment, treatment completion, treatment goals achieved, and treatment services received. This tool was developed and refined during prior PPW evaluation activities to obtain a picture of what services the women received while participating in these PPW substance abuse treatment programs as well as how often and where the services were provided.

• Staff Completed Women’s Items: These few items were developed for this Evaluation to measure pregnancy status, problems during pregnancy, pregnancy outcomes, and an explanation if no followup was obtained.

• Items Administered to Women: These few items were developed for this Evaluation to measure the number of children residing with the mother during treatment, maternal tobacco use, physical and sexual abuse in the past year, and additional services received since leaving treatment.

Father and partner focused tools:

• Ferrans and Powers Quality of Life Index (QLI): see above

Child focused tools:

• Brief Infant Toddler Social and Emotional Assessment (BITSEA; children 12–35 months): This tool identifies children who may have social-emotional and behavioral problems and/or delays, or deficits in social-emotional competence. The BITSEA includes items on a broad range of social-emotional problems and competencies including internalizing, externalizing, and regulatory domains, as well as rare behaviors that may be indicative of autism spectrum disorders or other significant psychopathology.

• Child Data Collection Tool (all children): The purpose of this tool is to collect comprehensive demographic and health information to create a profile of each minor child who receives services from PPW programs. Further, it is intended to help the treatment field to identify resources needed for these children and to help develop programs for these children. It was developed for and refined during prior PPW evaluation activities. This tool is used to collect information on the child’s biological background, socio-economic background, legal background, educational background, spiritual background, recreation/leisure background, background of parental relationships, alcohol and other drug use, and health background/medical history.

• Children’s Discharge Tool (all children): This tool measures a child’s length of stay in treatment, treatment completion, treatment goals achieved, and treatment services received. This tool was developed and refined during prior PPW evaluation activities to obtain a picture of what services children received while participating in these PPW programs as well as how often and where the services were provided.

• CRAFFT (children 11–17 years): This is a 6-item substance abuse screening tool for adolescents. Receiving a score of two or higher indicates the need for further assessment.

• Newborn’s Medical Record Audit (childen birth–3 months): This tool collects common information on birth outcomes including: Apgar score, head circumference, length at birth, birth weight, gestational age, and results of drug toxicology screening.

• Parenting Relationship Questionnaire (PRQ; children 2–17 years): This tool obtains information on a parent’s perspective of the parent-child relationship. It assesses the following traditional parent-child dimensions: attachment, communication, discipline practices, involvement, parenting confidence, satisfaction with school, and relationship frustration.

• Parenting Stress Index (PSI; children 1 month – 12 years): This tool was designed to measure stress within parent-child dyads and to identify dysfunctional parent-child relationships that may lead to child emotional and behavioral problems. The PSI includes items to assess three major sources of stressors: (a) child characteristics (distractibility/hyperactivity, adaptability, reinforces parent, demandingness, mood, and acceptability); (b) parent characteristics (competence, isolation, attachment, health, role restriction, depression, and spouse); and (c) situation/demographic life stress (the amount of stress outside of the parenting relationship that the parent is experiencing such as the death of a relative or the loss of a job).

• Social Skills Improvement System (SSIS; children 3–17 years): This tool assesses children’s social skills functioning (communication, cooperation, assertion, responsibility, empathy, engagement, and self-control); competing problem behaviors (externalizing, bullying, hyperactivity/inattention, internalizing, and autism spectrum); and academic competence (reading and math performance, motivation, parental support, and general cognitive functioning).

• Trauma Symptom Checklist for Young Children (TSCYC; 3–12 years): This tool measures the effects of children’s exposure to trauma. The TSCYC includes eight clinical scales: anxiety, depression, anger/agression, posttraumatic stress-intrusion, posttraumatic stress-avoidance, posttraumatic stress-arousal, dissociation, and sexual concerns, as well as a summary posttraumatic stress scale.

• Staff Completed Child Items (children 0–17): These few items were developed for this Evaluation to measure prematurity, child’s recent primary residence, whether child resided in treatment with mother, and if not, how often child visited mother in treatment, and an explanation if no followup was obtained.

• Staff Completed Newborn Items (children 0–3 months): These few items were developed for this Evaluation to augment those in the Newborn’s Medical Record Audit to measure prematurity, length of stay in the hospital, treatment in the neonatal intensive care unit (NICU), and treatment for neononatal abstinence syndrome (i.e, withdrawal).

Data collection instruments (process evaluation). In order to help interpret client-level outcomes, the process evaluation tools will explore what grantees are actually doing, how well they are doing it, any challenges encountered, and strategies grantees used to address them. The focus of the process evaluation component is to examine the extent to which the goals and objectives of the RFA (i.e., client outcomes) are influenced by different factors such as: specific grantee characteristics, strategies, services, and the context of PPW grantee’s treatment programs. Data for the process evaluation will be obtained from site visits of each PPW grantee and followup interviews with grantee project directors about their biannual progress reports. See Attachments F-I for complete copies of each tool, and Table B-1 for the data collection schedule.

Process evaluation tools:

• Site Visit Protocol-Program Director Interview: This interview will gather information about overall programmatic issues such as the following: the types of services delivered on-site and off-site; sources of client referrals; staff background/experience; staff turnover, vacancies; accreditation/ licensing; discrepancies between proposed and implemented services; challenges to program implementation; and plans for the sustainability of grantees’ treatment programs.

• Site Visit Protocol-Clinical Director/Supervisor Interview: This interview will focus on information about clinical services. Data related to the following areas will be gathered through this tool: Evidence-based practices (EBP) being implemented by the grantee (e.g., modifications made to the EBP, degree of implementation of the EBP); program policies/ procedures (e.g., strategies for intake, retention and discharge); treatment process/delivery (e.g., written treatment manuals, sequence of services delivered to clients); treatment services for women (e.g., degree to which they are trauma-informed, gender-sensitive, and family focused); treatment services for children (e.g., types of treatment provided that is not child-care, level of involvement of parent in child’s treatment); staff/client ratio; and challenges to program implementation.

• Site Visit Protocol-Counselor Interview: This interview will gather information related to: counselor’s current responsibilities (e.g., case management, therapy); implementation of EBP and/or treatment manuals (e.g., degree to which EBP/treatment manual is used in daily services and implementation of policies/procedures); consistency of case load; implementation of services (e.g., client contact with partner/access to children, support materials used in treatment, continuing care services); and challenges to program implementation.

• Site Visit Protocol-Client Focus Group: These focus groups will gather information about: receipt of services; experience with services/staff (both on-site and off-site); length of time in grantee’s treatment program; variation in services among clients; program challenges/strengths; and recommended improvements for the treatment program.

• Biannual Project Director Telephone Interview: This interview will focus on clarifying information reported in grantees biannual progress reports. Specifically, information will be gathered about: professional development activities; TA requested/received; context of the treatment program (changes in local conditions/sources of client referrals); sustainability plan strategic planning process; and challenges to implementation.

3. Use of Information Technology

Data collection using these paper-based instruments is conducted in the course of normal service delivery (as is the generally accepted assessment technique within clinical settings). Project staff will photocopy the instruments and send them to the CSAT-designated contractor for data entry. Projects retain a copy of each instrument for their own clinical use. Current technology is used to manage, secure, store, clean and analyze the data. An electronic data tracking system is also used to electronically notify grantees when data collection instruments are due. Grantees also have access to a toll-free help line and email helpline for quick response from the contractor to their questions.

Web-based data collection is not practicable for this data collection for several reasons. First, the development of a web-based data collection system is expensive, particularly considering the type of instruments included and the relatively modest number of participants responding to each. Second, it would not reduce burden. Data are collected as part of clinical practice and would need to be recorded and then entered into a computer. Third, submission of hard-copy instruments allows contractor staff to resolve errors before data entry occurs which ensures greater quality control.

4. Efforts to Identify Duplication

Information to be collected as part of this Evaluation with these 11 residential treatment projects is not available elsewhere. The battery of instruments to be used in this Evaluation has not been collected previously with this population. These data are specific to the needs of this Evaluation. Existing GRPA data from women in non-PPW CSAT programs and public use data from national surveys, (e.g., NSDUH substance use outcomes for pregnant women) and national surveillance systems (e.g., CDC National Center for Health Statistics (NCHS) vital statistics on birth outcomes) will be used as benchmark comparisons for PPW outcomes.

5. Involvement of Small Entities

The 11 projects aim to treat approximately 20 to 40 clients per year. Many of the questions are generally incorporated in instruments that are familiar to clinicians in the substance abuse treatment field. There is not a significant impact to these entities.

6. Consequences if Information is Collected Less Frequently

The effects of treatment and changes in client-level outcomes can only be assessed by collecting data at multiple time points. During this Evaluation, the frequency of data collection from projects, women, minor children, and family members will be held to the minimum necessary to meet the needs of the Evaluation. The data collection points for this Evaluation are generally accepted intervals for assessing the effectiveness of substance abuse treatment.²⁶ CSAT needs these data in order to assess that PPW program goals are adequately met..

Mothers will be asked to respond voluntarily at intake. Data will be collected from women at four time points (intake, 6-months post-intake, discharge, and 6-months post-discharge), consistent with the GPRA data collection schedule. The schedule for collecting child data is similar to the mother’s with the addition of a 3-months post-intake time point with selected tools to account for the changes that occur more quickly in young children, particularly among those children who are reunified with their mothers during treatment. Family members (i.e., women’s partners and children’s fathers) will be asked to voluntarily complete one instrument within 30 days of the woman’s intake and at the woman’s discharge.

Process data will be gathered at different time points than the client-level data. Project director telephone interviews will be conducted twice per year (every 6 months) and two site visits to each grantee will be conducted over the course of the Evaluation. Telephone interviews with grantee project directors need to be conducted every 6 months throughout the course of the evaluation to coincide with the submission of grantee biannual progress reports. The focus and purpose of these interviews is to clarify information reported within these reports, thus they cannot be conducted with less frequency.

Two site visits to each grantee are also important to the Evaluation. In order to examine grantees’ progress in the implementation of their projects it is necessary to conduct a minimum of two site visits over the course of their grant. Progress can only be assessed when data are obtained from two or more time points, thus two site visits are essential to the Evaluation.

7. Consistency with the Guidelines in 5 CFR 1320.5(d) (2)

This information collection fully complies with 5 C.F.R. § 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on June 29, 2010 (Vol.75, pg. 37445). One comment was received from the public in response to the notice. See Attachment K for the comment and SAMHSA’s response to the comment.

9. Payment to Respondents

Cash-equivalent incentives are used for all followup interviews conducted after the woman has left the residential treatment program and for all interviews of family members not in the treatment program with the woman. RFA recommendations state, that “the maximum allowable incentive is $20.00 or equivalent value in coupons, bus tokens, and personal care items per followup interview.” Cash equivalents are offered in lieu of cash payments. This remuneration is expected to aid in achievement of acceptable response rates.

Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. Substance abuse research has shown improved response rates when remuneration is offered to respondents. Brigham et al., (2010) examined the use of retail scrip incentives among pregnant substance users and concluded that incentives in $25-$30 range may serve to significantly increase attendance and retention. Women were nearly three times as likely to attend incentivized research visits as non-incentivized treatment visits.²⁷ Substance abusers are typically a harder-to-reach population for whom out-of-pocket costs (e.g., transportation, child care) are significant barriers to participation. Therefore, transportation, child care, and/or home visits are provided by projects as needed for participation in the followup. In addition, it is expected that projects follow the recommendations of the RFA with regard to remuneration amounts.

Results from the 2001 National Household Survey on Drug Abuse (NHSDA) incentive experiment were reported by Eyerman and Bowman (2001) and Wright, Bowman, Butler, and Eyerman (2002). Key conclusions from their analyses are quoted below:²⁸

• The $20 and $40 incentive payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for $40 than the $20 incentive within many of the subgroups addressed in the analysis.

• Both incentive payment groups more than paid for themselves due to decreased costs of followup and more productive screening resulting from the improved response rates.

• Some significant differences in prevalence rates were noted in comparisons between the $40 treatment and the control in some of the age, race, and historical response rate groups: two cases of significantly higher past month alcohol use and one case of significantly lower past month cigarette use.

• Persons who responded with incentives, but would not have responded without them, are different and have higher substance use than persons who would respond with or without incentives.

Incentives motivate (or obligate) respondents to admit to substance use that they might not have admitted without the incentive.

10. Assurance of Confidentiality

Client treatment records in Federally-assisted alcohol and drug abuse programs are protected under 42 CFR Part 2 (OMB No. 0930-0092). SAMHSA and its contractors do not receive identifiable client records. Provider-level information is aggregated to, at the least, the overall grant program level.

The directors of all selected projects and all other potential respondents are assured that privacy is maintained throughout data collection (to the extent permitted by law) in all project-level descriptive data gathered periodically from project staff by CSAT staff. All data are closely safeguarded, and no institutional or individual identifiers are used in assessment reports, in which only aggregated data are reported.

Protection of the rights of evaluation participants is assured through a combination of widely accepted survey practices. All PPW projects comply with applicable Federal and State laws and with ethical principals in the collection of information from and about persons enrolled in, or related to persons enrolled in, treatment. Among the rights commonly held for these types of studies are:

1. The right of informed consent/assent, which requires the evaluation team to provide sufficient information about the evaluation objectives, level of burden, and uses of participants’ information so that individuals may make an informed decision on participation;

2. The right to refuse to participate, which applies to the individual’s right to decline to participate at all in the Evaluation or to decline to answer specific questions, without penalty or loss of benefits;

3. The right to privacy, which guarantees against invasions of privacy as well as the specific protections provided by the Privacy Act of 1974.

It is the responsibility of individual projects to ensure privacy of participant data. No system of records containing identifiers is maintained by CSAT or its contractors. Before submitting these data to CSAT, projects are instructed to delete all personal identifiers (such as names, addresses, phone numbers, Social Security Numbers, medical record numbers, etc.) from the data files. The projects also are directed to assign an identification number to each client strictly for the purposes of the Evaluation. This number enables the contractor to keep track of individual client records in the absence of personal identifiers, and to link records over the course of the repeated submissions per client that take place as part of the time series design of the Evaluation. However, the correspondence between the true identity of the client and the number assigned for evaluation purposes are known only to the projects, who maintain parallel lists of the two types of client identifiers.

The Federal data collection affords no circumstance in which privacy of client data could be breached, since only anonymous information is received. It is the responsibility of CSAT to ensure that client data are reported only in aggregate form without linking information with a specific project.

SAMHSA is subject to the Privacy Act for the protection of data. Substance abuse treatment providers are subject to the Federal regulations for the privacy of alcohol and substance abuse patient records (42 CFR Part 2, OMB No. 0930-0092) which govern the protection of patient identifying data. In some cases, these same providers meet the definition of a HIPAA covered entity and are additionally subject to the Privacy Rule (45 CFR Parts 160 and 164) for the protection of individually identifiable data.

The PPW Program has been determined by the CSAT Director to fall under the SAMHSA Participant Protection Procedures. These procedures require each applicant to the RFA to provide information which is used to determine whether the level of protection of human subjects appears adequate or whether further provisions are needed according to standards set forth in 45 CFR Part 46 Protection of Human Subjects.

Adequate protection of human subjects is an essential part of an application and is carefully reviewed by the grant review panel. Applicants must report any foreseeable participation protection risks and the procedures developed to protect participants from those risks. Applicants must describe the selection of participants, consent/assent procedures, privacy procedures, and data collection including from whom the data are collected, the form of specimens, records, or data. In addition, projects must include a discussion of why the risks to subjects are reasonable in relation to the anticipated benefits to subjects and in relation to the importance of the knowledge that may reasonably be expected to result. The applications also are examined by a Federal Project Officer to determine whether these procedures are being met. The Project Officer works with applicants when the review panel has concerns or comments in order to enable award.

Although projects routinely obtain informed consent/assent from project participants, in which the women agree to allow information collected regarding them, CSAT has developed consent and assent forms to be used by previous projects and to be used by the 11 projects in this Evaluation. The forms can be found in Attachment A-2 and B-2. CSAT also provides needed oversight and training on issues relating to informed consent/assent for women, fathers/partners, and minor children. In addition, an informed consent form is completed by clients participating in focus groups about program treatment services as part of the process evaluation. The focus group consent form is also provided in Attachment A-2.2.

11. Questions of a Sensitive Nature

The primary purpose of most SAMHSA data collections is to gather sensitive information on substance abuse and mental health. SAMHSA’s mission is to improve the quality and availability of prevention, early intervention, treatment, and rehabilitation services for substance abuse and mental illnesses, including co-occurring disorders, in order to improve health and reduce illness, death, disability, and cost to society. In carrying out this mission it is necessary for service providers to collect sensitive information, such as criminal justice involvement, use of alcohol or other drugs, as well as issues of mental health.

The data submitted by each project to CSAT as part of the client-level assessment are based in large part on data that most of the projects are already routinely collecting. This primarily includes data on client demographics, substance abuse and treatment history, services received, and client and child outcomes. In addition, projects frequently ask their clients about their experiences of physical, sexual, and emotional abuse, as well as the custody and living arrangements of their minor children and the client’s involvement with Child Protective Services.

11. Estimates of Annualized Hour Burden

The total annualized burden to respondents for all components of the PPW program is estimated to be 8,404 hours. The burden estimates, detailed in Table A-1, are based on the reported experience of the FY 2006 grantees, proprietary instrument developer estimates and experience, pre-testing of the additional items completed by staff and administered to women, and pre-testing of process evaluation measures. The annualized hourly costs to respondents are estimated to be $81,744.73. There are no direct costs to respondents other than their time to participate.

Table A-1 presents a detailed breakdown of the annual burden for all data collection instruments for all respondents (i.e., mother, child, project staff, partner/father (family members), medical staff, project director, clinical director, counselor, program director). The number of respondents for all child-focused tools is weighted, based on the percentage of children within the appropriate age bracket in the prior PPW evaluation. With the exception of the CRAFFT, all child-focused tools are completed for the child by the mother or project staff. Regarding hourly wages, it is estimated that the mothers earn the minimum wage of $7.25 per hour.²⁹ The partners’/fathers’ hourly wage is estimated to be the median hourly wage as reported in the Bureau of Justice Statistics’ Occupational Employment Statistics - $15.57 per hour.³⁰ It is estimated that the minor children do not work, and therefore, do not earn anything (i.e., hourly wage is $0). All other wages are based on national estimates of hourly wages for similar professions as specified in the Bureau of Labor Statistics’ Occupational Employment Statistics (see Table A-1 notes).

Table A-1. Detailed Annual Burden for All Interviews & Surveys
Interviews and Surveys	Respondent	Number of Respondentsa	Responses per Respondent	Total Responses	Burden per Resp. (hrs.)	Total Burden (hrs.)	Hourly Wage Costb	Total Hour Cost
Child Focused Interviews:
CRAFFT (11-17 yrs)c	Child	70	5	350	0.08	28	$0.00	$0.00
Brief Infant Toddler Social and Emotional Assessment (12-35mos)d	Mother	141	5	705	0.17	120	$7.25	$870.00
Child Data Collection Tool (0-17 yrs)e	Mother	440	2	880	0.75	660	$7.25	$4,785.00
Parenting Relationship Questionnaire (2-17 yrs)f	Mother	387	5	1,935	0.25	484	$7.25	$3,509.00
Parenting Stress Index (1 month-12 yrs)g	Mother	418	10	4,180	0.5	2,090	$7.25	$15,152.50
Social Skills Improvement System (3-17 yrs)h	Mother	326	5	1,630	0.42	685	$7.25	$4,966.25
Trauma Symptom Checklist for Young Children (3-12 yrs)i	Mother	290	5	1,450	0.33	479	$7.25	$3,472.75
Women Focused Interviews:
BASIS-24®	Mother	440	4	1,760	0.25	440	$7.25	$3,190.00
Child Abuse Potential Inventory	Mother	440	4	1,760	0.33	581	$7.25	$4,212.25
Family Support Scale	Mother	440	4	1,760	0.17	299	$7.25	$2,167.75
Ferrans and Powers Quality of Life Index (Women)	Mother	440	4	1,760	0.17	299	$7.25	$2,167.75
Items Administered to Women	Mother	440	4	1,760	0.17	299	$7.25	$2,167.75
Fathers and Partners Interview:
Ferrans and Powers Quality of Life Index (Partners)	Partner/Father	110	2	220	0.17	37	$15.57	$576.09
Staff Completed Items/Record Reviews at 11 Facilities:
Children's Discharge Tool (0-17 yrs)j	Project Staff	11	80	880	0.58	510	$19.07	$9,725.70
Women's Discharge Tool	Project Staff	11	40	440	0.58	255	$19.07	$4,862.85
Newborn's Medical Record Audit (0-3 mos)k	Medical Staff	11	25	275	0.08	22	$31.31	$688.82
Staff Completed Newborn Items	Medical Staff	11	25	275	0.25	69	$31.31	$2,160.39
Staff Completed Child Items (0-17 yrs)l	Project Staff	11	400	4,400	0.08	352	$19.07	$6,712.64
Staff Completed Women's Itemsm	Project Staff	11	160	1,760	0.17	299	$19.07	$5,701.93
Process Evaluation:
Biannual Project Director Telephone Interview	Project Director	11	2	22	1	22	$23.16	$509.52
Site Visit Protocol - Client Focus Groupn	Mother	176	1	176	1.5	264	$7.25	$1,914.00
Site Visit Protocol - Clinical Director/Supervisor	Clinical Director/Supervisor	22	1	22	2	44	$19.05	$838.20
Site Visit Protocol - Counselor(s)	Counselor	33	1	33	1	33	$19.07	$629.31
Site Visit Protocol - Program Director	Program Director	11	1	11	3	33	$23.16	$764.28
TOTAL		4,701		28,444		8,404	$325.91	$81,744.73

^aData will be collected from women at four time points (intake, 6-months post-intake, discharge, and 6-months post-discharge), consistent with the GPRA data collection schedule. Figures in this table are based on 40 mothers per site with 2 children and 0.25 father/partner per mother. The schedule for collecting child data is similar to the mother’s with the addition of a 3-months post-intake time point with selected tools for a total of five time points. All child focused tools are completed by the mother or project staff, with the exception of CRAFFT. For fathers and partners, data will be collected at two points (intake and discharge).

^bMean hourly wages based on national estimates reported for profession in the Bureau of Labor Statistics' Occupational Employment Statistics, 2008. See http://www.bls.gov/oes/2008/may. For the purposes of these calculations, Project Staff & Counselor = Substance Abuse and Behavioral Disorder Counselor; Partner/Father (i.e., family member) = median hourly wage for all occupations reported; Medical Staff = Registered Nurse; Project Director & Program Director = Social Workers, Other; Clinical Director = Mental Health and Substance Abuse Social Worker.

^cBased on 8% of 880 minor children ages 11 to 17 at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^dBased on 16% of 880 minor children ages 12-35 months at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^eBased on 440 mothers having 2 minor children at intake and/or delivery.

^fBased on 44% of 880 minor children ages 2 to 17 at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^gBased on 95% of 880 minor children ages 1 month to 12 years (n=836). For simplicity, this calculation assumes that 95% of mothers have two children in this age group and complete the tool for each child at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^hBased on 37% of 880 minor children ages 3 to 17 at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

ⁱBased on 33% of 880 minor children ages 3 to 12 at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^jBased on 1 staff member at each of the 11 programs completing the tool for 80 children at discharge.

^kBased on 31% of 880 minor children ages 0-3 months at intake or delivery.

^lBased on 80 minor children per site ages 0 to 17 at intake at intake, 3 months, 6 months, discharge, and 6-months post-discharge.

^mBased on 1 staff member at each of the 11 programs completing items for 40 women at intake, 6 months, discharge, and 6-months post-discharge.

ⁿBased on 2 focus groups with 8 mothers at each site.

Table A-2 presents the summarized total annual respondent burden.

Table A-2. Summary Total Annual Respondent Burden
Respondent	Number of Respondents	Responses per Respondent	Total Responses	Hours per Response	Total Hour Burden	Hourly Wage Cost	Total Hour Cost
Mothers	440	---	19,756	---	6,700	$7.25	$48,575.00
Family Members	110	---	220	---	37	$15.57	$576.09
Children (11-17 yrs)	70	---	350	---	28	$0	$0.00
Medical Staff	11	---	550	---	91	$31.31	$2,849.21
Project Staff	11	---	7,480	---	1,416	$19.07	$27,003.12
Project Director	11	---	22	---	22	$23.16	$509.52
Clinical Director/Supervisor	22	---	22	---	44	$19.05	$838.20
Counselor	33	---	33	---	33	$19.07	$629.31
Program Director	11	---	11	---	33	$23.16	$764.28
TOTAL	719	---	28,444	---	8,404	---	$81,744.73

Note: Total number of respondents represents the number of each type of respondent that will be completing at least one tool across 11 sites over one year of data collection. The number of respondents (719) reported on this table differs from Table A-1 total number of respondents (4,701) which reflects completion of all tools across 11 sites over one year of data collection.

13. Estimates of Annualized Cost Burden to Respondents

There are no direct costs to respondents other than their time to participate. There are neither capital or startup costs nor are there any operation and maintenance costs.

14. Estimates of Annualized Cost Burden to the Government

The PPW Program has planned and allocated resources for the efficient and effective management and use of the information to be collected, including the processing of the information in a manner which shall enhance, where appropriate, the utility of information to the agencies and the public. The total average annual cost to the Federal government for the PPW Cross-site Evaluation is estimated to be approximately $1,968,462 which includes $1,931,719 evaluation costs plus $36,743 federal monitoring and information dissemination costs.

Evaluation costs include contract requirements for:

• Evaluation design;

• Instrument development and purchase;

• Training sites on data collection and data entry;

• Grantee support for PPW projects regarding evaluation efforts;

• Collection of project-level and client-level data from projects;

• Data cleaning, preparation of data files, and statistical and analytic support;

• Development of reports, and documentation and dissemination of findings.

Included in the Federal monitoring costs are those costs incurred by the government in personnel costs for oversight of the evaluation by one CSAT employee (GS-14) for approximately 20 percent of their time for $21,743 annually, and $15,000 for dissemination activities.

14. Changes in Burden

This is a new data collection.

14. Time Schedule, Publication and Analysis Plans

16.a. Time Schedule

The annualized schedule below shows when activities are estimated to occur in the months after approval of this data collection Evaluation for the 11 PPW grants in the second cohort.

Train sites & provide TA on data collection and data entry Month 1 & ongoing

Receive client data from sites Month 1 & ongoing

Data analysis Month 6 & ongoing

Validation of findings with projects Month 12 & 24 (or during

annual training meetings)

Final reports End of grant period

16.b. Publication Plans

The final reports will include a cumulative final report, a briefing book version, and materials for oral briefing(s). This report(s) will be distributed to those interested in the role of treatment and prevention of substance abuse among pregnant and perinatal women and the prevention and amelioration of its effect on minor children. In addition to these reports, we hope to make presentations at PPW trainings and other venues.

16.c. Analysis Plans

Our analysis plans are closely tied to the evaluation questions and conceptual framework, and reflect the anticipated realities of the data collection circumstances and the limited sample sizes of both sites and of participants in each site’s treatment programs.

The data items collected will be analyzed and first presented in reports using basic descriptive statistics and tabular displays. These analyses and report will serve CSAT’s needs for timely information to be used in the important program monitoring function that SAMHSA requires for performance reviews, improvement and oversight of the programs. Data can be aggregated to the project level and the summary statistics compared among the programs. Results may also be examined for subpopulations of interest such as age or intake problem-severity categories.

The information to be collected will enable CSAT to describe the demographic and other characteristics of clients entering PPW programs and to assess them at intake and subsequent time points on a series of measures that assess RFA outcomes including client substance use and abuse; safe and healthy pregnancies; maternal and child physical and mental health; women’s exposure to crime, violence, sexual and physical abuse; and child abuse and neglect. Basic frequency distributions and measures of central tendency and variability will be generated where appropriate, to permit examination of the overall distribution of the data and participant characteristics and the variations across the sites. Additional process data will be used to describe the projects interventions, staffing, utilization rates, and community contexts. Further analyses of both program process and outcomes will be conducted as appropriate when sufficient data have been collected and as the interests and needs of CSAT and other stakeholders are further specified.

More specific analysis specifications include the following:

Descriptive. Various descriptive statistics such as location (mean, median, mode), dispersion (range, standard deviation, etc.), and moments (variance, skewness, etc.) will be used for the continuous variables as appropriate to describe PPW participants and outcome measures. Frequencies and/or contingency tables (cross-tabs) are will also be used describe and analyze the relationships among the categorical variables.

Basic multivariate analyses. Basic multivariate modeling will be conducted to examine the extent to which the goals of the PPW program are addressed by grantees. This will include identifying the types of measures to incorporate into the models--both the form of the dependent measure and categories of predictors to use. For categorical outcomes such as abstinence we will use logistic regression and we will use ordinary least squares (OLS) multiple regression for continuous measures. For dependent measures that are highly skewed we will use more specialized approaches such as Poisson regression that reflect the underlying distribution.

Multilevel longitudinal modeling. Multilevel models are well suited to sort out the cause-and-effect relationships between variations PPW program activities and client outcomes. By properly adjusting standard errors for within-program clustering of clients and the serial correlation of longitudinal outcomes, they will also increase the confidence with which observed changes can be attributed to the implementation of PPW-supported activities. This in turn will lead to better-grounded recommendations for improving PPW program effectiveness in the future. As in other analyses of cross-site evaluations,^31,32,33,34 these methods take account of the “nested” character of the longitudinal outcome data within clients, as well as nesting of clients within sites.³⁵ The approach can also reveal sites that, for any reason, are discrepant from others with similar characteristics (“outliers” in the distributions of outcomes) through graphic display of estimates and residuals. This can be the basis for beginning further analyses of the reasons for such discrepancies, which may involve values of other variables available in the data but not included in the multilevel model, or point the way to other, more global characteristics highlighted only in more qualitative data on the PPW programs. Modeling will be performed using Hierarchical Linear Modeling (HLM) Version 6.³⁶

Case-Mix adjustment. Case-mix adjustment and subgroup analyses will be conducted to compare the outcomes client and grantee groups. The goal of case-mix adjustment is to adjust statistically for measured differences in client’s pre-treatment characteristics so that is possible to make more valid comparisons of outcomes across grantees whose clients have different “mixes” of characteristics and levels of substance abuse severity.

Case-mix adjustment methods have been applied widely in evaluations of physical and mental health treatments, where they are also called “risk adjustment” methods.³⁷ Applications of case-mix adjustment developed especially for substance abuse treatment have become fairly common in published studies in recent years.^38,39,40 These types of analyses require a minimum number of clients and grantees and can only be conducted if these requirements are met.

Propensity scoring for constructed benchmark groups. The identification of groups to be compared will follow on the description and analysis of the PPW programs and models across sites. Propensity scoring^41,42 frees the matching process from its usual limitation of reliance on a small number of covariates and simple functional forms (linear main effects only). When used to compare groups, the propensity score carries all the information from the complex covariate model in a single variable, consuming only one degree of freedom. Thus the method can handle a large number of confounding variables, yet avoids the potentially adverse effects of multicollinearity on the stability of the estimates.

1. Display of Expiration Date

The expiration date for OMB approval will be displayed on all data collection instruments for which approval is being sought.

1. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The estimated universe of individual respondents includes women and their minor children, as well as fathers and/or partners of the women entering treatment across the 11 grant projects. All clients entering treatment will be asked to participate in the Evaluation. For each project, the starting point for data collection is at intake – that is, the time when the mother’s intake GPRA is completed. Women who leave the program at any time prior to treatment completion are targeted along with those who complete the program.

Based on other related projects currently reporting GPRA data to CSAT, it is expected that most clients will not refuse to participate in the program, thus PPW projects should not have a problem reaching the 80 percent response rates for intake or followup because of client refusal.

In addition, as described in B.3, response rates will be maximized by maintaining contact with participants through social relationships and relatives as well as by the use of incentives for followup. CSAT will work with project directors and senior staff to assure that projects understand the need for accurate and timely followup of all clients.

2. Information Collection Procedures

Table B-1 summarizes the data collection schedule for each instrument, method of administration, by whom, and the source of the information.

Table B-1. Data Collection Schedule

Data Collection Tools	Respondent (and child age if applicable)		Data Collector	Data Collection Method		Data Collection Time
				Interview	Record Audit	Intake/ Birth	3-mos post-intake	6-mos post-intake	Discharge	6-mos post-discharge
GPRA	Mother					▲		▲	▲	▲
BASIS-24®	Mother		Project Staff	▲		▲		▲	▲	▲
Ferrans & Powers Quality of Life Index	Mother Father/Partner at intake & discharge		Project Staff	▲		▲		▲	▲	▲
Women’s Discharge Tool	Staff		Project Staff		▲ (mother’s)				▲
Family Support Scale	Mother		Project Staff	▲		▲		▲	▲	▲
Child Abuse Potential Inventory	Mother		Project Staff	▲		▲		▲	▲	▲
Staff Completed Women’s Items	Staff/Mother		Project Staff	▲	▲ (mother’s)	▲		▲	▲	▲
Items Administered to Women	Mother		Project Staff	▲		▲		▲	▲	▲
CHILD
Child Data Collection Tool	Mother (0-17 yrs)		Child Specialist, Medical Staff	▲	▲ (child’s)	▲
Children's Discharge Tool	Staff (0-17 yrs)		Project Staff		▲ (child’s)				▲
Newborn’s Medical Record Audit	Staff (0-3 mos)		Medical Staff		▲ (child’s)	▲
Parenting Stress Index	Mother (1 mo-12 yrs)		Project Staff	▲		▲	▲	▲	▲	▲
Brief Infant Toddler Social and Emotional Assessment	Mother (12-35 mos)		Project Staff	▲		▲	▲	▲	▲	▲
Parenting Relationship Questionnaire	Mother (2-17 yrs)		Project Staff	▲		▲	▲	▲	▲	▲
Trauma Symptom Checklist for Young Children	Mother (3-12 yrs)		Project Staff	▲		▲	▲	▲	▲	▲
Social Skills Improvement System	Mother ( 3-17 yrs)		Project Staff	▲		▲	▲	▲	▲	▲
CRAFFT	Child (11-17 yrs)		Project Staff	▲		▲	▲	▲	▲	▲
Staff Completed Newborn Items	Treatment Program Staff		Medical Staff		▲ (child’s)	▲
Staff Completed Child Items	Mother/Staff (0-17 yrs)		Project Staff	▲	▲ (child’s)	▲	▲	▲	▲	▲

INSTRUCTIONS

Please collect data from all new admissions. If a client is readmitted to the program, keep original ID but start data collection and followup times fresh.

DATA COLLECTION WINDOWS

Table 1. Summary of Data Collection Windows

	Intake – mother only	Intake/Birth – children, partners, fathers	*Followup & Discharge
PPW-Specific Tools	7 days post-intake GPRA	30 days after mother’s intake GPRA	14 days prior, 14 days after

*NOTE. Followup is based on the date the mother’s intake GPRA was completed. Maternal intake = date mother’s intake GPRA completed.

PPW-Specific Tools—Tools specific to the PPW Cross-site Evaluation are to be administered, depending on the tool, at maternal intake GPRA time (or child’s birth, if child born while mother is in treatment); at 3-months post-intake, 6-months post-intake (regardless of whether client still receiving services); at discharge; and at 6-months post-discharge, as summarized in Table 1.

• Admission

  • For women, data must be collected within 7 days of the woman’s intake GPRA.

  • For all minor children (i.e., under 18 years) receiving services (regardless of whether or not they reside in the facility), data must be collected within 30 days of the mother’s intake GPRA or within 30 days of the child’s birth (if child born while mother is in treatment).

  • For the rare cases when children do not receive services until after the initial 30 days, administer all intake tools within 30 days of their initial receipt of services, but do not collect followup data until discharge.

• Followup

  • The allowable window period for all followup interviews is 14 days before to 14 days after to the date the followup is due.

  • The precise “due date” for the mother’s, children’s, fathers’, and partners’ followup is based on the mother’s precise intake GPRA and discharge dates – that is, 3- and 6-months after maternal intake GPRA and 6-months after maternal discharge.

  • Followup data collection for newborns born while the mother is in treatment is 3 and 6 months after the infant’s delivery date. Discharge for these infants will follow the maternal discharge date.

• Discharge—The allowable window period for all followup interviews is 14 days before to 14 days after maternal discharge. If a client’s discharge data collection window overlaps with a followup (or intake) data collection window, the tool should only be administered once and will be used for both that data collection point and for discharge. Children, partners, and fathers follow the maternal discharge date.

GPRA—In compliance with the Government Performance and Results Act (GPRA), CSAT’s Discretionary Services grantees are required to collect data at admission, six months post admission and at discharge. Admission data must be collected within 7 days of admission to treatment. With respect to the six month post admission data collection point, grantees are required to obtain an 80% follow up rate.  There is an allowable window period for this data collection point of 30 days prior to the date the follow up is due to 60 days after the date the follow up is due.

Data for the Cross-site Evaluation will be collected a maximum of five points in time: intake (or for children, within 30 days of the mother’s GPRA intake) or within 30 days of an infant’s birth; 3 months post-intake (for child instruments only); 6 months post-intake; discharge; and 6 months post-discharge. In some cases, the child’s primary residence may be with someone other than the mother, and the mother may not be able to provide the answer to all questions about a child. In these cases, program staff may support the mother in gathering that information from others in the child’s life. In many cases, this approach would have the added clinical benefit of helping the mother learn more about her child.

Data will be collected by intake workers, counselors, medical staff, child specialists, and/or other project staff, as is standard practice. In instances where participants are no longer in direct contact with the service provider, staff from the program will locate participants using a variety of tracking techniques. Followup interviews are conducted in person whenever possible. CSAT will work closely with the projects to establish client tracking protocols, to assure maintenance of privacy during tracking, and to train project staff in tracking methodology using strategies that have worked effectively for GPRA data collection.

The instruments and associated interviewer instructions can be found in Attachments A-E. Projects will be given TA and training in the administration of all aspects of the data collection procedures. CSAT and its contractors will provide training on all the instruments involved in the Evaluation and assist projects in identifying appropriate persons to receive training. In addition, CSAT will provide for TA throughout the Evaluation. For data abstraction from records review, CSAT will work with projects to develop procedures to abstract the required data elements from their existing files. In some cases, the instruments may be more appropriately administered in a language other than English. If such occasions arise, TA will be provided, if necessary, to ensure that foreign language assistance is available.

As shown in Table B-1, client-level data collection methods include interviews and records audit and extraction.

The primary data collection activities for the process evaluation component are site visits and biannual project director telephone interviews. Two rounds of site visits will be conducted approximately 14-16 months apart beginning in fall 2010 (pending OMB approval). During each site visit, interviews will be conducted with the following staff: program director; director of clinical services for women and for children; and at least two treatment providers (e.g., counselors, case managers). In addition, at least one focus group will be conducted with 6-8 clients at each grantee site visit.

Biannual project director telephone interviews will be conducted every six months over the course of the Evaluation. CSAT and its contractors will schedule interviews with each grantee project director after they have submitted their biannual progress report.

3. Methods to Maximize Response Rates

We aim to have a response rate for women of 80 percent or higher. We believe this is possible given that a prior version of PPW evaluation achieved a response rate of 87.8 percent. (This figure is based on the number of cases that had both GPRA and evaluation data and assumes 100 percent response rate for GPRA, which may be a slight overestimate.) To maximize the response rate for this Evaluation, we have developed an electronic system that notifies grantees weekly as to when Evaluation intake and follow-up interviews are due for each instrument based on GPRA intake and discharge dates. Several additional methods will be used to retain participants, maximize response rates, and to optimize data completeness:

1. Clients will be asked to sign a consent or assent form during their orientation to the program. The intrinsic value of their participation in the data collection for their own treatment and for the future treatment of other women will be stressed.

2. Information will be gathered from the women on next of kin, close friends, or other emergency contacts. This information will be used when necessary to follow up on women who drop out of the program or who otherwise become difficult to reach and will be updated with each round of data collection, including discharge. Detailed contact information will be collected for each client and will be updated at each data collection stage.

3. Each client will be asked which contact methods are acceptable to her to arrange for the in-person followup interview, including best times to call, best days to call, and where the in-person interview can best take place. This information will be updated during all post-treatment contacts.

4. Clients will be reminded of the followup interview during the discharge interview. Project staff that makes regular contact with clients after discharge also will offer a reminder of the followup date at each contact.

5. Centralizing the data entry at one contractor will reduce the reporting burden on projects and improves the quality and completeness of data by allowing the contactor to resolve errors and inconsistencies in the data before the data set is finalized and analyzed.

6. Clients will be engaged in residential treatment to the maximum extent possible so that early dropout is minimized.

7. Projects are encouraged to offer cash equivalents to participate as a means of retaining their cooperation with the followup data collection effort and for fathers/partners. Projects will be advised to offer a $20 cash equivalent to each participant who completes the followup data collection.

8. Low-cost tracking procedures will be employed during the period between discharge and followup. GPRA tracking procedures will be used and have been shown to be effective.

CSAT will implement several strategies to assist the projects with followup activities. First, CSAT has conducted training for GPRA followup. The training program is designed to assist projects in learning about and conducting the followup at their sites and is offered to all projects. Individual project TA is made available for sites that need additional followup instruction. It is anticipated that these strategies will continue to improve the followup rates.

4. Tests of Procedures

The following instruments were previously approved by OMB (OMB No. 0930-0269) and used successfully under the prior cross-site assessments of the PPW program with the FY 2003, 2006, and 2008 PPW grantees. Their development is briefly described below:

• Child Data Collection Tool – Several iterations of this instrument were developed and shared with women’s treatment experts in the field. Initial drafts of this tool were updated based upon review and feedback from these experts and used with past PPW grantees. Preliminary scale development revealed Chronbach’s alphas from .66 to .78 for the following subscales: Educational (.66), Socioeconomic Status (.72), Legal (.74), Parental Relationships (.75), and Spiritual (.78).

• Women’s Discharge Tool and Children’s Discharge Tool – These instruments are based on standardized discharge questions/categories that are gleaned from a records review by project staff. They were developed based on standard items typically collected at discharge. Data are based on staff review of clients’ medical records.

• Newborn’s Medical Record Audit – These instruments are based on elements that are typically found in medical records as part of standard medical practice. Responses are gleaned from a medical records review by medical staff. The Newborn’s Medical Record Audit captures basic birth outcomes of the newborn.

The following proprietary instruments are being used in the field by organizations other than SAMHSA.

• BASIS-24® (mothers) ─ The psychometric properties of the BASIS 24® have been demonstrated for White, Latino, and African American clients in large inpatient and outpatient mental health and substance abuse treatment samples (Eisen, Gerena, Ranganathan, Esch, & Idiculla, 2006).⁴³ Specifically, Chronbach’s alpha among these groups ranged from .87 to .91 for the overall summary score.

• Ferrans and Powers Quality of Life Index (mothers and their partners and children’s fathers) ─ Based on numerous studies, Chronbach’s alphas have ranged from .73 to .99 overall and .70 to .94 for the subscales. Test-retest reliability has also been good at .81 for a one-month interval. Face validity, content validity, and construct validity (by using factor analysis) have also been established (Ferrans & Powers, 1992).⁴⁴ For the first cohort, the internal consistency was .94 for the overall instrument. The subscale alphas were adequate (i.e., .88 for Health and Functioning, .77 for Social and Economic, and .90 for Psychological-Spiritual), except for the Family Subscale, which was low (.65).

• Family Support Scale (mothers) ─ This scale measures parents’ satisfaction with the support they receive in raising young children. Internal consistency reliability is .77 while split-half reliability is slightly lower at .75. Validity is evidenced by comparisons between results on the Family Support Scale and the Parent-Child Play Scale which yielded consistent but weak correlations.

• Child Abuse Potential Inventory (mothers) ─ Overall, the 77-item Child Abuse Potential (CAP) Inventory has high internal consistency reliabilities (i.e., .92-.96 for controls and .95-.98 for abusers); temporal stability estimates for the abuse scale also are adequate (i.e., .91 and .75 for one-day and three-month intervals, respectively). Instrument contains three validity scales including lie, random response, and inconsistency.

• Parenting Stress Index (children 1 month to 12 years) ─ The Parenting Stress Index Short Form consists of 3 scales: Parental Distress, Parent-Child Dysfunctional Interaction, & Difficult Child. Coefficient alpha reliability coefficients were calculated for each subscale, domain, and Total Stress score. Parenting Stress Index was found to maintain its validity across diverse cultures. Alphas range from .70 to .83 for children and .74 to .83 for parents. Test-retest reliability is .63 for children and .91 for parents. A body of validity research includes correlations with other scales (Bayley), diagnoses, and behavior problems.

• Brief Infant Toddler Social and Emotional Assessment (BITSEA) (children 2 to 17 years) ─ BITSEA is used in identifying children ages 12 months to 35 months 30 days who may have social-emotional and behavioral problems and/or delays, or deficits in social-emotional competence. Validation studies demonstrate sensitivity to subtle growth and developmental stages of infants. Multiple reliabilities are reported for BITSEA. Internal consistency reliability ranges from .65 to .80, inter-rater reliability ranges from .61 to .68 and test-retest reliability ranges from.85 to .87. Predictive validity is .71.

• Parenting Relationship Questionnaire (children 2 to 17 years) ─ Two forms of reliability have been reported including internal consistency and test-retest. Internal consistency is measured with coefficient alpha statistic with alphas above .80. Median alpha for parents is .82 and .76 for child/adolescent scales. Test-retest reliability ranges from .72 to .89. Instrument includes validity indexes to detect exaggerated or careless responding.

• Trauma Symptom Checklist for Young Children (TSCYC) (children 3 to 12 years) ─ Average clinical alpha coefficient is .86 with a range of .78 to .92. Similar results were found in clinical and child abuse treatment samples. Homogeneity-corrected test-retest correlation coefficients for TSCYC scales ranged from .68 to .96, with a median coefficient of .88. Discriminant, predictive, and construct validity have been demonstrated for the TSCYC in multiple samples and studies in predicting trauma exposure.

• Social Skills Improvement System (SSIS) (children 3 to 17 years) ─ Three forms of reliability have been used including internal consistency (coefficient alpha used), test-retest reliability, and inter-rater reliability. Median scale alphas range in the .90s. Test-retest reliability is slightly lower at .84. Collection of reliability and validity evidence exists as evidenced by correlations with other measures and consistency with research on special populations.

• CRAFFT (children 11 to17 years) ─ The documented reliability is .68 and criterion-related validity is .72 with strong scores for sensitivity (.80) and specificity (.86).⁴⁵

The following brief lists of items were developed for this Evaluation to fill in gaps in the other proposed tools. They are briefly described below:

• Staff Completed Women’s Items (mothers) – These four items include whether the woman is pregnant, postpartum or both; if pregnant, what trimester; if pregnant, problems experienced during pregnancy; if postpartum, pregnancy outcome; and if a follow-up was not obtained, why not. A combination of these items is administered at intake, 6 months post-intake, discharge, and 6 months post-discharge.

• Items Administered to Women (mothers) – These six items include number of children residing at the treatment facility with the mother; tobacco use; exposure to physical and sexual violence; and at 6 months follow-up, services obtained after being discharged from the residential treatment program. A combination of these items is administered at intake, 6 months post-intake, discharge, and 6 months post-discharge.

• Staff Completed Newborn Items (children birth to 3 months) – These six items are administered at intake only and completed at the same time as the Newborn’s Medical Record Audit. These items include whether the child was born premature; newborn date of birth; newborn hospital discharge date and time; newborn length of stay in hospital; and questions about NICU stay and neonatal abstinence syndrome. These items are based on elements that are typically found in medical records as part of standard medical practice. Responses are gleaned from a medical records review by medical staff.

• Staff Completed Child Items (all children) – These four items include child gender; child age; child’s primary residence; whether the child was born premature; whether the child resided with the mother during treatment; whether the child visited the mother during treatment; and if a follow-up was not obtained, why not. A combination of these items is administered at intake, 3 months post-intake, 6 months post-intake, discharge, and 6 months post-discharge.

Pre-tests of client-level additional items and process evaluation measures were conducted with fewer than nine participants in February 2010. Minor changes were made to each of these tools as a result of these pre-tests. All of these changes were focused on making the interview process easier and less burdensome for the respondent. For the client-level additional items, changes were made to the wording of two questions to clarify the information being requested, and an additional response category of “Not Applicable” was added to several questions.

For the process evaluation tools, minimal changes resulting from pre-testing were made to the wording of several questions and some questions were deleted. The changes made to the biannual project director telephone interview focused on making the wording of a few questions more conversational for the interviewer and respondent. The changes made to the process evaluation site visit tools (program director interview, clinical director interview, counselor interview, client focus group protocol) all focused on modifying the language of some items and refining response categories to clarify the information being requested. Additionally, some questions in the client focus group protocol were deleted based on feedback from pilot test respondents that these questions were redundant.

5. Statistical Consultants

The individuals listed below reviewed statistical aspects of this Evaluation and any differences were reconciled. The names and phone numbers of the statistical consultants are as follows:

Statistical Consultants for the PPW Evaluation
Name	Address	Contact Information
Wendy Kissin, Ph.D.	Westat, 1600 Research Boulevard Rockville, MD 20850	Phone: 301-294-3885 [email protected]
Robert Orwin, Ph.D.	Westat, 1600 Research Boulevard Rockville, MD 20850	Phone: 301-251-2277 [email protected]
Joseph Sonnefeld, M.A.	Westat, 1600 Research Boulevard Rockville, MD 20850	Phone: 240-314-2522 [email protected]
Project Officer for the PPW Evaluation
Deepa Avula	SAMHSA,CSAT, OESAS, 1 Choke Cherry Rd., Room 5-1118, Rockville, MD 20857	Phone: 240-276-2961 [email protected]

ATTACHMENTS

Attachment A-1: Instruments for Mothers

A-1.1 Brief Infant Toddler Social and Emotional Assessment

A-1.2 Child Data Collection Tool

A-1.3 Parenting Relationship Questionnaire

A-1.4 Parenting Stress Index

A-1.5 Social Skills Improvement System

A-1.6 Trauma Symptom Checklist for Young Children

A-1.7 BASIS-24®

A-1.8 Child Abuse Potential Inventory

A-1.9 Family Support Scale

A-1.10 Ferrans and Powers Quality of Life Index

A-1.11 Items Administered to Women

A-1.12 Site Visit Protocol-Client Focus Group

Attachment A-2: Supporting Documents for Mothers

A.2.1 Consent Form

A.2.2 Informed Consent Form-Focus Groups

Attachment B-1: Instrument for Children

B-1.1 CRAFFT

Attachment B-2: Supporting Document for Children

B-2.1 Assent Form

Attachment C-1: Instrument for Partners/Fathers

C-1.1 Ferrans and Powers Quality of Life Index

Attachment C-2: Supporting Documents for Partners/Fathers

C.2.1 Consent Form

Attachment D: Instruments for Project Staff

D-1 Children’s Discharge Tool

D-2 Women’s Discharge Tool

D-3 Staff Completed Women’s Items

D-4 Staff Completed Child Items

Attachment E: Instruments for Medical Staff

E-1 Newborn’s Medical Record Audit

E-2 Staff Completed Newborn Items

Attachment F: Instrument for Project Director

F-1 Biannual Project Director Telephone Interview

Attachment G: Instrument for Clinical Director/Supervisor

G-1 Site Visit Protocol-Clinical Director/Supervisor Interview

Attachment H: Instrument for Counselor

H-1 Site Visit Protocol-Counselor Interview

Attachment I: Instrument for Program Director

I-1 Site Visit Protocol-Program Director Interview

Attachment J: PPW Evaluation Materials

J-1 CSAT’s 2008 Request for Applications (RFA)

J-2 PPW Evaluation Logic Model

J-3 Overview of PPW Evaluation Design

Attachment K: 60-day Federal Register Notice (FRN)

K-1 60-day FRN Comment

K-2 SAMHSA’s Response to 60-day FRN Comment

1 Clark, K., Dee, D., Bale, P., and Martin, S. (2001). Treatment compliance among prenatal care patients with substance abuse problems. American Journal of Drug and Alcohol Abuse, 27(1), 121-136.

2 SAMHSA (2009a). Results from the 2008 National Survey on Drug Use and Health: National findings. Rockville, MD: Office of Applied Studies, NSDUH Series H-36, HHS Publication No. SMA 09-4434.

3 Illicit drug use includes use of: marijuana, cocaine, heroin, hallucinogens, and inhalants and the nonmedical use of prescription-type pain relievers, tranquilizers, stimulants, and sedatives (SAMHSA, 2009a).

4 SAMHSA (2009a). Results from the 2008 National Survey on Drug Use and Health: National findings. Rockville, MD: Office of Applied Studies, NSDUH Series H-36, HHS Publication No. SMA 09-4434

5 White, R., Thompson, M., Windsor, D., Walsh, M., Cox, D., & Charnaud, B. (2006). Dexamphetamine substitute-prescribing in pregnancy: a 10-year retrospective audit. Journal of Substance Use, 11, 205-216.

6 Anega, A., Igbal, M. M., & Ahmed, K. (2006). The effects of amphetamine use during pregnancy and lactation. Directions in Psychiatry, 26, 237-251.

7 SAMHSA (2009b). The language of Fetal Alcohol Spectrum Disorders. U.S. Department of Health and Human Services.

8 SAMHSA (2009b). The language of Fetal Alcohol Spectrum Disorders. U.S. Department of Health and Human Services.

9 Barnard, M. & McKeganey, N. (2004). The impact of parental problem drug use on children: What is the problem and what can be done to help? Addiction, 99, 552-559.

10 Kumpfer, K. L. & Fowler, M. A. (2007). Parenting skills and family support programs for drug-abusing mothers. Seminars in Fetal and Neonatal Medicine, 12, 134-142.

11 Barnard, M. & McKeganey, N. (2004). The impact of parental problem drug use on children: What is the problem and what can be done to help? Addiction, 99, 552-559.

12 Frame, L. (2002). Maltreatment reports and placement outcomes for infants and toddlers in out-of-home care. Infant Mental Health Journal, 23, 517-540.

13 Metsch, L. R., Rivers, J. E., Miller, M., Bohs, R., McCoy, C. B., & Morrow, C. J. (1995). Implementation of a family-centered treatment program for substance-abusing women and their children: Barriers and resolutions. Journal of Psychoactive Drugs, 27, 73-83.

14 Metsch, L. R., Rivers, J. E., Miller, M., Bohs, R., McCoy, C. B., & Morrow, C. J. (1995). Implementation of a family-centered treatment program for substance-abusing women and their children: Barriers and resolutions. Journal of Psychoactive Drugs, 27, 73-83.

15 Jessup, M. A., Humphreys, J. C., Brindis, C. D., & Lee, K. A. (2003). Extrinsic barriers to substance abuse treatment among pregnant drug dependent women. Journal of Drug Issues, 33, 285.

16 SAMHSA (2009a). Results from the 2008 National Survey on Drug Use and Health: National findings. Rockville, MD: Office of Applied Studies, NSDUH Series H-36, HHS Publication No. SMA 09-4434.

17 Arria, A.M., Derauf, C., LaGasse, L., et al. (2006). Methamphetamine and other substance use during pregnancy: Preliminary estimates from the infant development, environment, and lifestyle (IDEAL) study. Maternal and Child Health Journal; 10(3); 293-302.

18 Substance Abuse and Mental Health Services Administration (SAMHSA) (2008). Results from the 2007 National Survey on Drug Use and Health: National findings. Rockville, MD: Office of Applied Studies, NSDUH Series H-36, HHS Publication No. SMA 09-4434.

19 ^?Arria, A.M., Derauf, C., LaGasse, L., et al. (2006). Methamphetamine and other substance use during pregnancy: Preliminary estimates from the infant development, environment, and lifestyle (IDEAL) study. Maternal and Child Health Journal; 10(3); 293-302.

20 Aneja, A., Igbal, M.M., & Ahmed, K. (2006). The effects of amphetamine use during pregnancy and lactation. Directions in Psychiatry, 26(3); 237-251.

21 White, R., Thompson, M., Windsor, D., Walsh, M., Cox, D., & Charnaud, B. (June 2006). Dexamphetamine substitute-prescribing in pregnancy: A 10-year retrospective audit. Journal of Substance Use, 11(3); 205-216.

22 Infant Mortality and Low Birth Weight Among Black and White Infants – United States, 1980-2000. (July 2002). Center for Disease Control MMWR Weekly, 51(27); 589-92. See http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5127a1.htm

23 SAMHSA (2009b). The language of Fetal Alcohol Spectrum Disorders. U.S. Department of Health and Human Services.

24 http://www.cdc.gov/ncbddd/fasd/data.html

25 Clark, K., Dee, D., Bale, P., and Martin, S. (2001). Treatment compliance among prenatal care patients with substance abuse problems. American Journal of Drug and Alcohol Abuse, 27(1), 121-136.

26 The Urban Institute. (2003). “Finding Out What Happens to Former Clients,” Series on Outcome Management for Nonprofit Organizations.

27 Brigham, G., Winhusen, T., Lewis, B., and Kropp, F. (2010). Incentives for retention of pregnant substance users: A Secondary Analysis. Journal of Substance Abuse Treatment, 38, 90-95.

28 http://www.drugabusestatistics.samhsa.gov/NHSDA/2k1NHSDA/vol2/attachmentc.htm

29 http://www.dol.gov/whd/minimumwage.htm

30 http://www.bls.gov/oes/2008/may/oes_nat.htm#b00-0000

31 Orwin, R.G., Goldman, H.H., Sonnefeld, L.J., Ridgely, M.S., Garrison-Mogren, R., and O’Neill, E. (1994). Alcohol and drug abuse treatment of homeless individuals: Results from the NIAAA Community Demonstration Program. Journal of Health Care for the Poor and Underserved, 5(4), 326-352.

32 Orwin, R.G. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95, 5309-5328.

33 Orwin, R.G., Hornik, R.C., Judkins, D., Zador, P., Sridharan, S., and Baskin, R. (2004). Innovative design and analysis strategies in the evaluation of the National Youth Anti-Drug Media Campaign: Propensity scores and counterfactual projection weights in a national probability survey. Proceedings of the Federal Committee on Statistics and Methodology. Available at: http://www.fcsm.gov/03papers/Orwin.pdf.

34 Orwin, R.G., and Sonnefeld, L.J. (2005). Multilevel Approach to Relating System Integration, Service Delivery, and Outcomes. Presented at American Evaluation Association, Toronto, ON, Canada, Oct.

35 Murray, D.M. (1998). Design and analysis of group-randomized trials. New York: Oxford University Press.

36 Raudenbush, S.W., Congdon, C., and Bryk, A.S. (2004). HLM6: Hierarchical Linear and Nonlinear Modeling. Lincolnwood, IL: Scientific Software International.

37 Iezzoni L, Risk Adjustment for Measuring Health Care Outcomes, 3rd Ed. (2003).

38 Hoffman, N.G., Floyd, A.S., Zywiak, W.H., & DeHart, S. (1999). Strategies for case-mix adjustments in addictions treatment evaluations: Prognostic indicators in public sector populations. Report prepared under CSAT Contract 270-95-0023.

39 Koenig, L.; Fields. E.L., Dall, T.M., Ameen, A.Z., & Harwood, H.J. (2000). Using case-mix adjustment methods to measure the effectiveness of substance abuse treatment: Three examples using client employment outcomes. Rockville, MD: Substance Abuse and Mental Health Services Administration (DHHS/PHS), Center for Substance Abuse Treatment.

40 Phibbs, C.S., Swindle, R.W., & Recine, B. (1997). Does Case Mix Matter for Substance Abuse

Treatment? A Comparison of Observed and Case Mix-Adjusted Readmission Rates for

Inpatient Substance Abuse Treatment in the Department of Veterans Affairs, Health Services

Research, 31(6), 755-771.

41 Rosenbaum, P.R., and Rubin, D.B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524.

42 Rosenbaum, P.R. (2002). Observational studies. New York: Springer.

43 Eisen, S.B., Gerena, M., Ranganathan, G., Esch, & Idiculla, T. (2006). Reliability and Validity of the BASIS-24© Mental Health Survey for Whites, African-Americans, and Latinos. The Journal of Behavioral Health Services & Research, 33(3,) 304-323.

44 Ferrans, C., & Powers, M. (1992). Psychometric assessment of the Quality of Life Index. Research in Nursing and Health, 15, 29-38.

45 Knight, J., Sherritt, L., Shrier, L., Harris, S.K., & Chang, G. (2002). Validity of the CRAFFT Substance Abuse Screening Test Among Adolescent Clinic Patients. Archives of Pediatrics and Adolescent Medicine, 156, 607-614.

16