Acyf-cb-pi-19-06

ACF

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration on Children, Youth and Families

Administration

1. Log No: ACYF-CB-PI-19-06

for Children

3. Originating Office: Children's Bureau

and Families

4. Key Words: Family First Prevention Services Act, State Title IV-E
Prevention and Family Services and Programs, Title IV-E Prevention
Program

2. Issuance Date: July 18, 2019

PROGRAM INSTRUCTION
TO: State Agencies Administering or Supervising the Administration of Titles IV-B and IV-E
of the Social Security Act
SUBJECT: Transitional Payments for the Title IV-E Prevention and Family Services and
Programs
LEGAL AND RELATED REFERENCES: 471(e)(4)(C) of the Social Security Act (the Act);
ACYF-CB-PI-18-09
PURPOSE: To instruct states on the procedures for transitional payments for title IV-E
prevention services and programs.
BACKGROUND: Section 474(e)(4)(C) of the Act requires that title IV-E prevention services
must be rated through an independent systematic review of evidence as promising, supported, or
well-supported in accordance with Department of Health and Human Services (HHS) criteria and
be approved by HHS, and that HHS establish a public clearinghouse of rated practices. In
ACYF-CB-PI-18-09 (State Requirements for Electing Title IV-E Prevention and Family Services
and Programs), we indicated that the Title IV-E Prevention Services Clearinghouse (the
Clearinghouse) would conduct this review and issue ratings. We are issuing this PI to provide
instruction that allows a state to claim transitional payments for services and associated costs
under the title IV-E prevention program until the Clearinghouse can review and rate a program
or service, if a state submits sufficient documentation as outlined in this PI by October 1, 2021.
States must conduct independent systematic reviews of prevention services and programs to
complete this documentation. The Clearinghouse will make the final determination about
whether a program or service is assigned a promising, supported, or well-supported rating. We
intend that this process for transitional payments will support ACF’s goal to review programs
and services as quickly as possible in the early years of implementation of the title IV-E
prevention program.

1

INSTRUCTION: A state must complete and submit the checklist in Attachment B, with all
required documentation, to request transitional payments for a title IV-E prevention program or
service which has not yet been rated by the Clearinghouse. The state must submit this checklist
as part of the five-year plan, or as an amendment to an approved five-year plan by October 1,
2021. The checklist documents that, in determining the state’s designation(s) of promising,
supported, or well-supported, for HHS consideration, the state: 1) conducted the independent
systematic review; and 2) met the criteria outlined in section 471(e)(4)(C) of the Act and
Attachment C to ACYF-CB-PI-18-09.
Once a state’s program(s) or service(s) is approved as part of the five-year plan, any other state
may submit a five-year plan for approval of a transitional payment for those same programs or
services, but must submit the plan or amendment by October 1, 2021 (see section 471(e)(5) of
the Act; section B.1 of the state title IV-E prevention program five-year plan pre-print).
All other requirements for the title IV-E prevention program outlined in ACYF-CB-PI-18-09
remain in effect for transitional payments. This includes, for example, the title IV-E plan
approval requirements, maintenance of effort requirement, financial reporting on the Form CB496, trauma-informed service delivery, and the requirement that at least 50 percent of the
amounts expended by the state for a fiscal year for the title IV-E prevention program must be for
services that meet the well-supported practice criteria. However, HHS will not approve a waiver
of the evaluation requirement in section 471(e)(5)(C)(ii) of the Act, unless the Clearinghouse has
rated the program as well-supported (and the state has submitted documentation assuring that the
evidence of the effectiveness of the practice is compelling and the state meets the continuous
quality improvement requirements included in section 471(e)(5)(B)(iii)(II) of the Act).
We strongly encourage states to follow the procedures in the Title IV-E Prevention Services
Clearinghouse Handbook of Standards and Procedures (the Handbook), which was developed
specifically to meet the independent systematic review, practice, and other requirements of
section 471(e)(4)(C) of the Act. A state may use standards and procedures other than those
described in the Handbook to demonstrate that the criteria in section 471(e)(4)(C) of the Act and
Attachment C to ACYF-CB-PI-18-09 were met as outlined in section II of Attachment B.
Regardless of the procedures the state uses for the independent systematic review, the
Clearinghouse will use the Handbook procedures to make the final determination of the rating
for the program or service.
The following applies to title IV-E transitional payment claims for prevention services and
associated costs upon the Clearinghouse assigning a rating:
 Once the Clearinghouse rates a program or service with the same or higher designation as
the designation approved as part of the state title IV-E plan, the Clearinghouse rating
becomes effective and the state may continue to claim title IV-E prevention services and
associated costs.
 If the Clearinghouse does not rate a program or service as meeting the promising,
supported, or well-supported criteria, HHS will make transitional payments for such

2

services (and associated costs) provided only through the end of the Federal fiscal quarter
following the Federal fiscal quarter in which the Clearinghouse rating was assigned.
 If the Clearinghouse rates a program or service with a lower designation than the
designation approved as part of the state title IV-E plan, the transitional payments
designation will remain in effect through the end of the Federal fiscal quarter following
the Federal fiscal quarter in which the Clearinghouse rating was assigned.
PAPERWORK REDUCTION ACT: Under the Paperwork Reduction Act of 1995 (Public
Law 104-13), an agency may not conduct or sponsor, and a person is not required to respond to,
a collection of information unless it displays a currently valid OMB control number. The OMB
control number of the Title IV-E Plan Pre-Print is 0970-0433, approved through April 30, 2022.
Attachment B is labeled draft as it is under review by OMB.
INQUIRIES TO: Children’s Bureau Regional Program Managers

/s/
Jerry Milner
Acting Commissioner,
Administration on Children, Youth and
Families
Attachments:
A. Children’s Bureau Regional Program Managers
B. Checklist for Program or Service Designation for HHS Consideration

3

Attachment A – Regional Program Managers – Children’s Bureau

1

Region 1 - Boston
Bob Cavanaugh
[email protected]
JFK Federal Building, Rm.
2000 15 Sudbury Street
Boston, MA 02203
(617) 565-1020
States: Connecticut, Maine, Massachusetts, New
Hampshire, Rhode Island, Vermont

2

Region 2 - New York City
Alfonso Nicholas
[email protected]
26 Federal Plaza, Rm. 4114
New York, NY 10278
(212) 264-2890, x 145
States and Territories: New Jersey, New
York, Puerto Rico, Virgin Islands

3

Region 3 - Philadelphia
Lisa Pearson
[email protected]
The Strawbridge Building
801 Market Street
Philadelphia, PA 19107-3134
(215) 861-4030

6

Region 6 - Dallas
Janis Brown
[email protected]
1301 Young Street, Suite 945
Dallas, TX 75202-5433
(214) 767-8466
States: Arkansas, Louisiana, New
Mexico, Oklahoma, Texas

7

Region 7 - Kansas City
Deborah Smith
[email protected]
Federal Office Building, Rm.
349 601 E 12th Street
Kansas City, MO
64106 (816) 426-2262
States: Iowa, Kansas, Missouri, Nebraska

8

States: Delaware, District of Columbia,
Maryland, Pennsylvania, Virginia, West Virginia

4

5

Region 4 - Atlanta
Shalonda Cawthon
[email protected]
61 Forsyth Street SW, Ste. 4M60
Atlanta, GA 30303-8909
(404) 562-2242
States: Alabama, Mississippi, Florida, North
Carolina, Georgia, South Carolina, Kentucky,
Tennessee

Region 5 - Chicago
Kendall Darling
[email protected]
233 N. Michigan Avenue, Suite 400
Chicago, IL 60601
(312) 353-9672
States: Illinois, Indiana, Michigan,
Minnesota, Ohio, Wisconsin

9

10

Region 8 - Denver
Marilyn Kennerson
[email protected]
1961 Stout Street, 8th Floor
Byron Rogers Federal Building
Denver, CO 80294-3538
(303) 844-1163
States: Colorado, Montana, North Dakota, South
Dakota, Utah, Wyoming
Region 9 - San Francisco
Debra Samples
[email protected]
90 7th Street - Ste 9-300
San Francisco, CA 94103
(415) 437-8626
States and Territories: Arizona, California,
Hawaii, Nevada, Outer Pacific—American Samoa
Commonwealth of the Northern Marianas,
Federated States of Micronesia (Chuuk, Pohnpei,
Yap) Guam, Marshall Islands, Palau
Region 10 - Seattle
Paula Bentz
[email protected]
701 Fifth Avenue, Suite 1600, MS-73
Seattle, WA 98104
(206) 615-3662
States: Alaska, Idaho, Oregon, Washington

DRAFT

Attachment B: Checklist for Program or Service Designation for HHS Consideration

Instructions:
Section I: The state must complete Section I (Table 1) once to summarize all of the programs and services that the state
reviewed and submitted and the designations for HHS consideration.
Section II: The state must complete Section II (Tables 2 and 3) once to describe the independent systematic review
methodology used to determine a program or service (listed in Table 1) designation for HHS consideration. Section II
outlines the criteria for an independent systematic review. To demonstrate that the state conducted an independent
systematic review consistent with sections 471(e)(4)(C)(iii)(I), (iv)(I)(aa) and (v)(I)(aa) of the Act, the state must answer
each question in the affirmative. If the independent systematic review used the Prevention Services Clearinghouse
Handbook of Standards and Procedures, the relevant sections must be indicated in the “Handbook Section” column. If
other systematic standards and procedures were used, states must submit documentation of the standards and
procedures used to review programs and services. States should determine the standards and procedures to be used
prior to beginning the independent systematic review process. If the state cannot answer each question in Table 2 and
Table 3 in the affirmative, ACF will not make transition payments for the program or service reviewed by the state using
those standards and procedures.
Section III: The state must complete Section III (Tables 4 and 5) for each program or service listed in Table 1, and provide
all required documentation. Section III outlines the requirements for the review of the program or service. States
should complete Table 4 prior to conducting an independent systematic review to determine if a program or service is
eligible for review. For a program or service to be eligible for review, the answer to both questions in Table 4 must be
affirmative and the state must provide the required documentation. If a program or service is eligible for review, the
state must conduct the review and identify each study reviewed in Table 5, regardless of whether a study was
determined to be eligible to be included in the review.
Section IV: The state must complete Section IV (Tables 6-10) for each program or service (listed in Table 1) reviewed and
submitted and provide all required documentation. Section IV lists studies the state determined to be “well-designed”
and “well-executed” and outlines characteristics of those studies. Do not include eligible studies that were not
determined to be “well-designed” and “well-executed” in Tables 6 -10. States should complete Table 6 with a list of all
eligible studies determined to be “well-designed” and “well-executed.” States should complete Table 7 to describe the
design and execution of each eligible “well-designed” and “well-executed” study. States should complete Table 8 to
describe the practice setting and study sample. States must answer in the affirmative that the program or service
included in each study was not substantially modified or adapted from the version under review. States must detail
favorable effects on target outcomes present in eligible studies determined to be “well-designed” and “well-executed.”
States must detail unfavorable effects on target and non-target outcomes present in eligible studies determined to be
“well-designed” and “well-executed.”
Section V: The state must complete Section V (Table 11) for each program or service reviewed and submitted. Section V
lists the program or service designation for HHS consideration and verification questions relevant to that designation.
The state must answer the questions applicable to the relevant designation in the affirmative.

1

DRAFT

Section I: Summary of Programs
and Services Reviewed and their
Designations for HHS Consideration

2

DRAFT

Section I. Summary of Programs and Services Reviewed
Table 1. Summary of Programs and Services Reviewed
To be considered for transitional payments, list programs and services reviewed and provide designations for HHS
consideration.
Program or Service Name
(if there are multiple versions, specify the specific version
reviewed)

Proposed Designations for HHS consideration
(Promising, Supported, or Well-Supported)

3

DRAFT

Section II: Standards and
Procedures for an Independent
Systematic Review

4

DRAFT

Section II. Standards and Procedures for a Systematic Review

(Complete Table 2 and Table 3 to provide the requested information on the independent systematic review.
The same standards and procedures should be used to review all programs and services.)
Table 2. Systematic Review
Sections 471(e)(4)(C)(iii)(I), (iv)(I)(aa) and (v)(I)(aa) of the Act require that systematic standards and procedures must be
used for all phases of the review process. In the table below, verify that systematic (i.e., explicit and reproducible)
standards and procedures were used and submit documentation of reviewer qualifications. If the systematic review used
the Prevention Services Clearinghouse Handbook of Standards and Procedures, indicate the relevant sections in the
“Handbook Section” column. If other systematic standards and procedures were used, submit documentation of the
standards and procedures.
Table 2. Systematic Review

 to
Verify

Handbook
Section

Were the same systematic standards and procedures used to review all programs and services?
Were qualified reviewers trained on systematic standards and procedures used to review all
programs and services?
Were standards and procedures in accordance with section 471(e) of the Social Security Act?
Were standards and procedures in accordance with the Initial Practice Criteria published in
Attachment C of ACYF-CB-PI-18-09?
Program or Service Eligibility: Were systematic standards and procedures used to determine if
programs or services were eligible for review? At a minimum, this includes standards and
procedures to:
• Determine if a program or service is a mental health, substance abuse, in-home
parent-skill based, or kinship navigator program; and
• Determine if there was a book/manual or writing available that specifies the
components of the practice protocol and describes how to administer the practice.
Literature Review: Were systematic standards and procedures used to conduct a
comprehensive literature review for studies of programs and services under review? At a
minimum, this includes standards and procedures to:

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Search bibliographic databases; and Search other sources of publicly available

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

•

Studies (e.g., websites of federal, state, and local governments, foundations, or other
organizations).
Study Eligibility: Were systematic standards and procedures used to determine if studies found
through the comprehensive literature review were eligible for review? At a minimum, this
includes standards and procedures to:
• Determine if each study examined the program or service under review (as described
in the book/manual or writing) or if it examined an adaptation;
• Determine if each study was published or prepared in or after 1990;
•

•

Determine if each study was publicly available in English;

•

Determine if each study had an eligible design (i.e., randomized control trial or quasiexperimental design);
Determine if each study had an intervention and appropriate comparison condition;

•
•

Determine if each study examined impacts of program or service on at least one
‘target’ outcome that falls broadly under the domains of child safety, child
permanency, child well-being, or adult (parent or kin-caregiver) well-being. Target
outcomes for kinship navigator programs can instead or also include access to, referral
to, and satisfaction with services; and

Blank

--

Blank

--

Blank

--

Blank

--

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

5

DRAFT

 to
Verify
Blank

Handbook
Section
Blank

Blank

Blank

Blank

Blank

Assess the equivalence of intervention and comparison groups at baseline and
whether the study statistically controlled for baseline differences;
Assess whether the study has design confounds;

Blank

Blank
Blank

Assess, if applicable, whether the study accounted for clustering (e.g., assessed risk of
joiner bias 1);
Assess whether the study accounted for missing data; and

Blank

Table 2. Systematic Review
•

Identify studies that meet the above criteria and are eligible for review.

Study Design and Execution: Were systematic standards and procedures used to determine if
eligible studies were well-designed and well-executed? At a minimum, this includes standards
and procedures to:
• Assess overall and differential sample attrition;
•
•
•
•

Determine if studies meet the above criteria and can be designated as well-designed
and well-executed.
Defining Studies: Sometimes study results are reported in more than one document, or a single
document reports results from multiple studies. Were systematic standards and procedures
used to determine if eligible, well-designed and well-executed studies of a program and service
have non-overlapping samples?
Study Effects: Were systematic standards and procedures used to examine favorable and
unfavorable effects in eligible, well-designed and well-executed studies? At a minimum, this
includes standards and procedures to:
• Determine if eligible, well-designed and well-executed studies found a favorable effect
(using conventional standards of statistical significance) on each target outcome; and
• Determine if eligible, well-designed and well-executed studies found an unfavorable
effect (using conventional standards of statistical significance) on each target or nontarget outcome.
Beyond the End of Treatment: Were systematic standards and procedures used to determine
the length of sustained favorable effects beyond the end of treatment in eligible, well-defined
and well-executed studies? At a minimum, this includes standards and procedures to:
• Identify (and if needed, define) the end of treatment; and
•

•

Calculate the length of a favorable effect beyond the end of treatment.

Blank

Blank

Blank
Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank

Blank
Blank

Usual Care or Practice Setting: Were systematic standards and procedures used to determine if
Blank
a study was conducted in a usual care or practice setting?
Risk of Harm: Were systematic standards and procedures used to determine if there is evidence
Blank
of risk of harm?
Designation: Were systematic standards and procedures used to designate programs and
services for HHS consideration (as promising, supported, well-supported, or does not currently Blank
meet the criteria)? At a minimum, this includes standards and procedures to:
• Determine if a program or service has one eligible, well-designed and well-executed
Blank
study that demonstrates a favorable effect on a target outcome and should be
considered for a designation of promising;
• Determine if a program or service has at least one eligible, well-designed and wellexecuted study carried out in a usual care or practice setting that demonstrates a
Blank
favorable effect on a target outcome at least 6 months beyond the end of treatment
and should be considered for a designation of supported; and
• Determine if a program or service has at least two eligible, well-designed and wellBlank
executed studies with non-overlapping samples carried out in usual care or practice

Blank
Blank
Blank
Blank
Blank
Blank

Blank
Blank

If a cluster randomized study permits individuals to join clusters after randomization, the estimate of the effect of the intervention
on individual outcomes may be biased if individuals who join the intervention clusters are systematically different from those who
join the comparison clusters.

1

6

DRAFT

Table 2. Systematic Review
settings that demonstrate favorable effects on a target outcome; at least one of the
studies must demonstrate a sustained favorable effect of at least 12 months beyond
the end of treatment on a target outcome; and should be considered for a designation
of well-supported.
Reconciliation of Discrepancies: Were systematic standards and procedures used to reconcile
discrepancies across reviewers? (applicable if more than one reviewer per study)
Author or Developer Queries: Were systematic standards and procedures used to query study
authors or program or service developers? (applicable if author or developer queries made)

 to
Verify

Handbook
Section

Blank

Blank

Blank

Blank

Table 3. Independent Review
The systematic review must be independent (i.e., objective and unbiased). In the table below, verify that an independent
review was conducted using systematic standards and procedures by providing the names of each state agency and
external partner that reviewed the program or service. States must answer all applicable questions in the affirmative.
Submit MOUs, Conflict of Interest Policies, and other relevant documentation.
List all state agencies and external partners that reviewed programs and services.

Table 3. Independent Review

 to Verify

Was the review independent (conducted by reviewers without conflicts of interest including those that
Blank
authored studies, evaluated, or developed the program or service under review)?
Was a Conflict of Interest Statement signed by reviewers attesting to their independence? If so, attach the
Blank
statement.
Was a Memorandum of Understanding (MOU) signed by external partners (if applicable)? If so, attach MOU(s). Blank

7

DRAFT

Sections III-V: Describe and
Document Findings from Each
Program and Service Reviewed and
Submitted

8

DRAFT

Section III. Review of Programs and Services
(Complete Tables 4-5 for each program or service reviewed.)
Table 4. Determination of Program or Service Eligibility
Fill in the table below for each program or service reviewed.
Table 4. Determination of Program or Service Eligibility:

 to Verify

Does the program or service have a book, manual, or other available documentation specifying the
Blank
components of the practice protocol and describing how to administer the practice?
Provide information about how the book/manual/other documentation can be accessed OR provide other information
supporting availability of book/manual/other documentation.
Is the program or service a mental health, substance abuse, in-home parent-skill based, or kinship
navigator program or service?
Identify the program or service area(s).

Blank

9

DRAFT

Table 5. Determination of Study Eligibility
Fill in the table below for each study of the program or service reviewed. Provide a response in every column; N/A or unknown are not acceptable responses. The
response in columns iii, v, vi, vii, and ix must be “yes” or “no.” The response in column ix is “yes” only when the responses in columns iii, v, vi, and vii are “yes.”
i. Study
Title/Authors

ii. Publicly Available
Location

iii. Is the
study in
English?
(Yes/No)

iv. Design
(RCT, QED, or
other). If
other, specify
design.

Example
Title. Smith,
A.B., Jones,
C.D., and
Doe, E.F.

https://www.acf.hhs
.gov/opre

Yes

RCT

v. Did the
intervention
condition receive
the program or
service under review
in accordance with
the
book/manual/docu
mentation? (Yes/No)
Yes

vi. Did the
comparison
condition receive
no or minimal
intervention or
treatment as
usual? (Yes/No)

vii. Did the
study examine
at least one
target
outcome?
(Yes/No)

viii. Year
Published

ix.
Eligible
for
Review?
(Yes/No
)

Yes

Yes

1997

Yes

10

DRAFT

Section IV. Review of “Well-designed” and “Well-executed” Studies
(Complete Tables 6-10 for each program or service reviewed.)
Table 6. Studies that are “Well-Designed” and “Well-Executed” 2
Provide an electronic copy of each of the studies determined to be eligible for review and determined to be “well-designed” and “well-executed.”
List all eligible studies that are “well-designed” and “well-executed’ (Study Title/Author)

2
For reference, the Prevention Services Clearinghouse Handbook Chapter 5 defines “well-designed” and “well-executed” studies as those that meet design and execution
standards for high or moderate support of causal evidence. Prevention Services Clearinghouse ratings apply to contrasts reported in a study. A single study may have multiple
design and execution ratings corresponding to each of its reported contrasts.

11

DRAFT

Table 7. Study Design and Execution
For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below. Provide a response in every column; N/A or
unknown are not acceptable responses for columns i, ii, iii, v, vi, and vii. The response in column ii must be “yes.”
i. Study
Title/Authors

ii. Verify the Absence of all
Confounds? (Yes/No)

iii. List Measures
that Achieved
Baseline
Equivalence

Example
Title. Smith,
A.B., Jones,
C.D., and
Doe, E.F.

Yes

-Center for
Epidemiologic
Studies
Depression Scale
(CES-D)
-Child Behavior
Checklist (CBCL)

iv. List Measures
that did NOT
Achieve Baseline
Equivalence but
were Statistically
Controlled for in
Analyses
-Income

v. Overall
Attrition 3
(for RCTs
only)

vi. Differential
Attrition 4 (for
RCTs only)

vii. Does
Study
Meet
Attrition
Standards?

viii. Notes,
as needed

2.0
percent

4.3 percentage
points

Yes

N/A

For reference, the Prevention Services Clearinghouse Handbook section 5.6 defines overall attrition as the number of individuals without post-test outcome data as a
percentage of the total number of members in the sample at the time that they learned the condition to which they were randomly assigned.
4
For reference, the Prevention Services Clearinghouse Handbook section 5.6 defines differential attrition as the absolute value of the percentage point difference between the
attrition rates for the intervention group and the comparison group.
3

12

DRAFT

Table 8. Study Description
For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below to describe the practice setting and study
sample as well as affirm that the program or service evaluated was not substantially modified or adapted from the version under review. Provide a response in
every column; N/A or unknown are not acceptable responses. The response in column v must be “yes.”
i. Study
Title/Autho
rs

ii. Was the study
conducted in a
usual care or
practice setting?
(Yes/No)

iii. What is
the study
sample
size?

iv. Describe the sample
demographics and
characteristics of the
intervention group

v. Describe the sample demographics and
characteristics of the comparison group

Example
Title.
Smith, A.B.,
Jones, C.D.,
and Doe,
E.F.

Yes

N=354

Caregivers – Average age = 37
years old (SD = 5 years); 95%
female; 35% Black or African
American, 25% White, 30%
Latino or Hispanic, and 10%
other; and 78% of households
living 200% below the federal
poverty level.

Caregivers – Average age = 35 years old (SD =
5 years); 93% female; 33% Black or African
American, 26% White, 31% Latino or Hispanic,
and 10% other; and 76% of households living
200% below the federal poverty level.

Caregivers,
N = 177
Child, N =
177

Children – Average age = 5 years
old (SD=1.3 years); 47% female;
37% Black or African American,
27% White, 32% Latino or
Hispanic, and 4% other.

vi. Verify that the program or
service evaluated in the study
was NOT substantially
modified or adapted from the
manual or version of the
program or service selected
for review (Yes/No)
Yes

Children – Average age = 5 years old (SD=1.4
years); 45% female; 34% Black or African
American, 28% White, 33% Latino or Hispanic,
and 4% other.

13

DRAFT

Table 9. Favorable Effects
For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below listing only target outcomes with favorable
effects. Provide a response in every column; N/A or unknown are not acceptable responses.
i. Study
Title/Authors

ii. List the Target
Outcome(s)

iii. List the
Outcome Measures

iv. List the
Reliability
Coefficients
for Each

Example Title.
Smith, A.B.,
Jones, C.D.,
and Doe, E.F.

Parent/Caregiver
Mental Health
(Depression)

CES-D

Child Behavioral
and Emotional
Functioning
(Externalizing
Behaviors)
Child Behavioral
and Emotional
Functioning
(Internalizing
Behaviors)

CBCL (Aggressive
Behavior Scale)

Cronbach’s
alpha
coefficient =
0.91
Cronbach’s
alpha
coefficient =
0.94

CBCL
(Anxious/Depressed
Scale)

Cronbach’s
alpha
coefficient =
0.84

v. Are
Each of
the
Outcome
Measures
Valid?
Yes

vi. Are Each of
the Outcome
Measures
Systematically
Administered?

viii. List the
Size of Effect
for Each of
the Outcome
Measures

ix. Indicate the
Length of Effect
Beyond the End
of Treatment
(in months)

Yes

vii. List the
P-Values
for Each of
the
Outcome
Measures
p = 0.04

d = 0.13

8 mos

Yes

Yes

p = 0.03

d = 0.24

0 mos

Yes

Yes

p = 0.23
(non-sig)

N/A

0 mos

14

DRAFT

Table 10. Unfavorable Effects
For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below listing only target outcomes with
unfavorable effects. Provide a response in every column; N/A or unknown are not acceptable responses.
i. Study
Title/Authors

ii. List the Target
or Non-Target
Outcome(s)

iii. List the Outcome
Measures

iv. List the
Reliability
Coefficients
for Each

v. Are Each
of the
Outcome
Measures
Valid?

vi. Are Each of
the Outcome
Measures
Systematically
Administered?

vii. List the
P-Values
for Each of
the
Outcome
Measures

viii. List the
Size of Effect
for Each of
the
Outcome
Measures

Example Title.
Smith, A.B.,
Jones, C.D.,
and Doe, E.F.

Adult Height

Inches

Cronbach’s
alpha
coefficient =
0.99

Yes

Yes

p = 0.047

d = -0.05

ix. Indicate
the Length of
Effect
Beyond the
End of
Treatment
(in months)
0 mos

15

DRAFT

Section V. Program or Service Designation for HHS Consideration
Table 11. Program or Service Designation for HHS Consideration
Fill out the table below for the program or service reviewed. Only select one designation. Answer questions relevant to the selected designation; relevant
questions must be answered in the affirmative.
Table 11. Program or Service Designation for HHS Consideration

There is NOT sufficient evidence of risk of harm such that the overall weight of evidence does not support the
benefits of the program or service.
Blank
Well-Supported

 to Verify
Blank
 the Designation and Provide a
Response to the Questions Relevant
to that Designation
Blank

•

Does the program or service have at least two eligible, well-designed and well-executed studies
with non-overlapping samples 5 that were carried out in a usual care or practice setting?

Blank

•

Does one of the studies demonstrate a sustained favorable effect of at least 12 months beyond
the end of treatment on at least one target outcome

Blank

Supported
•

Blank
Does the program or service have at least one eligible, well-designed and well-executed study
that was carried out in a usual care or practice setting and demonstrate a sustained favorable
effect of at least 6 months beyond the end of treatment on at least one target outcome?

Promising
•

Blank
Blank

Does the program or service have at least one eligible, well-designed and well-executed study
and demonstrate a favorable effect on at least one ‘target outcome’?

Blank

5

Samples across multiple sources of a study are considered overlapping if the samples are the same or have a large degree of overlap. Findings from an eligible study
determined to be “well-executed” and “well-designed” may be reported across multiple sources including peer-reviewed journal articles and publicly available government and
foundation reports. In such instances, the multiple sources would have overlapping samples. The findings across multiple sources with these overlapping samples should be
considered one study when designating a program or service as “well-supported,” “supported,” and “promising.”

16