2-14-08 memo

From: Kelly, Brendan (ACF) [mailto:[email protected]]
Sent: Thursday, February 14, 2008 3:43 PM
To: Matsuoka, Karen Y.
Cc: Wright, Girley A (ACF); Koerper, Karl (ACF); Goldstein, Naomi (ACF)
Subject: FW: philadelphia HtE ICR
Importance: High
Hi Karen,
Thanks again for your help last week. We’re off and running again in the Bay State.
Below you will find ACF’s responses to the Philadelphia HtE ICR. If these are satisfactory, I’m
happy to have the revised Supporting Statements and Scripts uploaded into ICRAS/ROCIS upon
your approval.
In addition, I wanted to inquire as to the status of our other HtE ICR pertaining to the Rhode
Island site as well as our clearance on the Supporting Healthy Marriage (SHM) evaluation.
Rhode Island was sent at approximately the same time as this Philadelphia ICR, while the 30 day
comment period for SHM expired a couple of weeks ago. In both cases we were hopeful to
receive comments from OMB soon.
Thanks for your attention. If you feel a conference call or further clarification of these answers is
needed, please do not hesitate to let me know.
Thanks,
Brendan

Brendan Conor Kelly
Social Science Research Analyst
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
370 L'Enfant Promenade SW
Washington, DC 20447
ph: 202-401-5695
fax: 202-205-3598
From: Wright, Girley A (ACF)
Sent: Thursday, February 14, 2008 3:29 PM
To: Kelly, Brendan (ACF)
Cc: 'Lauren Cates'; Koerper, Karl (ACF)
Subject: FW: philadelphia HtE ICR
Importance: High

From: Matsuoka, Karen Y.
Sent: Monday, January 28, 2008 6:46 PM
To: Kelly, Brendan (ACF)
Subject: Philadelphia HtE ICR

Brendan, just a few questions on this ICR. The consent forms do not appear to have been
submitted. Just for future reference, I’ll have to return submissions like this as incomplete in the
future. Thanks. – Karen

1. please elaborate more on the cost study. If ACF is conducting a cost study, why not also
a study of the benefits? It would seem that ACF will bias the results against the
interventions if an analysis of benefits does not accompany an analysis of costs.
The cost-only analysis is a very common and informative alternative to a full cost-benefit analysis.
The cost analysis in this study will present detailed information on the costs to operate the TWC
and STEP programs, as they were implemented, and will account for levels of participation in
program services. The cost analysis will be included in a report with early impact findings that will
show the effects of the programs on financial outcomes such as participants TANF receipt,
earnings, and income. Taken together, the cost and impact analysis will allow program
administrators to make informed determinations about whether it is financially feasible and
practical to operate similar programs in the future. The cost analysis will present direct
comparisons of costs for the TWC and STEP programs over and above the costs for services
provided to the control group in the community. Importantly, the cost-only analysis also allows for
comparisons with the costs of other interventions with similar goals.

2. Precisely what costs will be included and how will they be calculated?
The cost team has collected financial expenditure data and staff time and salary allocations from
TWC and STEP. Unit costs for each program component will be estimated using the TWC and
STEP data. Ultimately a cost per program group member will be estimated using a combination of
the unit costs estimates (from TWC and STEP financial data) and program participation data
(from MIS and client survey data). In addition to overall program costs and overhead costs (such
as rent and utilities), costs for each of the main program components will be estimated and shown
separately. Some of the program components we hope to cost out separately include
recruitment, assessment and orientation, professional development activities, case management
and job development, transitional worksites (for TWC), retention, and monitoring and program
oversight.

3. on page 3 under the description of TWC, it says that that the services offered to
participants in the HtE project are the same as those offered to TANF recipients at TWC
who are not in the study. How will the costs for this cost analysis be calculated?
Often financial data from organizations is not tracked separately by client type. Because TWC's
program does not differ for HtE study participants compared to other TWC client populations, we
can safely assume that unit costs do not differ for the study participants. Thus, we are able to
estimate unit costs per participant from TWC's overall financial and caseload data. In turn those
unit cost estimates can be applied to the HtE sample and combined with program participation
estimates from MIS data for the study sample. This will allow us to accurately estimate program
costs per participant and per program group member without making additional assumptions to
account for differences in TWC's treatment of the sample.

4. are there ways to ensure that the control group have not participated in any support
programs of their own, independent of the HtE project? If they have, will they still be a
real “control” group?
It would be neither possible nor desirable to exclude the control group from participating in
services on their own. Indeed, the study was designed to compare TWC and STEP to other
services individuals could obtain in the community – in other words, this study is comparing TWC
& STEP to “services as usual.” This is a policy-relevant comparison, and one that state policy-

makers identified as being of interest at the outset of the study. This design has been used in
many studies of welfare-to-work programs in the past. The client surveys will capture the
treatment difference between the research groups, and the study reports will document this
treatment difference.

5. are the respondents all adults? Or will some children be interviewed?
The respondents are all adults. Children will not be interviewed, but the parent will be asked a
series of questions about child outcomes for one focal child per family.

6. are there consent forms for this study? They do not seem to be included in the
submission.
MDRC’s IRB reviewed this study and determined that consent was not required, as per 45 CFR §
46.116, which reads in part: An IRB may approve a consent procedure which does not include,
or which alters, some or all of the elements of informed consent set forth above, or waive the
requirement to obtain informed consent provided the IRB finds and documents that: (1) The
research or demonstration project is to be conducted by or subject to the approval of state or
local government officials and is designed to study, evaluate, or otherwise examine: (i) public
benefit or service programs; (ii) procedures for obtaining benefits or services under those
programs; (iii) possible changes in or alternatives to those programs or procedures; or (iv)
possible changes in methods or levels of payment for benefits or services under those programs.

7. will the survey instruments be modified at all as a result of the pre-testing/pilot
testing? In what ways?
If any changes are needed, we will propose them to OMB in accordance with usual procedures.

8. A10 mentions an introductory script that will inform respondents when information
will be revealed to the appropriate authorities. OMB cannot find this script. Can you
point us to where it is?
The scripts are attached. We apologize for not including them in the original submission.

9. because ACF does not have statutory authority to provide assurances of
confidentiality, the word “confidential” should not appear anywhere on the
scripts. Nor should it appear in the supporting statement. Rather, ACF can provide
assurances to keep information “private to the extent permitted by law.”
The scripts do not use the word “confidential.” Our response to Section A10 of the supporting
statement made reference to confidentiality, but we have attached a revised version of this
document that replaces all mentions of confidentiality with assurances of privacy only.

10. the supporting statements says this study will cost the following amounts in the
following years: 2007 ($140,000), 2008 ($320,000), 2009 ($320,000).
a. Is the $140,000 cost for 2007 related to this collection or to previous
collections (e.g. the 15month study)?
This figure is for the 36-month collection only.

b. Why does ROCIS report a cost to the government of $2,007?
This appears to be a data entry error, wherein the year 2007 was mistakenly entered into a dollar
field.