Supporting Statement Part B

Implementation, Systems and Outcome Evaluation of the Tribal and Low-Income Health Profession Opportunity Grants (HPOG)

OMB Clearance Request

Supporting Statement Part B

June 8, 2011

Part B: Statistical Methods (Used for Collection of Information Employing Statistical Methods)

This section provides supporting statements for each of the five points outlined in Part B of the OMB guidelines, in order to collect information through establishing a Performance Reporting System for the Health Profession Opportunity Grant (HPOG) Implementation, Systems, and Outcome Evaluation. As noted in Part A, this submission seeks clearance for the Performance Reporting System, but the data collected in the system will also be used in other ongoing and future evaluation activities, including the National Evaluation currently being designed, the HPOG Tribal Evaluation, university-based research activities, and additional impact studies.

B1. Respondent Universe and Sampling Methods

The HPOG Performance Reporting System will include the participant population of the 32 organizations receiving Health Profession Opportunity Grants. As discussed, the system will provide data at the grantee, program and individual levels. Thus, data will be collected from the 32 grantees on their program design and implementation and on the estimated 30,000 participants they will serve. No sampling techniques will be employed for the HPOG Performance Reporting System.

B2. Procedures for Collection of Information

Respondents (grantees) must update data at least semi-annually, but may enter data at any time either manually or by uploading data from existing data systems into an Internet-based management information system constructed for the purpose of collecting data for the HPOG program. The Internet-based management information system will be housed at and managed by The Urban Institute.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

We expect to obtain a 100 percent response rate from the HPOG grantees, all of which have committed to provide the required performance data and reports to ACF as part of the cooperative agreements made in accepting their grants. However, it is expected that some grantees may have more technical challenges than others in using the HPOG Performance Reporting System, depending on the staff that will be entering data or the programmers developing the streaming feeds into the HPOG Performance Reporting System. Thus, training and technical assistance on the HPOG Performance Management System will be provided to grantees by Abt Associates and the Urban Institute to assist grantees in complying with the data requirements.

We anticipate that offering training and ongoing technical assistance on the HPOG Performance Reporting System to the grantees will also improve the reliability of the data. In addition to clearly labeling the data fields and providing definitions and instructions within the system, a separate HPOG Performance Reporting System Guidance Manual for grantees will be developed to provide a detailed explanation of each data element, definitions, and how to enter the data. This document will be available to grantees electronically so users can readily access it whenever needed. Several webinars on the HPOG system will also be conducted. One webinar will be scheduled to demonstrate the system. A second webinar will provide training on the system to staff users. Periodic (e.g., annual) “refresher” webinars will also be offered to system users (e.g., to train new staff).

Additionally, contractor staff will be available to provide technical assistance to grantees via a free telephone number. These staff will be able to answer data collection and data entry related questions, and can refer the caller to other staff as needed for assistance of a more technical nature, such as data entry issues. Further, a dedicated email address will be provided for grantee to use to send questions that do not require telephone interaction. HPOG contract research project staff will respond to questions related to data collection. IT staff will respond to questions of a technical nature; however, it is the responsibility of the grantee to do any programming needed to link its data systems to the HPOG Performance Reporting System based on guidelines provided by the HPOG project IT team.

B4. Tests of Procedures or Methods to Be Undertaken

The HPOG Performance Reporting System is being developed by the contractors for the Implementation, Systems and Outcome Evaluation of the Health Professions Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals (Abt Associates and The Urban Institute) in consultation with ACF and the HPOG Tribal evaluation contractor (NORC at the University of Chicago). Federal staff and evaluation team members had informal discussions with grantees at ACF’s initial grantee orientation conference on possible data system designs and data elements. A test website is being established to pre-test and pilot the HPOG Performance Reporting System. The first stage of testing will be internal, done at the contractor’s secure facility, entering fictional data. In the second stage testing, the test version online will be used to train grantee staff on data entry procedures, allowing the contractor to identify any outstanding aspects that must be addressed before the system is fully operational. The test version will remain “live” for a minimum of six months after the system begins full operation (September 30, 2011) to allow continuous testing and necessary modifications.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for this study is being collected by Abt Associates, a research consulting firm, and its subcontractor The Urban Institute, on behalf of ACF. With ACF oversight, Abt Associates and The Urban Institute are responsible for developing the HPOG Performance Reporting System and providing the TANF and low-income grantees with support for using the system to produce the required semi-annual reports. NORC will use data from the system in its descriptive analysis of the Tribal grants and their accomplishments, and will provide the Tribal grantees with support for using the system to produce the required semi-annual reports.

Statistical analyses of the data for annual program performance reports will be limited to descriptive tabulations included in the contractor’s annual reports to ACF (see A.16). Other as yet unspecified statistical analyses may be planned for the National Evaluation currently being designed and other future research efforts.

The contractor for the Performance Reporting System will establish and report performance measures based on the data in the system. Key input to the statistical aspects of the design of the data system was received from the following individuals:

Abt Associates Team

Alan Werner, Project Director, Abt Associates Inc.

Jacob Klerman, Co-Principal Investigator, Abt Associates Inc.

Demetra Nightingale, Co-Principal Investigator, The Urban Institute

Harry Hatry, Performance Management Team Leader, The Urban Institute

Department of Health and Human Services, Administration for Children and Families

Molly Irwin, Project Officer, Office of Planning, Research, and Evaluation

Naomi Goldstein, Director, Office of Planning, Research, and Evaluation

Mark Fucello, Director, Division of Economic Independence, Office of Planning, Research, and Evaluation

Stan Koutstaal, HPOG Program Manager, Office of Family Assistance

Hilary Forster, Project Officer, Office of Planning, Research, and Evaluation

Michael Dubinsky, HPOG Tribal Evaluation Project Officer, Office of Planning, Research, and Evaluation

Brendan Kelly, Innovative Strategies for Improving Self-Sufficiency Project Officer, Office of Planning, Research, and Evaluation

Other HPOG Coordination Partners

Michael Meit, Project Director, HPOG Tribal Evaluation, NORC at the University of Chicago

David Fein, Project Director, Innovative Strategies for Improving Self-Sufficiency Project, Abt Associates Inc.

Technical Working Group – Experts

Joshua Wiener, Ph.D.

Distinguished Fellow and Program Director, Aging, Disability, and Long-Term Care Program

RTI International

Robert I. Lerman, Ph.D.

Institute Fellow and Professor of Economics

The Urban Institute and American University

James A. Riccio, Ph.D.

Director, Low-Wage Workers and Communities Policy Area

MDRC

Kevin Hollenbeck, Ph.D.

Vice President and Senior Economist

Upjohn Institute

Chris Hulleman, Ph.D.

Assistant Professor of Psychology

James Madison University

Maureen Conway

Executive Director

Economic Opportunities Program

The Aspen Institute

Maria P. Aristigueta, Ph.D.

Professor and Director, School of Public Policy and Administration

University of Delaware

Christine Kovner, Ph.D.

Professor, College of Nursing

New York University

Olivia Golden, Ph.D.

Institute Fellow, Center on Labor, Human Services, and Population

The Urban Institute

John Holahan, Ph.D.

Center Director, Health Policy Center

The Urban Institute

Howard Rolston, Ph.D.

Principal Associate

Abt Associates, Inc.

Stephen Bell, Ph.D.

Principal Scientist

Abt Associates, Inc.

Inquiries regarding statistical aspects of the study design should be directed to:

Alan Werner

Abt Associates, Inc.

55 Wheeler St.

Cambridge, MA 02138

(617) 349-2832

The HHS project officer, Molly Irwin, Ph.D., has overseen the design process and can be contacted at:

Molly Irwin

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

370 L'Enfant Promenade, S.W.

Washington, DC 20477

(202) 401-5538

Part B: Collection of Information B-5