SS-Part B_CTG.SDM_022812 doc (2)

Community Transformation Grants: Use of System Dynamic Modeling and Economic Analysis in Select Communities

Supporting Statement

Part B

Contact:

Robin Soler, PhD

Technical Monitor and Principal Investigator

Division of Community Health

Centers for Disease Control and Prevention

4770 Buford Hwy, NE MS K-45

Atlanta, GA 30341

Telephone: (770) 488-5103

FAX: (770) 488-6450

February 16, 2012

TABLE OF CONTENTS

Section Page

B. Collection of Information Employing Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for the Collection of Information 1

B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4

B.4 Test of Procedures or Methods to be Undertaken 4

B.5 Individuals Consulted on Statistical Aspects and/or Analyzing Data 6

LIST OF ATTACHMENTS

Attachment 1a. Authorizing Legislation: American Recovery and Reinvestment Act of 2009

Attachment 1b. Authorizing Legislation: Patient Protection and Affordable Care Act

Attachment 1c: Authorizing Legislation: Public Health Service Act

Attachment 2. 60-day Federal Register Notice

Attachment 3. CTG Strategic Directions and Examples of CDC-Recommended Evidence- and Practice-Based Strategies

Attachment 4. List of CTG Awardees

Attachment 5. Overall CTG Evaluation Questions

Attachment 6a. CTG Cost Study Instrument (CTG-CSI)

B. Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

The data collection process does not employ statistical methods. Respondents will be 30 Community Transformation Grant (CTG) implementation awardees receiving American Reinvestment and Recovery Act (ARRA) funds to participate in the Community Transformation Grants initiative. The 30 implementation awardees will be selected by CDC based on the list of priority interventions selected for the cost study (currently under development). Exact selection criteria will be established after awardees finalize their Community Implementation Plans (CTIPs). Choosing a smaller sample to survey will not allow for a comprehensive assessment of the intervention costs from all community programs (awardees). Variations across the awardees cannot be explained without detailed data from the 30 awardees. In addition, in order to guide individual program cost-effectiveness and cost and health benefits analyses, data are required on an individual program level.

B.2 Procedures for the Collection of Information

We are developing a Web-based CTG Cost Study Instrument (CTG-CSI) to collect information from the awardees (Attachment 6a). All 30 awardees participating in cost data collection will receive comprehensive training on using the Web-based CTG-CSI. A detailed User’s Manual will be provided with the cost instrument to assist the awardees in providing the requested data accurately (Attachment 6b). Training on the use of the CTG-CSI will be provided via a Webinar conducted by RTI and during a site visit that will be conducted by RTI staff to each awardees’ before the first data collection. Automated data checks will be incorporated in the tool, and this will allow the respondents to review and check data prior to transmission.

Once the data are received, they will be logged and archived. The cost data will then be reviewed for accuracy and completeness. Thorough data validation will be performed to assess the quality of the data available to perform the planned analysis. All data collected in the CTG-CSI will be assessed for missing information (percentage of fields with missing data) and incorrect data (percentage of data elements with formats that are not recognized; percentage with inappropriate range of values). Quarterly and at the end of the funding period, we will also review whether the subcategories sum up to the expected total costs. Discrepancies between the total amount of funds expended for all interventions and the total itemized costs will be identified and clarified with the awardees.

Based on each awardee’s submission, a report will be produced that contains counts and associated percentages for blank field errors, inter-field relationship errors, and inter-record relationship errors, in each data set. RTI will then have a conference call with program directors who have error reports and, if necessary, identify strategies to improve the integrity of the data. We will then create an aggregated analysis file for generating reports and publications.

A flowchart depicting the cost study instrument development and data collection process is presented in Exhibit B.2-1.

Exhibit B.2-1. Flowchart for Data Collection Process

Throughout the project period, data validation will be performed to assess the quality of the data available to perform the planned analysis. All data collected in the CTG-CSI (Attachment 6a) will be assessed for missing information (percentage of fields with missing data), and incorrect data (percentage of data elements with formats that are not recognized; percentage with inappropriate range of values). In-kind contributions will also be reviewed to ensure that only those contributions that represent true opportunity cost are included. Opportunity cost is defined as the “advantage forgone as the result of the acceptance of an alternative.” For example, a person who volunteers his or her time will not be able to devote the time spent on the CTG program to other activities. The time spent should therefore be valued at the market rate and included as a cost to the program. Quarterly and at the end of the funding period, we will also review whether the subcategories sum to at least the total spending during that period. Discrepancies between the total amount of funds expended and total itemized costs will be identified and clarified with the awardees through telephone calls and/or e-mail exchanges with the primary awardees’ respondent. The findings from data validation will be reviewed to determine whether any statistical or other corrections are required to generate accurate cost estimates.

Costs will be estimated for each of the 30 respondents each quarter. For these analyses, costs will be estimated for each respondent at multiple levels. For example, aggregate costs will be estimated, as well as costs for each resource category captured in the CTG-CSI (labor/personnel; partners; consultants; materials, travel, services; grant administration; and in-kind resources). Costs will also be estimated at the awardees’ level for each objective and intervention listed in the awardees’ CTG Implementation Plan (CTIP). Cumulative costs will also be assessed for each awardee at aggregate, resource, objective, and intervention levels.

In quarterly reporting, RTI will summarize the quarterly and cumulative costs across all awardees by providing the mean, median, minimum, and maximum cost values for all of the various cost outcomes. The cost outcomes that will be summarized in quarterly reports are, at a minimum, aggregate quarterly and cumulative spending, and quarterly and cumulative spending by intervention and resource category.

In addition to quarterly reports, final analyses will compare how total awardees’ costs differed, depending on the combination of objectives and interventions used. Final analysis will also identify a unit of analysis for each objective to allow for comparisons of unit costs across awardees. These analyses will provide a more meaningful comparison of costs across awardees of different sizes and different populations (e.g., will allow for comparisons of costs for small, tribal communities to costs for large, urban areas). For example, costs to implement smoke-free housing policies in public housing may be divided by the number of people using public housing in an area to enable comparisons across awardees of the policy’s cost per public housing resident. The final cost analyses will also address the extent to which efforts to target hard-to-reach populations affect costs. For these analyses, it is anticipated that intervention costs for awardees with similar intervention strategies, but different target populations (e.g., minority women versus the entire community), will be compared to estimate the excess costs associated with reaching hard-to-reach populations.

B.3 Methods to Maximize Response Rates and Deal with Non-response

The Centers for Disease Control and Prevention (CDC) expects that all 30 selected awardees will participate in this study. Therefore, there should be no non-response. We expect that selected CTG awardees will report data in a timely manner; however, awardees that have difficulty submitting data will be provided with technical assistance.

Training in the use of the data reporting system will be provided to program directors, or managers, and the support staff assisting with cost data collection and reporting. Awardees will also receive a User’s Manual that provides complete written instructions regarding the cost data submission requirements. This document will support consistent submissions across awardees.

B.4 Test of Procedures or Methods to be Undertaken

Components of a previously developed and tested instrument used for the Study of the Costs Associated with Community Activities under the CPPW Initiative were modified for development of the CTG-CSI. The CPPW CSI was approved by Office of Management and Budget (OMB) on January 21, 2011 (OMB control number 0990-0365, expiration date January 31, 2014).

In January-February 2012, RTI conducted a pilot test of the CTG-CSI with four respondents representing CTG implementation awardees. The pilot test assessed respondents’ ability to understand the data elements requested, identify the cost information required, complete the tool within the allocated timeframe, and finalize the time burden estimates.

The information learned from pretesting was used to finalize the CTG-CSI and User’s Manual. Feedback from pretesting was incorporated to create the final Web-based cost instrument that will serve as the data collection instrument for all of the CTG awardees.

The CTG-CSI requests expenditure details for the following categories:

1. Labor/Personnel Expenditures

1. Partner Expenditures

2. Consultant Expenditures

3. Costs Associated with Materials, Travel, Services

4. Other Administrative Costs (e.g., telephone, rent)

5. Labor and Non-Labor In-Kind Resources

For each category except in-kind resources, the CTG-CSI will request information about total spending during the quarter and percentage allocations of total expenditures or time across the awardee’s objectives and interventions. The objectives and interventions will be specified in the CTIP developed by each awardee. Allocations of spending across objectives and interventions will not be requested for administrative costs, which will be allocated using the average proportionate allocation for all their costs. For in-kind resources, the CSI will request hours of donated time for volunteers and the estimated value of donated resources for non-labor donations. To ensure that accurate objective-/intervention-level costs can be estimated for each awardee, the CSI will also request percentage allocations of in-kind resources across the awardee’s objectives/interventions (e.g., 25% time or cost for Objective 1 and 75% for Objective 4).

Using the information collected through the CTG-CSI, intervention cost estimates will be generated quarterly and cumulatively, resulting in cost-effectiveness and health and cost benefit analyses that will then be integrated into the system dynamics model. Each quarter, the following costs will be estimated for each respondent (30 respondents in total):

• total spending in quarter,

• cumulative spending to date,

• quarterly and cumulative spending by category (e.g., labor, in-kind), and

• quarterly and cumulative spending by objective and by intervention.

For the final analyses, cost-effectiveness and health and cost benefit comparisons of interventions will be made across awardees. To enable comparisons, unit costs will be estimated, where the unit of analysis may differ across objectives. For example, the unit of analysis of a media campaign to increase physical activity in the county may be county population, whereas the unit of analysis for initiatives focused on specific public housing complexes may be the number of residents in those complexes.

B.5 Individuals Consulted on Statistical Aspects and/or Analyzing Data

Robin Soler, PhD (770-488-5103), of CDC is the Principal Investigator and Technical Monitor for the study. She has overall responsibility for overseeing the design and administration of the survey, and she will be responsible for analyzing the survey data.

RTI International is the project contractor responsible for developing the online version of the CTG-CSI; providing training, guidance, and technical assistance to the CTG awardees; and collecting and analyzing cost data of the CTG Cost and Modeling Initiative. Justin Trogdon, PhD (919-541-6893), serves as RTI’s Project Director. In this role, he is the primary contact with the Technical Monitor and oversees work on all project tasks.

The survey instrument, sampling and data collection procedures, and analysis plan were designed in collaboration with researchers at HHS, CDC, and RTI. The following personnel are involved in the design of the protocol and data collection instrument:

Diane Orenstein, PhD

Research Psychologist

CDC

Specific contribution: Consultant on study and survey instrument design

(770) 488-8003

 

Kristine L. Day, MPH

Public Health Analyst

CDC

Specific contribution: Consultant on study and survey instrument design

(770) 488-5446

Julia Spencer, PhD, MSPH

Director, Division of Science Policy

ASPE

Specific contribution: Development of the CPPW cost study instrument

(202-690-7287)

Donatus U. Ekwueme, PhD
Senior Health Economist
CDC
Specific contribution: Consultant on study and survey instrument design
(770) 488-3182

Olga Khavjou, MA
Research Economist
RTI International
Specific contribution: Cost Instrument Development Task Leader
(919) 541-6689

Amanda Honeycutt, PhD
Senior Economist
RTI International
Specific contribution: System Dynamics Model and Cost Data Integration Task Leader
(919) 597-5129