HCV_Pretest_Draft_Part_B_1-17-12[1]

Part B. Statistical Methods

B1 Potential Respondent Universe

The respondent universe for this data collection effort is the HCV program staff at four PHAs that will be selected from the programs that were confirmed as operating high-performing HCV programs during the study’s reconnaissance site visits.

The goal of the pretest is to test the methodology that is expected to be used for the full study of administrative fees and costs in high-performing HCV programs. Thus, we will focus our selection on the group of agencies that are confirmed as operating high-performing HCV programs. Forty-seven of 59 PHAs were confirmed as high performers during the first (reconnaissance) phase of the study. The sample of four PHAs for the pretest will be selected from this group of agencies and will include large, medium, and small HCV programs.

The rationale for the sample size of four programs is to enable some variation in PHA size and characteristics, while at the same time completing the pretest within a short amount of time and within available resources. We expect one of the pretest PHAs to administer less than 250 vouchers, one to administer between 500 and 1,249 vouchers, one to administer between 1,250 and 5,000 vouchers, and one to administer more than 5,000 vouchers.

B2 Statistical Methods

B2.1 Sampling Plan

The HCV programs that will be included in the pretest will be selected purposively from among the confirmed high-performing programs. We will try to select very dissimilar agencies in order to test the data collection protocols in a variety of environments. PHAs will be selected to vary by size of HCV program, geography, and methodology for allocating overhead costs. We plan to pretest the data collection in one PHA with fewer than 250 vouchers, one with between 500 and 1,249 vouchers, one with between 1,250 and 5,000 vouchers, and one with more than 5,000 vouchers. Our goal for the pretest is to include one PHA in the west, one in the south, one in the northeast, and one in the mid-Atlantic or mid-west. There are four main methods PHAs use for allocating overhead programs, and each may have implications for collection of cost data. Thus, the pretest sites will include PHAs that use each of the four methods: one PHA that allocates overhead costs based on an approved Central Office Cost Center (COCC) plan; one PHA that is subject to asset management and but does not have a COCC and instead uses another, approved allocation method; one PHA that is not subject to asset management and uses an approved allocation method; and one voucher-only PHA that does not need to allocate overhead costs across programs since it only operates one program.

As was the case in the reconnaissance data collection, we expect that some PHAs selected for the pretest will not be able to participate (due to scheduling issues or refusals). Thus, we will select a primary sample of four PHAs along with two backups for each of the sites. The backups will be contacted if the primary site is not able to participate in the pretest.

Within each of the selected agencies, all PHA staff working on the HCV program will participate in the time measurement data collection using RMS, so there will be no sampling. For the purposes of calculating study burden, we have assumed that a maximum of 72 staff across the four sites will participate in RMS, ranging from 3 staff at the smallest site to (potentially) more than 30 at the largest. In addition, in each site we will work with the HCV program director and a finance person to identify the other program costs (fringe, overhead, and non-labor costs) that should be allocated to the HCV program. We will also ask the HCV director to provide data (and answer questions) related to product counts, that is, how many times a given HCV program activity takes place per HCV applicant or participant and per year.

B2.2 Estimation Procedures

The data collected at each pretest site will be used to estimate the overall cost of administering the HCV program at that site, and the costs associated with specific activities such as intake, recertification, and annual inspections and subtasks such as briefings, waitlist management, and leasing.

We will use the time per activity collected from the RMS, along with product counts, salary, benefits, overhead and direct cost data to estimate a total cost and a per-activity cost of administering the HCV program at each of the pretest sites.

B2.3 Justification of Level of Accuracy

This data collection effort is not intended to provide statistically reliable data for the HCV program. The data will be used to test the data collection protocols for the full study of HCV administrative fees.

The key issue in terms of accuracy is ensuring that through the RMS data collection we have enough data points to estimate the time spend on each activity accurately. The number of data points per activity will depend on the frequency of the random moments (i.e., how often staff are contacted to provide information), the number of times the activity is done over the data collection period, and the amount of time that staff spend on the activity. The estimates will be more precise as the contacts are more frequent (so we have more data points), and as the activities or sub-tasks are aggregated to larger task-groups (so that the percent of time spent on any recorded task is larger). However, grouping activities and sub-tasks together into large categories may not provide useful information on time and costs for important subtasks. By the same token, contacting people too often may cause staff to ignore the contacts or feel over-burdened by the data collection. A key goal for the pretests is to learn how what the appropriate balance is between these two dimensions in order to obtain the most detailed information possible on activities and sub-tasks without sacrificing accuracy or risking staff burnout.

B2.4 Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems associated with this sample.

B2.5 Any Use of Periodic (less frequent than annual) Data Collection Cycles to Reduce Burden.

Not applicable to this study as it requires data to be collected just once.

B3 Maximizing Response Rates

Our procedures for recruiting PHAs to participate in the pretest are designed to achieve maximum participation. HUD will send a letter to the Executive Director and HCV Program Director of each of the four sampled agencies to invite and encourage them to participate in the pretest. (A copy of the letter is provided in Appendix D.) The letter will provide an overview of the purpose of the study and highlight what will be required of participating PHAs. The letter will be followed by a telephone call from a senior member of the Abt Associates team, who will speak with the Executive Director and HCV Director to describe the study again and request the agency’s participation. (A copy of the telephone script is provided in Appendix E.)

For the PHAs that agree to participate, Abt will prepare a document describing the obligations of the PHA for the pretest. We will ask the PHA to review the document carefully and share it among key HCV program staff. The Abt contact for the agency will then hold a follow-up call with the agency to ensure that PHA staff members understand their responsibilities for the pretest. Once the follow-up call is complete, the research team will proceed with making arrangements for implementing the pretest.

As discussed in Section A9, we also propose to provide monetary compensation to PHAs that participate in the pretest, as will be done for PHAs that participate in the full study. We recognize that participating in the pretest places burdens on staff members who will be required to meet with the study team, receive training on the data collection methodology, and then either respond to random moment sampling via smart phone for two months. Participating in the pretest will also require staff to work with the study team to review overhead and non-labor costs.

Based on input from HCV program experts on the study team, we have determined that a flat fee of $2,800 plus an amount equal to $300 per staff participating in RMS would be appropriate compensation for agencies participating in the pretest (and in the full study). We do not think this amount is sufficiently large as to place undue pressure on agencies to participate in the study, as it would not fully cover the staff costs associated with study participation. The compensation would be provided as unrestricted funds in order to provide maximum flexibility to agencies.

B4 Tests of Procedures or Methods

Drafts of the data collection instruments have been reviewed by HUD personnel, members of the Abt Associates research team, and the Expert and Industry Technical Review Group created for the study to ensure that the instruments are clear, flow well, and are as concise as possible. The goal of this data collection effort is to test the procedures that will ultimately be used in the full national study.

B5 Statistical Consultation and Information Collection Agents

HUD’s Office of Policy Development and Research will work with the contractor, Abt Associates, to conduct the proposed data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Government Technical Representative (GTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star can be contacted at (202) 402-5705 and (202) 402-6139, respectively. The Abt Associates Principal Investigator is Dr. Meryl Finkel, who can be reached at (617) 349-2380.

Abt Associates Inc. Part B. Statistical Methods 4