HCV Admin Fee_OMB Full Study_Part B_11-19-12

Part B. Statistical Methods

B1 Potential Respondent Universe

B1.1 Potential Respondent Universe for Time Measurement Study

The respondent universe for time measurement data collection effort is the HCV program staff at 50 to 60 PHAs that will be selected from the programs that were confirmed as operating high-performing HCV programs during the study’s first and second rounds of reconnaissance site visits.

At all but the largest PHAs in the sample, all staff who work on front-line functions of the HCV program, including supervisors, will be included in the time measurement data collection. If a PHA has more than 100 staff working on front-line HCV functions, a sample of 100 staff (covering all roles and functions) will be selected for time measurement. We expect this to be the case at 4 of the 60 PHAs.

Rationale for Limiting Sample to 50 to 60 PHAs

The final sample size for the time measurement study will depend on the availability of funds but will be between 50 and 60 PHAs. HUD wants as large a sample as possible for developing the new formula for allocating administrative fees within available resources.

Rationale for Limiting Sample to High Performers

The reason for limiting the study sample to PHAs with high-performing HCV programs is that HUD wants to model administrative costs for all agencies based on what it should cost at a high-performing program. The study is not intended to provide guidance on how standard and low performing agencies could improve performance or efficiency.

B1.2 Potential Respondent Universe for Telephone Interviews with Small Programs

The respondent universe for the telephone interviews with small programs is the 502 PHAs that operate HCV programs with less than 250 vouchers and that received a High Performer rating on SEMAP in three of the past four years (FY08-FY11) or for the years in which the HCV program was scored (if not scored in all four years). At each PHA, the typical interview respondent will be the HCV Program Director and/or one of the PHA’s financial staff.

B2 Statistical Methods

B2.1 Sampling Plan

B2.1.1 Sampling PHAs for the Time Measurement Study

The national study of administrative fees in the HCV program will be based on time measurement and cost data collection at 50 to 60 verified high-performing and efficient HCV programs.

The sample of 50 to 60 PHAs is being selected in two groups. The first group of approximately 35 PHAs was selected following the first round of reconnaissance site visits conducted between May and September 2011. The remainder of the sample will be selected following a second round of reconnaissance visits that will take place between May and September 2012. OMB approval for the second round of reconnaissance visits was received in March 2012.

In brief, the protocol for sample selection in each round was first to define the universe of potential high-performing HCV and efficient programs using HUD administrative data and recommendations from HUD Field Office staff. The main source of administrative data was HUD’s SEMAP system, which provides an overall measure of HCV program performance. Each year, or every other year for smaller PHAs, HCV programs are scored on a combination of administrative data measures reported to HUD through the PIC system and self-reported measures. Each program receives a rating of “High Performer,” “Standard Performer,” or “Troubled” based on its SEMAP score.

The initial group of potential HCV programs was selected based on receiving three years of “High Performer” ratings on SEMAP for fiscal years 2007, 2008, and 2009. The supplementary sample was selected based on receiving three years of “High Performer” ratings on SEMAP from among the most recent four years available (fiscal years 2008, 2009, 2010, and 2011). In the case of small programs that are not scored every year, the screen for inclusion in the universe is being scored on SEMAP for at least two of the past four years and receiving “High Performer” ratings for all years scored. However, HUD also understands that some PHAs might not have consistently achieved high performer status in SEMAP but nevertheless operate highly effective HCV programs. As a result, for both rounds of reconnaissance visits, HUD solicited nominations from its Field Offices for additional high-performing programs and these were included in the sampling universe.

HUD also determined that some HCV programs should be excluded from the sampling universe. These include past and current participants in the Moving to Work (MTW) demonstration, PHAs with major unresolved Independent Public Audit or Fair Housing findings in the past three years, and PHAs with substantial budget shortfalls in their HCV programs. We were able to use administrative data to exclude the MTW agencies prior to selecting the sample, but there was no automated way to identify PHAs with unresolved audit findings or budget shortfalls so we first selected a sample of PHAs and then sent the list to HUD Field Office and headquarters staff for individual review.

We then selected samples for the first and second round of reconnaissance visits randomly by HCV program size. We selected 60 PHAs for the first round and 35 for the second round.

In the first round, we completed site visits to 59 of the 60 sampled PHAs.¹ Of these 59 PHAs, 45 met the study criteria and were verified to have high-performing and efficient HCV programs. Seven of these agencies had fewer than 100 vouchers, our threshold for inclusion in the time measurement portion of the study. Of the remaining 38 PHAs, one refused to participate when contacted for the pre-test and two were eliminated by HUD due to recent performance issues at the agency. Thus, 35 of the first round programs can be included in the time measurement study.

We anticipate needing to add another 20 to 25 high performing and efficient HCV programs to the study, depending on the final size of the time measurement sample, which will likely be 55 to 60 PHAs. The actual number of additional sites needed might be larger, because additional sites from the first round may refuse or otherwise become ineligible when we approach them for the full study. We are conducting site visits to 35 more PHAs in order to identify 20 to 25 PHAs that meet the study criteria and are verified to have high-performing and efficient HCV programs.²

Although the reconnaissance samples were selected randomly within program size categories, we expect the final sample for the study to show variation in geographic location, PHA type, and a variety of other characteristics that could potentially affect costs, such as:

• Number of HUD and other programs administered;

• Size of service area;

• Local wage rates and presence of unions;

• Use of project-basing;

• Special voucher programs administered;

• Program goals (for example, deconcentration or mobility goals, goals related to serving hard to house populations);

• Characteristics of client population (age, disability, family size, non-English speakers);

• Characteristics of housing stock (age, condition, type, vacancy rates);

• Characteristics of landlord population (for example, whether the rental stock is owned by a few large landlords or many small ones); and

• Local Legal Aid environment.

When the second round of reconnaissance visits is complete (expected December 2012) and the final sample of 50 to 60 PHAs has been identified, we will analyze the sample along many of the above dimensions using HUD administrative data. If we find that the sample over- or under- represents a particular PHA characteristic (relative to the universe of PHAs) that is believed to affect HCV program costs, we will use weighting in the final analysis to control for those differences.

Exhibit B-1 shows the expected characteristics of the final study sample of 60 PHAs based on the characteristics of the 50 PHAs approved for the study thus far. The characteristics of the final sample will not be exactly as shown in Exhibit B-1, because we have not yet confirmed 10 of the 60 study sites. In addition, some of the PHAs confirmed as high performing in the first round of reconnaissance (and included in the sample in Exhibit B-1) might decline to participate in the study and need to be replaced.

Exhibit B-1. Expected Characteristics of Time Measurement Sample, Compared to SEMAP High Performers and All HCV Programs with More than 100 Vouchers

	Expected Sample		SEMAP High Performers		All HCV Programs
	Number	Percent	Number	Percent	Number	Percent
Program Size (vouchers)1
10,000+	4	7%	7	1%	32	1%
5,250-9,999	7	12%	25	2%	44	2%
1,250-5,249	10	17%	206	15%	311	13%
500-1,249	16	27%	350	25%	445	19%
250-499	18	30%	301	22%	438	19%
101-249	5	8%	292	21%	549	23%
1-100	0	0%	210	15%	532	23%
Total	60	100%	1,391	100%	2,351	100%
Region2
Midwest	29	48%	370	27%	609	26%
Northeast	8	13%	394	28%	647	28%
South	15	25%	458	33%	817	35%
West	8	13%	169	12%	278	12%
Total	60	100%	1,391	100%	2,351	100%
Agency Type2
Public Housing and HCV	40	67%	918	66%	1,457	62%
HCV only	20	33%	473	34%	875	37%
Type missing	0	0%	0	0%	19	1%
Total	60	100%	1,391	100%	2,351	100%

¹Based on the desired distribution of the sample by size category.

²Based on the characteristics of the 50 PHAs identified for the study thus far.

B2.1.2 Sampling HCV Staff for the Time Measurement Study

For most of the agencies in the study, all staff working on front-line HCV program activities will participate in the time measurement data collection using RMS, so there will be no sampling. Front-line HCV activities means work related to: intake and eligibility; lease up; ongoing occupancy; inspections; monitoring, supervisory, and accounting activities directly related to the HCV program; supportive services for HCV participants; special voucher programs; HCV Family Self Sufficiency Program (FSS); general customer service; community and owner relations; and customer service related to the HCV program. PHA staff that support the HCV program indirectly, serving other programs as well with shared overhead functions like payroll support, will not be included in the time measurement data collection. We will work with each PHA prior to the start of the time measurement data collection to identify which PHA staff do HCV front-line work as defined above.

For the purposes of calculating study burden, we have assumed an average of 20 staff per PHA will participate in RMS, ranging from 3 staff at the smallest site to up to 100 at the largest. At the largest PHAs in the study, where more than 100 staff work on HCV front-line activities, we will sample among those staff. It would not be efficient or necessary from an analytic point of view to collect time measurement data from more than 100 staff at any given PHA. In the event that a PHA selected for the study has more than 100 staff working on the HCV program, we will sample 100 of these staff randomly within job category to achieve a representative sample of all the program functions. For example, we will work with the PHA staff to identify all staff that serve unique functions in the HCV programs, as well as those staff that do substantially the same work. We will include all the staff that serve unique functions in our sample, and only sample from among those groups of staff who serve the same function. For example, a PHA might have 225 HCV program staff overall, 25 of which serve unique functions that are not done by any other staff at the agency. We would include all 25 of these staff in RMS. We would then allocate the 75 remaining RMS “slots” across the 200 staff that are in positions where at least one other staff person is doing the same work, based on the proportion of staff in each job category. For example, if the PHA has 20 inspectors that all perform the same work, we may determine that we need to include 4 of these inspectors in RMS. We will request a list of these 20 inspectors from the PHA and randomly select 4 for participation. In all cases where we sample staff for RMS, we will assign the sampled staff a sampling weight so that the time measurement data can be scaled up to account for the staff not included in the data collection. For example, if there are 20 inspectors and we sample 4, the sampling weight is 5 for each measured inspector.

For PHAs that administer the HCV program statewide or over a large geographic area through branch offices, we will select, in consultation with the PHA, 2 to 3 branch offices at which to do RMS data collection. These branch offices should be representative of the other branch offices in terms of the work performed and the staff included in RMS should cover the full range of activities required for HCV administration. In some cases, this may mean collecting time measurement data from headquarters as well as branch office staff. Observations from the sampled offices will be weighted to reflect the agency totals.

B2.1.3 Sampling PHAs for the Telephone Interviews with Small HCV Programs

The main goal of this study is to measure how much it costs to administer a high-performing and efficient HCV program, to identify the factors that affect administrative costs, and to develop a new fee formula based on the cost data collected. However, a secondary research question for the project is whether there is a minimum number of vouchers that a PHA needs in order to operate a financially feasible HCV program on HCV administrative fees alone. We plan to use the telephone interviews with small programs to quantify the costs of small programs to help answer this question and to compare to the costs of the larger programs included in the time measurement study.

As with the time measurement study, the telephone interviews with small programs will be limited to PHAs with high-performing HCV programs. The sampling universe for the telephone interviews with small programs is PHAs with less than 250 vouchers that received a “high-performer” rating on SEMAP in three of the past four years, or in any year in which they were rated between FY2008 and FY2011, or were recommended by the HUD Field Office or determined to be high-performing in the first round of the study.

We propose to select a sample of 180 PHAs with less than 250 vouchers. We will select a random sample of 90 PHAs in two size categories—1 to 100 vouchers and 101 to 249 vouchers—as shown in Exhibit B-2.

We expect a 70 to 75 percent response rate to the interviews. This estimated response rate reflects our experience in recruiting for the study that a substantial share of PHAs may decline to participate because they already feel overburdened by program requirements. With a response rate of 70 to 75 percent, we expect to be able to complete up to 130 interviews from the sample of 180. We will analyze the available administrative data on PHA size, agency type, and cost per voucher (as reported in HUD’s administrative data) to evaluate any possible non-response bias.

Exhibit B-2. Sampling for Small Program Interviews

	Sampling Universe		Interview Sample		Expected Completed Interviews
	Number	Percent	Number	Percent	Number	Percent
101-249 vouchers	292	58%	90	50%	65	50%
1-100 vouchers	210	42%	90	50%	65	50%
Total	502	100%	180	100%	130	100%

B2.2 Estimation Procedures

B2.2.1 Estimation Procedures for the Time Measurement Study

The data collected at each site will be used to estimate the overall cost of administering the HCV program at that site, the costs associated with specific activities such as intake, recertification, and annual inspections and, to the extent possible, subtasks such as income determinations within recertification or drive time for inspections.

Based on staff responses to the RMS we will calculate the time spent on each activity by multiplying the percent of time spent on each activity by the total number of work hours over the 40-day time measurement period.

Using the transaction counts and total time spent on each activity, we will be able to estimate the staff time per activity. That information, along with the salary and benefits for each staff member, the share of overhead costs that apply to the HCV program, and the direct costs of the HCV program, will be used to estimate a total cost and a per-activity cost of administering the HCV program at each of the study sites.

We will generate confidence intervals for each estimate so that hypotheses can be tested comparing times and costs across agencies with different characteristics. The comparisons will focus on the 60 PHAs that participate in the time measurement study. Based on analysis of FASS data we assume a standard deviation on cost per UML (unit months leased) of $27.80. Assuming an event split between two subgroups of interest (so each with a responding sample size of 30), our projection of the minimum detectable true difference in cost per UML is about $20.  That is what can be detected with 80 percent power given a two-sided test with alpha =0.05.  This number was derived as

We will select groups for comparisons carefully to ensure that any comparisons provide meaningful information, and are not based on samples that are too small to provide useful information.

We will develop a regression model that estimates total administrative costs per voucher as a function of program, client, and location characteristics across the 50 to 60 study sites. The model will be used to predict costs in the remaining HCV programs across the country.

Exhibit A-6 in Part A presents a preliminary list of variables that might be included in the regression analysis of total costs, which will serve as a basis for a HCV administrative fee formula. Inclusion in the formula will be determined based on which of these variables have the largest impacts on program costs. Variables believed to have a substantial impact on program costs and that can be collected across all HCV programs will be included in the regression analysis. In considering which variable to include in the regression analysis, we will need to be mindful of the relationship between the number of variables in the regression and the sample size needed to be able to detect effects. Also, the regression analysis can only include variables that are available for all programs, so that costs can be estimated for the sites not in the study.

B2.2.2 Estimation Procedures for the Telephone Interviews with Small Programs

For each PHA participating in the small program interviews, we will estimate a total administrative cost per voucher based on what we learn about how the PHA allocates staff time, direct costs, and overhead costs across different programs. All PHAs track and report to HUD their HCV program costs, so we will use their reported costs from the most recently completed fiscal year as a starting point. Through the data collection, we will seek to identify costs that are attributable to the HCV program but that are paid for by other programs or revenue sources and costs that are paid for by the HCV program that are attributable to other programs. We will also collect detailed information on the PHA’s use of contracted labor and materials for the HCV program and costs, such as office space, that might be provided by another entity (such as the local government) and not charged to the HCV program.

At the end of the data collection, we will have an accurate estimate of the complete cost of operating the HCV program, including the dollar cost per voucher, the number of FTEs per voucher, and the dollar amount of goods and services provided to the HCV program free of charge. We will analyze these variables by the number of vouchers and other PHA characteristics, such as the type of PHA (standalone PHA, a unit of government, or nonprofit), whether the PHA is privately managed, the number of programs (other than HCV) that the PHA administers, and the percent of family versus elderly or disabled clients in the HCV program, to identify characteristics associated with higher and lower per unit costs. We might find that there is a close relationship between PHA size and costs and a clear point at which the per voucher costs jump up as the HCV program gets smaller. Alternatively, we could find that the smallest HCV programs maintain financial viability by cross-subsidies from other parts of the PHA or by sharing staff with other small programs. We might also find—in comparing the interview data to the cost data collected through the time measurement study—that small HCV programs are no more or less likely than larger programs to rely on goods and services provided from another program within the PHA or a partner entity in order to be able to operate their programs effectively under the current fee structure.

B2.3 Justification of Level of Accuracy

B2.3.1 Level of Accuracy for Time Measurement Data Collection

The key issue in terms of accuracy is ensuring that through the RMS data collection we have enough data points to estimate the time spend on each activity accurately. The number of data points per activity will depend on the frequency of the random moments (i.e., how often staff are contacted to provide information), the number of times the activity is done over the data collection period, and the amount of time that staff spend on the activity. The estimates will be more precise as the contacts are more frequent (so we have more data points) and as the activities or sub-tasks are aggregated to larger task-groups (so that the percent of time spent on any recorded task is larger). However, grouping activities and sub-tasks together into large categories may not provide useful information on time and costs for important subtasks. By the same token, contacting people too often may cause staff to ignore the contacts or feel over-burdened by the data collection.

Based on the results from the pretests thus far, we have learned that PHA staff can tolerate two months of data collection at a rate of 12-15 notifications per day, but would find it difficult without additional incentives (financial or otherwise) either to extend the data collection period or to respond to more than two notifications per hour. Based on the data collected so far, we think that we will be able to develop robust estimates for 10-15 HCV program activities and subtasks for PHAs with more than 250 vouchers. For PHAs with less than 250 vouchers, we may not have sufficient data points to estimate a time per activity for the full range of activities. However, we will have sufficient data – even at the smallest PHAs in the time measurement study – to arrive at an estimate the time spent on the HCV program overall and to translate that time estimate into a cost per voucher which is what is needed in order to develop the regression model that will serve as the basis for a new formula.

Thus, we anticipate that we will have task-level time estimates for 50 to 60 PHAs in the study, and estimates of the overall cost per voucher (on which the fee formula will be based) for all PHAs in the study. The sample size of 50 to 60 PHAs limits the number of cost drivers that we can include in the regression analysis to approximately 10 to16, but 50 to 60 is the maximum number of PHAs given the available resources. The actual number of cost drivers that can be included will be determined analytically, depending on how well variations in selected variables can explain the variation in costs per voucher.

B2.3.2 Level of Accuracy for Small Program Interviews

We propose to complete up to 130 small program interviews. The final sample size will depend on the response rate but will likely be 100 to 130 PHAs. Based on HUD’s VMS data, which provides a rough approximation of per unit costs, the cost per UML (unit months leased) for high-performing PHAs with between 1 and 249 vouchers ranges from a low of $21 to a high of $222. The mean cost per UML for these programs is $64.7 and the median is $55.

With a sample size of 130, we can be 95 percent confident that our estimate of the mean cost per UML (unit months leased) will be within $4.75 of the true mean cost per UML. With a sample size of 100, we can be 95 percent confident that our estimate of the mean cost per UML (unit months leased) will be within $5.50 of the true mean cost per UML.

B2.4 Unusual Problems Requiring Specialized Sampling Procedures

Going from Sampled Moments to Estimates of Time Breakdowns

Under the RMS method, all PHA staff who work directly on the HCV program carry a smartphone for a two-month period and  respond to 12 to 15 notifications per day to identify the task the staff was working on at a given point in time. The responses for each person are aggregated to create estimates of the total share of that staff person’s time on each activity. The share of time spent on each activity is assumed to be equal to the share of notifications assigned to that activity. To calculate the total time each person spends on each activity, we multiply the shares of notifications by activity by the total work time available during the observation period. The total work time available is based on the work hours reported during the period. To calculate the total time on a given activity in each site, we will sum across all staff completing random moment sampling in the agency. This approach provides reliable estimates of the time per activity over the 40 day date collection period.

B2.5 Any Use of Periodic (less frequent than annual) Data Collection Cycles to Reduce Burden.

Not applicable to this study as it requires data to be collected just once.

B3 Maximizing Response Rates

Our procedures for recruiting PHAs to participate in the study are designed to achieve maximum participation. HUD will send a letter to the Executive Director and HCV Program Director of each of the sampled agencies to invite and encourage them to participate in the study. Appendix F provides a copy of the letter that would be sent to the PHAs selected for the time measurement study and Appendix G provides a copy of the letter that will be send to the PHAs selected for the small program interviews. The letters describe the purpose of the study and highlight what will be required of participating PHAs. The letters will be followed by a telephone call from a senior member of the Abt Associates team, who will speak with the Executive Director and HCV Director to describe the study again and request the agency’s participation. (Sample scripts for these calls are provided at the end of Appendices F and G.)

For the PHAs that agree to participate in the time measurement study, Abt will prepare a Memorandum of Understanding (MOU) that describes the obligations of the PHA for the pretest. (A copy of the MOU is provided in Appendix E.) We will ask the PHA to review the document carefully and share it among key HCV program staff. The Abt contact for the agency will then hold a follow-up call with the agency to ensure that PHA staff members understand their responsibilities for the study. Once the follow-up call is complete, the research team will proceed with making arrangements for implementing the study.

As discussed in Section A9, we also propose to provide monetary compensation to PHAs that participate in the time measurement study, as was done for PHAs participating in the pretest. We recognize that participating in the study places burdens on staff members who will be required to meet with the study team, receive training on the data collection methodology, and then either respond to random moment sampling via smart phone for two months. Participating in the time measurement study will also require staff to work with the study team to review overhead and non-labor costs.

Based on input from HCV program experts on the study team and the experience of the pretest, we have determined that a flat fee of $2,800 plus an amount equal to $300 per staff participating in RMS would be appropriate compensation for agencies participating in the time measurement study. We do not think this amount is sufficiently large as to place undue pressure on agencies to participate in the study, as it would not fully cover the staff costs associated with study participation. The MOU between Abt Associates and the PHA will state that “the compensation is expected to be used to defray personnel costs due directly to this information collection.”

The 130 PHAs participating in the small program interviews will not be compensated because of the lower level of burden on PHA staff (approximately 8 hours per PHA).

B4 Tests of Procedures or Methods

Drafts of the data collection instruments have been reviewed by HUD personnel, members of the Abt Associates research team, and the Expert and Industry Technical Review Group created for the study to ensure that the instruments are clear, flow well, and are as concise as possible. In addition, we tested a draft version of the small program telephone interview with six PHAs in January and February 2012 and tested the data collection procedures for the time measurement study with four PHAs between March and May 2012. Based on the feedback from the pretest, we have made modest revisions to the data collection instruments, reflected in the appendices. Additional revisions may be warranted based on feedback provided by the Expert and Industry Technical Review Group in October 2012, but we expect these to be very minor.

B5 Statistical Consultation and Information Collection Agents

HUD’s Office of Policy Development and Research will work with the contractor, Abt Associates, to conduct the proposed data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Government Technical Representative (GTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star can be contacted at (202) 402-5705 and (202) 402-6139, respectively. The Abt Associates Principal Investigator is Dr. Meryl Finkel, who can be reached at (617) 349-2380.

1 One PHA dropped out too late in data collection to be substituted.

2 Depending on the final sample size for the study and the number of PHAs that meet the study criteria, we might need to conduct reconnaissance visits to more than 35 PHAs in this second round. We obtained OMB approval in March 2012 to visit up to 45 PHAs if needed.

Part B: Statistical Methods ▌pg. 10