HCV Admin Fee_OMB Revised Draft_Part B_3-4-11_rev2

Part B. Statistical Methods

B1 Potential Respondent Universe

The respondent universe for this study is a sample of 60 PHAs of different sizes and in different parts of the country that are believed to operate high-performing HCV programs.

Rationale for Visiting 60 PHAs

The rationale for the sample size of 60 is that HUD wanted as large as possible a sample during the reconnaissance phase, within available resources, to be able to study the many different factors, including local market characteristics, HCV program size, and PHA organizational structure, believed to affect program administrative costs. HUD also anticipates that many of the programs studied during the reconnaissance phase will be validated as high-performing and efficient programs and thus will be suitable for inclusion in the full national study. However, the full national study will likely include additional PHAs that were not studied in the reconnaissance phase, and one of the purposes of the site visits is to determine how to select those additional PHAs without conducting full site visits.

After the first 20 site visits are complete we will assess findings regarding the validation of high performance and the factors that affect program administrative costs and efficiency. At that point, we may decide that we need to alter the sample in order to achieve all of the purposes of the site visits. In addition, we may decide that we should conduct abbreviated site visits or remote data collection at a larger number of agencies, instead of conducting full site visits to 40 additional agencies.

Rationale for Focusing on High Performers

The reason for limiting the study sample to PHAs with high-performing HCV programs is that HUD intends the full national study of administrative fees to be based on high-performing and efficient programs. The full national study will provide cost information to inform the development of a new administrative fee formula for the HCV program, and HUD wants to model administrative costs at only those HCV programs that are high performers. This first phase of the study—the site visits to 60 PHAs—will provide the information on drivers of program costs needed to design the full administrative costs and will also provide a group of PHAs suitable to participate in the full study, although HUD also anticipates needing to recruit additional PHAs for the full study.

Selection of the Sample of 60 PHAs

The national study of administrative fees in the HCV program—to be conducted under a subsequent Task Order—will ultimately be based on costs observed at a group of HCV programs that appear to be high-performing and efficient. As a result, this first study of 60 PHAs will also focus, to the extent possible, on programs that are believed to be high-performing and efficient.

The sample of 60 PHAs is not intended to be representative of the universe of HCV programs. Nor will it include agencies that are not high-performers for the purpose of comparing cost drivers for high-performing and non-high-performing programs. HCV programs can be non-high-performing for a wide variety of reasons, and HUD does not want to spend the limited resources available for this project studying program administration at agencies known not to be high performers.

The first stage in selecting the sample of 60 PHAs is to define the universe of high-performing HCV and efficient programs using HUD administrative data and recommendations from HUD Field Office staff. The main source of administrative data is HUD’s SEMAP system, which provides an overall measure of HCV program performance. Each year, all HCV programs are scored on a combination of administrative data measures reported to HUD through the PIC system and self-reported measures. Each program receives a rating of “High Performer,” “Standard Performer,” or “Troubled” based on its SEMAP score.

HUD has identified three years of consecutive “High Performer” ratings on SEMAP (2007, 2008, and 2009) as an appropriate first screen for identifying the study universe. (In the case of small programs that are not scored every year, the screen for inclusion in the universe is being scored on SEMAP for at least two years between 2006 through 2009, and receiving “High Performer” ratings for all years scored.) However, HUD also understands that some PHAs might not have achieved high performer status in all three years but nevertheless operate highly effective HCV programs. As a result, HUD has solicited nominations from its Field Offices for high-performing programs. Field Offices nominated 64 PHAs for inclusion in the study, of which 22 were not SEMAP high performers.

HUD also determined that programs that were granted waivers to immediately reduce payment standards for calendar year 2010 should be excluded from the sampling frame. A total of 28 agencies received payment standard waivers in 2010. Three of these were SEMAP high-performers in 2007, 2008, and 2009 and were excluded from the sampling universe. HUD does not think that agencies that received waivers in 2009 should be excluded because of the circumstances under which PHAs were operating in that year.

HUD has also determined that PHAs in the Moving to Work (MTW) demonstration should not be included in the study. Although many MTW agencies are believed to be high-performing and efficient, MTW agencies have been given flexibilities that allow them to streamline their operations to increase efficiency. There are currently 32 PHAs in the MTW demonstration.

Finally, HUD determined that PHAs with major unresolved Independent Public Audit or Fair Housing findings in the past three years should be excluded from the sample. Because there was no automated way to identify these PHAs, we first selected a sample of 180 PHAs (as discussed below) and sent that list to HUD Field Offices and headquarters staff for individual review. The exclusions resulting from this review are discussed below.

In summary, the universe from which we selected the reconnaissance sample included 877 HCV programs:

• HCV programs scored as high performers on SEMAP in all three years 2007 through 2009 (635 PHAs).

• Small programs (less than 250 vouchers) that were not scored in all three years, but were scored as high performers in at least two years from 2006 through 2009, and were scored as high performers in all years for which they received a score across the four year period (220 PHAs).

• Programs that were recommended as high performers by their HUD Field Office even if they did not receive high performer scores across the three years (22 PHAs).

Exhibit B-1 shows the region and size characteristics of the 877 SEMAP high-performing programs relative to all HCV programs which represent the sampling universe.

Exhibit B-1. Description of Sampling Universe for HCV Program Administrative Fee Study

	SEMAP High Performers in 2007, 2008, and 2009		All HCV programs
	Number	Percent	Number	Percent
Region
Northeast	220	25%	639	27.4%
Midwest	277	32%	607	26.1%
South	244	28%	810	34.8%
West	136	16%	272	11.7%
Total	877	100%	2,328a	100%
Program Size (vouchers)
Extra Large (10,000+)	6	1%	20	1%
Large (1,250 - 9,999)	151	17%	327	14%
Medium (500 - 1,249)	225	26%	437	19%
Medium Low (250 - 499)	174	20%	446	19%
Small (50 - 249)	280	32%	916	39%
Extra small (1 - 49)	41	5%	185	8%
Total	877	100%	2,331	100%
Agency Type
Public Housing and HCV	587	67%	1,449	62%
HCV Only	290	33%	882	38%
Total	877	100%	2,331	100%

Note ^a Exhibit does not show three HCV programs in Guam, Mariana Islands, and Virgin Islands. These programs did not meet the study standards for high performance.

From the sampling universe of three-year SEMAP high performers plus Field Office nominations, we selected a stratified random sample of 180 agencies using HCV program size as the sampling strata. We selected the 180 agencies to obtain a sample of 60 with 2 backups for each agency sampled. We then sent the list of 180 agencies to the HUD Field Office and headquarters staff to be reviewed for audit findings. PHAs with major unresolved Independent Public Accountant (IPA) audit findings or fair housing findings from Fair Housing and Equal Opportunity (FHEO) reviews in the past three years related to the HCV program were excluded from the sample. The Field Office and headquarters review resulted in 17 agencies being excluded from the sample. We replaced the PHAs in the initial sample that were eliminated through Field Office and headquarters review with the designated backup agency. The final sample will depend on which PHAs agree to participate in the study. If a PHA in the initial sample refuses to participate, the agency will be replaced by the backup that is the closest match in terms of HCV program size, region, and PHA type.

Although we will not be sampling explicitly for these characteristics, we anticipate that the final sample of 60 PHAs will exhibit variety along the following dimensions believed to affect program costs and efficiency:

• Type of administration (local PHA, state/regional administration, non-profit administrator, independent PHA vs. PHA that is operated within a local government);

• Number of HUD and other programs administered;

• Physical size of service area;

• Local wage rates/unions;

• Special programs administered (VASH, FUP);

• Program goals (for example, deconcentration or mobility goals, goals related to serving hard to house populations);

• Use of project-basing;

• Characteristics of client population (age, disability, family size, non-English speakers);

• Characteristics of housing stock (age, condition, type, vacancy rates);

• Characteristics of landlord population (for example, whether the rental stock is owned by a few large landlords or many small ones); and

• Local Legal Aid environment.

The type of PHA administration and the number of programs administered may be particularly important factors affecting the ability of small programs (that is, programs with fewer than 250 vouchers) to administer the HCV program effectively. We expect to use data from HUD’s FASS system on the type of PHA administration and the number of programs to test that we have sufficient diversity in these areas among the small program sub-sample before finalizing the sample. If a random sample does not produce sufficient diversity, we may elect to sample purposively for the small program sub-sample only.

B2 Statistical Methods

B2.1 Sampling Plan

As noted, the study data collection for this project includes a sample of 60 HCV programs from which we will collect information on program performance, costs, and areas of efficiency, and identify potential study sites for the full national study of HCV administrative costs.

The sample for the full national study and the data collection approach for that study will be the subject of a separate request for OMB approval. Based on our current understanding of the factors that affect HCV program administrative costs and feedback from the Expert and Industry panels, we anticipate that the sample for the full national study will include at least 30 to 40 PHAs, and perhaps more PHAs if there is a large amount of variation in administrative costs. The exact sample size, characteristics of the sample, and procedures for identifying additional PHAs (beyond those reviewed in the first phase) will be determined after the site visits are complete.

B2.2 Justification of Level of Accuracy

Data from the 60 site visits will not be combined in any way to provide statistical or summary data for the HCV program as a whole. The data will be used to confirm (or not) the high-performing status of each PHA and to collect information on cost drivers and program efficiency that will inform the design of the full national study.

The full study will involve regression modeling to identify the factors that affect HCV program administrative costs and the relative weight that should be applied to each in an administrative fee formula. This analysis will be discussed in the OMB request developed for that part of the study.

B3 Maximizing Response Rates

Our procedures for recruiting PHAs to participate in the study are designed to achieve maximum participation. Once the sample of 60 PHAs has been identified, HUD will send a letter to the Executive Director and HCV Program Director of each agency to invite and encourage them to participate in the study. (A copy of the letter is provided in Appendix C.) The letter will provide an overview of the purpose of the study and highlight what will be required of participating PHAs. The letter will be accompanied by a brief description of the study in the form of a Frequently Asked Questions document (also provided in Appendix C.) The letter will be followed by a telephone call from a senior member of the Abt Associates team, who will speak with the Executive Director and HCV Director to describe the study again and request the agency’s participation. (A copy of the telephone script is provided in Appendix D.) If the PHA requests it at this stage, we will send the agency a copy of the PHA interview guide and a list of documents to be obtained from the PHA. Otherwise we will send these materials to the PHA as part of preparing for the site visit.

For all PHAs that agree to participate, Abt will prepare a document describing the obligations of the PHA for the study. We will ask the PHA to review the document carefully and share it among key HCV program staff. The Abt contact for the agency will then hold a follow-up call with the agency to ensure that staff understand their responsibilities for the study. The purpose of developing the document and vetting it thoroughly with agency staff is to minimize the chances that a PHA will participate in the first phase of the study but drop out prior to the full national study. Given that at least part of the sample for the full national study will be drawn from the initial sample of 60 PHAs, it is important that PHAs remain engaged for the full duration of the study. Once the follow-up call is complete, the research team will proceed with making arrangements for the site visit.

Should a PHA decline to participate in the study, we will identify a replacement PHA from the backup sample and begin the recruitment process anew with that PHA. Based on feedback obtained from HUD and the industry, we do not foresee major challenges to recruiting PHAs to the study. We believe that most PHAs are very interested in the study and will be motivated to participate. However, some PHAs may decide that the burden on staff is too high or may be reluctant to participate in a study that requires sharing information about the agency’s cost structure. The senior staff members responsible for agency recruitment will provide clear guidance to the PHA on the safeguards in place to protect sensitive information and the expectations regarding level of effort by the PHA.

We do not think that monetary compensation to PHAs is warranted given the time burden for PHAs to participate. The largest burden will be on the five beta-test sites, where additional data collection through staff timesheets, random moment sampling, and direct observation will take place over one week.¹

B4 Tests of Procedures or Methods

Drafts of the data collection instruments have been reviewed by HUD personnel, Abt Associates staff, and two advisory panels (see Exhibit A-3) to ensure that the instruments are clear, flow well, and are as concise as possible. In addition, the data collection instruments submitted in this package will be pre-tested at three PHAs.

B5 Statistical Consultation and Information Collection Agents

HUD’s Office of Policy Development and Research will work with the contractor, Abt Associates, to conduct the proposed data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Government Technical Representative (GTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star can be contacted at (202) 402-5705 and (202) 402-6139, respectively. The Abt Associates Principal Investigator is Dr. Meryl Finkel, who can be reached at (617) 349-2380.

1 We do expect to provide financial compensation for PHAs participating in the full national study. For the full study, we expect to collect data on staff time per activity over a longer period of time than the beta-tests, probably two months. The amount of compensation for PHAs participating in the full study will be determined based on the final research design and data collection approach for the study. We will prepare a separate request for OMB approval for the full national study.

Abt Associates Inc. Part B. Statistical Methods 6