HCV Admin Fee_OMB Revised Draft_Part B_3-26-12

Request for OMB Approval for Additional On-Site Data Collection for HCV Administrative Study

Part B. Statistical Methods

B1 Potential Respondent Universe

The respondent universe for additional on-site data collection site visits is up to 45 PHAs of different sizes and in different parts of the country that are believed to operate high-performing HCV programs.

Rationale for Visiting Up to 45 PHAs

The rationale for visiting up to 45 PHAs is to select additional high performing and efficient HCV programs to achieve a data collection sample of up to 60 programs for the full national study that is representative (of high performing HCV programs) by HCV program size, location, and program type. From the reconnaissance phase, we identified 31-35 PHAs that met the study criteria for high performance and efficiency and that have large enough HCV programs to be considered for the full study.¹ Our current sampling plan for the full study calls for collecting data from up to 60 PHAs. As a result, we need to identify an additional 25-29 PHAs that meet the study’s criteria for high performance to be included in the full study, plus several more PHAs to serve as backup agencies in case any of the PHA’s approved in the first reconnaissance phase declines to participate (so we need to identify approximately 34 PHAs in total). This OMB request is to conduct site visits similar to those done in the reconnaissance phase at up to 45 PHAs to identify another 34 PHAs for potential inclusion in the full study.

The reason that we need to identify 34 additional PHAs for the full study is that we need at least 25 high performing agencies to make up the current gap between the 31-35 agencies identified as viable candidates in the first stage and the up to 60 PHAs needed for the full study. The reason we need to visit 45 agencies to identify the 34 additional PHAs is that we expect (based on the findings from the reconnaissance phase) that only about three-quarters of the agencies visited will meet the criteria for inclusion in the full study and a number of PHAs confirmed in the first reconnaissance site visits might decline to participate in the full study.

Rationale for Focusing on High Performers

The reason for limiting the study sample to PHAs with high-performing HCV programs is that HUD intends the full national study of administrative fees to be based on high-performing and efficient programs. The full national study will provide cost information to inform the development of a new administrative fee formula for the HCV program, and HUD wants to model administrative costs at only those HCV programs that are high performers. HCV programs can be non-high-performing for a wide variety of reasons and HUD does not want to spend the limited resources available for this project studying program administration at agencies known not to be high performers.

Selection of the Sample of 45 PHAs

The full national study of administrative fees in the HCV program—which will be the subject of a separate OMB request—will ultimately be based on costs observed at a group of HCV programs that are high-performing and efficient. As a result, the additional site visits will focus, to the extent possible, on HCV programs that are believed to be high-performing and efficient.

The sample of 45 PHAs for additional on-site data collection will generally be selected in a manner similar to the reconnaissance sample selection in the first phase of the study. The sample of 45 PHAs will be drawn using random stratified sampling from the universe of HCV programs that were designated as “high performers” by HUD’s SEMAP system in three of the four years 2007, 2008, 2009, or 2010.

SEMAP system is the main source of administrative data on overall HCV program performance available for the universe of HCV programs. The SEMAP system scores HCV programs on a combination of administrative data measures reported to HUD through the Public and Indian Housing Information Center (PIC) system and self-reported measures. Each program receives a rating of “High Performer,” “Standard Performer,” or “Troubled” based on its SEMAP score. Most, but not all, programs receive a SEMAP certification every year

HCV programs with “High Performer” ratings on SEMAP in three of the past four years (2007-2010) will form the basic sampling universe for the additional on-site data collection. PHAs that don’t have SEMAP certification for all four years (such as small programs) will also be included if they are SEMAP high performers in all the years in which data were reported.

The following PHAs will be excluded from the sampling universe:

• PHAs in the Moving to Work (MTW) demonstration (37 PHAs). HUD has determined that MTW PHAs demonstration should not be included in the study. Although many MTW agencies are believed to be high-performing and efficient, MTW agencies have been given flexibilities that allow them to streamline their operations to increase efficiency.

• PHAs that were visited as part of the reconnaissance phase (59 PHAs). This is because these agencies have already been confirmed as meeting or not meeting the study’s performance and efficiency criteria.

• PHAs that were rejected during the field office review or HUD review of the reconnaissance sample (17 PHAs) and PHAs that were recruited to participate in the reconnaissance phase and declined to participate (32 PHAs).

This gives us a starting universe of 1,125 “high-performing” PHAs. Since we only need to conduct additional site visits to PHAs with at least 250 vouchers,² PHAs with fewer than 250 units (370 additional PHAs) will also be removed from the sampling universe. In summary, the universe from which we will select PHAs for additional on-site data collection includes 755 HCV programs.

Exhibit B-1 shows the region and size characteristics of the 755 SEMAP high-performing programs relative to all HCV programs which represent the sampling universe.

Exhibit B-1. Description of Sampling Universe for Additional On-Site Data Collection for the Administrative Fee Study

	SEMAP High-performers 2007-2010 with at least 250 vouchers (N=755)*		All HCV Programs (N=2,334)
	Number	Percent	Number	Percent
Census Region
Northeast	178	24%	642	28%
Midwest	182	24%	607	26%
South	262	35%	812	35%
West	133	18%	273	12%
Program Size
1 - 49 units			251	11%
50 - 249 units			915	39%
250 - 499 units	260	34%	410	18%
500 - 1,249 units	309	41%	413	18%
1,250 - 9,999 units	182	24%	320	14%
10,000 or more units	4	1%	25	1%
Agency Type
Combined	518	69%	1,452	62%
Section 8	237	31%	877	38%
State Agency
Not a State Agency	738	98%	2,298	98%
State Agency	17	2%	36	2%
Comparison of Average (Col A & B) Administrative Fee to Universe Median
Lower Than Median (10%-)	117	15%	534	23%
Median (+-10%)	321	43%	933	40%
Higher Than Median (10%+)	317	42%	866	37%
Special Purpose Vouchers
No Special Purpose Vouchers	171	23%	1,092	47%
Has Special Purpose Vouchers	584	77%	1242	53%
Median Distance of Units from Office
<5 mile median distance	537	71%	1679	72%
5 - <10 mile median distance	115	15%	304	13%
10+ mile median distance	101	13%	347	15%
Two-Bedroom FMR
Up to $600	103	14%	656	28%
>$600 - $800	309	41%	780	33%
>$800 - $1000	169	22%	428	18%
>$1,000	173	23%	467	20%

Notes:

PHAs exclusions: MTW PHAs (37 PHAs); PHAs that were rejected by Field Office review or HUD (17 PHAs); PHAs that were in the reconnaissance phase (59 PHAs); and PHAs that refused to participate in the first round (32 PHAs).

From the sampling universe of 755 HCV programs, the study team will select a stratified random sample of 135 agencies using HCV program size, program type, and location as the sampling strata.

We will then send the list of 135 agencies to the HUD Field Office and headquarters staff to be reviewed for audit findings. PHAs with major unresolved Independent Public Accountant (IPA) audit findings or fair housing findings from Fair Housing and Equal Opportunity (FHEO) reviews in the past three years related to the HCV program will be excluded from the sample. A starting sample of 135 PHAs allows for backups should any of the initial sample of 45 decline to participate or get rejected by the HUD field office and headquarters staff. We will replace the PHAs in the initial sample that were eliminated through Field Office and headquarters review with the designated backup agency. Also, if a PHA in the initial sample refuses to participate, the agency will be replaced by the backup that is the closest match in terms of HCV program size, region, and PHA type.

We will keep track of the PHAs that refuse to be part of the full study and compare their characteristics to those that are part of the study sample. We will also ask the PHAs the reason for their refusal to participate in the full study. These data will be used to assess whether there appear to be any systematic differences between participating and non-participating programs, and to correct for any differences found using analytic weights.

The final sample of up to 60 PHAs for the full national study is intended to be representative of the overall universe of high-performing PHAs along HCV program size, location, and program type. Although we will not be sampling explicitly for these characteristics, we anticipate that the final sample of PHAs will exhibit variety along the following dimensions believed to affect program costs and efficiency:

• Type of administration (local PHA, state/regional administration, non-profit administrator, independent PHA vs. PHA that is operated within a local government);

• Number of HUD and other programs administered;

• Physical size of service area;

• Whether staff are subject to collective bargaining or not;

• Special voucher programs administered (e.g., project-based vouchers, mainstream vouchers for persons with disabilities, homeownership vouchers, family unification vouchers);

• Program goals (for example, deconcentration or mobility goals, goals related to serving hard to house populations);

• Characteristics of client population (age, disability, family size, non-English speakers);

• Characteristics of housing stock (age, condition, type, vacancy rates);

• Characteristics of landlord population (for example, whether the rental stock is owned by a few large landlords or many small ones); and

• Local Legal Aid environment.

B2 Statistical Methods

B2.1 Sampling Plan

As noted, the study data collection for this project includes a sample of 135 HCV programs from which we will visit up to 45 PHAs to collect information on program performance and areas of efficiency, with the purpose of identifying additional study sites for the full national study of HCV administrative costs.

B2.2 Justification of Level of Accuracy

Data from the site visits will not be combined in any way to provide statistical or summary data for the HCV program as a whole. The data will be used to confirm (or not) the high-performing status of each PHA and to collect information on program efficiency.

The full national study will involve regression modeling to identify the factors that affect HCV program administrative costs and the relative weight that should be applied to each in an administrative fee formula. This analysis will be discussed in the OMB request developed for that part of the study.

B3 Maximizing Response Rates

Our procedures for recruiting PHAs to participate in the study are designed to achieve maximum participation. Once the initial sample of 45 PHAs has been identified, HUD will send a letter to the Executive Director and HCV Program Director of each agency to invite and encourage them to participate in the study. (A copy of the letter is provided in Appendix A.) The letter will provide an overview of the purpose of the study and highlight what will be required of participating PHAs. The letter will be accompanied by a brief description of the study in the form of a Frequently Asked Questions document (also provided in Appendix A.)

The HUD letter will be followed by a telephone call from a senior member of the Abt Associates team, who will speak with the Executive Director and HCV Director to describe the study again and request the agency’s participation. (A copy of the telephone script is provided in Appendix B.) If the PHA requests it at this stage, we will send the agency a copy of the PHA interview guide and a list of documents to be obtained from the PHA. Otherwise we will send these materials to the PHA as part of preparing for the site visit.

Should a PHA decline to participate in the study, we will identify a replacement PHA from the backup sample and strata and begin the recruitment process anew with that PHA. Based on our experience in the reconnaissance phase, we believe that most PHAs are very interested in the study and will be motivated to participate. However, some PHAs may decide that the burden on staff is too high or may be reluctant to participate in a study that requires sharing information about the agency’s cost structure. The senior staff members responsible for agency recruitment will provide clear guidance to the PHA on the safeguards in place to protect sensitive information and the expectations regarding level of effort by the PHA.

We do not think that monetary compensation to PHAs is warranted given the time burden for PHAs to participate. We do expect to provide financial compensation for PHAs participating in the full national study. For the full study, we expect to collect data on staff time per activity over two months. The amount of compensation for PHAs participating in the full study will be determined based on the final research design and data collection approach for the study. We will prepare a separate request for OMB approval for the full national study.

B4 Tests of Procedures or Methods

Drafts of the data collection instruments have been reviewed by HUD personnel, Abt Associates staff, and the Expert and Industry Technical Review Group (EITRG) to ensure that the instruments are clear, flow well, and are as concise as possible. The on-site protocol and performance criteria will be substantially the same as used during the first reconnaissance phase (OMB Control Number 2528-026), except that the PHA interview protocol is shorter, eliminating all questions that do not directly relate to program performance.³

B5 Statistical Consultation and Information Collection Agents

HUD’s Office of Policy Development and Research will work with the contractor, Abt Associates, to conduct the proposed data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Government Technical Representative (GTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star can be contacted at (202) 402-5705 and (202) 402-6139, respectively. The Abt Associates Principal Investigator is Dr. Meryl Finkel, who can be reached at (617) 349-2380.

1 Overall, 45 of 59 PHAs met the study’s high-performance and efficiency criteria, but 8 are likely to be too small for inclusion in the full study (less than 100 vouchers) and 2 dropped out. Another 4 PHAs may be too small for inclusion in the full study (less than 250 vouchers), but this is being tested in the study’s pretest.

2 At present, we have enough smaller programs confirmed as high performers from the reconnaissance phase of the study. However, depending on the results of the pretest we might determine that additional small programs are needed, in which case we will include smaller programs in the sampling universe.

3 For the 59 reconnaissance site visits, the study team asked a series of questions about program processes and cost drivers that are being used in the design of the full administrative fee study; these questions have been eliminated for the purposes of recruiting additional agencies to the study.

Part B. Statistical Methods ▌pg. 11