Generic Customer Satisfaction Surveys

SURVEY DESIGN & PROTOTYPE REPORT

Annual Customer-Service Survey of

Performance-Based Contract Administrators

Draft

November 25, 2011

Submitted to:	Submitted by:
U.S. Department of Housing and Urban Development	Insight Policy Research, Inc.
Office of Housing, Procurement Management Division	1901 N. Moore Street
451 7th Street S.W.	Suite 601
Washington, DC 20410	Arlington, VA 22209

TABLE OF CONTENTS

I. INTRODUCTION 1

A. BACKGROUND 1

II. OVERVIEW OF THE SURVEY DESIGN 2

A. RESEARCH QUESTIONS 4

B. PERFORMANCE METRICS 6

III. DATA COLLECTION METHODOLOGY 7

A. PROPERTY OWNERS: PBCA OWNER /MANAGER SATISFACTION SURVEY 8

B. TENANTS: PBCA TENANT SATISFACTION SURVEY 11

C. TIMELINE 13

D. CHALLENGES 14

Iv. SAMPLE DESIGN PLAN 15

A. PROPERTY OWNERS: PBCA OWNER/MANAGER SATISFACTION SURVEY 16

B. TENANTS: PBCA TENANT SATISFACTION SURVEY 19

C. PRECISION/ RELIABILITY OF ESTIMATES 21

V. data analysis plan 22

VI. final report 23

APPENDIX A: OWNER/MANAGER SURVEY INSTRUMENT (DRAFT )

APPENDIX B: TENANT SURVEY INSTRUMENT (DRAFT)

I. INTRODUCTION

This document details our proposed design plan for the annual Customer Service Surveys of Performance-Based Contract Administrators (BPCAs). The purpose of this annual Customer-Service Survey of PBCAs is critical in enabling the U.S. Department of Housing and Urban Development’s (HUD) Office of Housing Assistance Contract Administration Oversight (HACAO) Multifamily Housing Programs (MFH) to decide which PBCAs have earned an Annual Incentive Fee for customer service. To accomplish this, Insight Policy Research (Insight) will develop and implement confidential customer-service surveys regarding PBCAs. This plan details our approach to developing and conducting surveys for two key stakeholder groups: owners and tenants.

Our proposed plan represents an approach that is based on our experience in designing and implementing comprehensive surveys. Chapter II presents an overview of the study and the research questions, including the survey goals and objectives and the customer satisfaction measures. Chapter III includes details on the data collection methodology and survey methods (e.g., data collection plans, data collection modes, nonresponse follow-up procedures, a timeline for the two surveys and any challenges to conducting the surveys). Chapter IV provides our sample design plan (e.g., target population, sampling frame, sampling, the statistical reliability of the survey results, etc.). Chapter V presents our plan for analyzing the data (e.g., themes of the analysis and proposed approach) and Chapter V presents our plans for developing a final customer-service survey report. .

We understand the need to work collaboratively with HUD throughout the duration of this contract in order to ensure successful execution of this study. With that understanding in mind, we will remain flexible and open to feedback as we refine our plan and move forward. We look forward to comments and suggestions.

A. BACKGROUND

The Section 8 Program, authorized by Congress in 1974 and administered by HUD, provides rental subsidies for low- and moderate-income eligible families. While some eligible families receive tenant-based vouchers to use toward rent in private-sector housing, other funding is paid directly to the property owners through the project-based Section 8 Housing Assistance Payments (HAP) Contracts. In these cases, HUD provides payments directly to property owners equal to the difference between the approved rental amount for the unit and the required rental contribution from eligible tenant families. In return, property owners are obligated to rent a specified portion of their units to Section 8 eligible families; maintain decent, safe, and sanitary housing for residents; and comply with various regulations and reporting requirements.

Compliance by property owners with program requirements is monitored by contract administrators under contract with HUD. In 2011, HUD began making these contracts performance based so that entities holding such contracts (e.g., public housing agencies or joint ventures including PHAs) would be incentivized to work with owners to ensure decent, safe, and sanitary housing under the Section 8 program. In March 2011, HUD issued an Invitation for Applications for Performance-Based Contract Administrators (PBCAs). Following the review of applications, PBCAs were awarded to PHAs covering nine States: Iowa, Maine, Minnesota, Montana, New Hampshire, North Dakota, South Dakota, Vermont, and Wyoming as well as Puerto Rico and the U.S. Virgin Islands. The PBCAs administer HAP contracts within their geographic boundaries. The principle tasks of the PBCAs include efforts to:

1. Monitor project owners’ compliance with their obligations and responsibilities to provide decent, affordable housing for eligible families;

2. Calculate and pay Section 8 HAP payments (e.g., rental subsidies) to property owners, submitting required documentation to HUD; and

3. Comply with Federal law and HUD regulations.

The awarded contracts, which are entitled ‘Performance-Based Annual Contributions Contracts (PB-ACCs)’, specify the responsibilities of the PBCA and the fees that HUD will provide to the PBCA in exchange for undertaking the principle tasks. In addition to the Basic Administrative Fee, the PBCA may also incur Annual Incentive Fees for customer service. The Annual Incentive Fee for customer service is equal to 5 percent of the sum of the Basic Administrative Fee earned during each 12-month period of the contract term. The Annual Incentive Fee can be earned for the performance of the following tasks:

• Management and occupancy reviews

• Adjust contract rents

• Review and pay monthly vouchers

• Renew HAP contracts and process any terminations or expirations

• Tenant health, safety, and maintenance issues

• Administration requirements including: 1) monthly and quarterly reviews; 2) ACC year-end reports and certifications, and 3) annual financial reports – PHA fiscal year end

II. OVERVIEW OF THE SURVEY DESIGN

Insight’s approach to the conduct of customer-service surveys of the owners and tenants is based on the best means of reaching a sample of each group that will yield results with sufficient precision to provide a basis for HUD to decide which PBCAs have earned an annual incentive fee for customer service. Critical to this approach is determining the survey mode by which the sample frame is best reached at reasonable cost. This depends on the target population, the type and quality of contact information available for them, the motivation of the population to participate, the self-perceived time to participate in the survey, and the cost per completed interview.

In summary, the strengths and weaknesses of each type of survey mode—and our recommendation of its applicability for the survey of PBCAs—are as follows:

• Mail surveys have a low cost per completed interview but typically achieve a low response rate, unless the population is highly motivated to participate. Response rates to selected populations can be increased with a prenotification letter sent about 1-2 weeks before the survey is mailed and the inclusion of a small cash incentive in each survey. Still, mail surveys can be easily ignored and misplaced. In addition, when returned, they must be key-entered or scanned, which adds to cost. Another potential complication is other languages; sending out an instrument in multiple languages can be costly, so typically, respondents are given the option to request the survey in another language via postcard or a toll-free telephone call.

Recommendation: Insight does not recommend the use of mail surveys for either the owner or tenant surveys. Other approaches, as explained below, are more suitable and more cost-effective.

• Web surveys are similar to mail surveys in their strengths and weaknesses, with the added factor of Web access. A customized Web address with a built-in individual identification code is sent to each person in the sampling frame. If email addresses are available, the notification can be sent by email and a Web link can be clicked on or copied and pasted into a browser for easy access. If no email is available for delivery, typing in a customized URL contained in a prenotification letter can be done. Web surveys generally have the added benefit of built-in edits and require no manual processing and data entry. It is quite easy to offer Web surveys in another language. Web surveys may be followed up with telephone calls to increase the response rate.

Recommendation: Insight recommends Web surveys for the owner survey. Since emails addresses are available for this group and they are highly likely to have Web access, Web surveys offer a cost-effective means of reaching this group. A combination of prenotification letters, email reminders, and telephone follow-up calls will maximize the response rate.

• Telephone surveys provide an opportunity for immediate completion and accurate administration using computer-assisted telephone interviewing (CATI) and the benefits of human interaction. Messages can be left via a toll-free callback number; and telephone surveys can be offered in languages other than English---for example, nearly all major U.S. telephone survey operations offer interviews in Spanish. One limitation is the availability of a telephone number. However, with accurate addresses, telephone numbers may be located. Several private sources offer a service to identify landline and/or cell-phone numbers associated with a person/address for a small fee. While the cost of using cell phone minutes used to be a concern among some populations, that concern has diminished due to the popularity of plans with unlimited or generous minutes.

Recommendation: Insight recommends CATI for the tenant survey. The tenant survey will be offered in Spanish as well as English. While lists provided by owners or their managing agents will include telephone numbers, additional telephone numbers will be purchased as required.

• In-person interviews are the most costly and generally recommended for obtaining and documenting sensitive information, as well as for long interviews. Lack of a telephone number is not a barrier, although it is costly to send an interviewer to an address compared to scheduling an interview by telephone.

Recommendation: Insight does not recommend in-person interviews for either survey.

Following, we present the research questions and performance metrics for the study. The draft survey instruments are contained in Appendix A (Survey of Property Owners) and Appendix B (Survey of Tenants).

A. RESEARCH QUESTIOnS

Below, we present a comprehensive list of PBCA customer-service survey outcomes to be measured. These outcomes served as the input for the development of the survey questions. They are based on the nature of interactions between PBCAs and owners and tenants, respectively, and the contractual responsibilities of the PBCA to the respondent. They focus on attributes of PBCA customer service that are important and relevant to owners and tenants. The questions/issues are shown respectively for the survey of owners and the survey of tenants living in project-based Section 8 housing. These topics represent our current thinking about the best way to address the study’s objectives and are subject to revision based on discussion with, and feedback from, the COTR and other knowledgeable HUD staff.

1. Survey of Owners

Demographics/Background

• Years of participation in the Section 8 HAP program (total, housing development sampled)

• Number of housing developments owned currently in the HAP program

• Number of Section 8 units in HAP program

Familiarity/Contact with PBCA

• Familiarity with PBCA’s responsibilities

• Amount of direct contact/communication with PBCA during the last 12 months

Performance of PBCA

• Level of satisfaction with the PBCA on:

• overall performance

• how knowledgeable they have been about Section 8 HAP issues

• how responsive they have been addressing and resolving your questions and inquiries

• the HAP contractor personnel’s level of courtesy and professionalism

• ease of access by phone

• Responsiveness to questions or requests made in person or by phone, or sent by postal mail or email

• Timeliness of payments

• Timely receipt of summary report of Maintenance and Occupancy Review (HUD-9834) and fairness of it

• Timeliness of processing requested rent adjustments

• Notification in writing of rent adjustment decisions

• Timeliness of rent adjustment decision appeals within 30 days

• Level of satisfaction/ fairness in deciding owner’s rent adjustment requests

• Timeliness of utility adjustment decision appeals

• Level of satisfaction/fairness in deciding owner’s utility adjustment requests

• Timeliness of reimbursement claims for . . .

  • unpaid rent

  • resident damage

  • vacancy losses

• Level of satisfaction/fairness overall in deciding owner’s reimbursement claims

• Notification in writing about required corrections if needed on vouchers submitted for payment

• Receipt of an original copy of the signed contract upon applying for a HAP contract renewal

• Timeliness of contacts regarding tenant concerns/inquiries regarding health, safety, or maintenance

• Examples, if any, of exceptional service or expertise in helping achieve a notable success or alleviate a serious problem at this property (if they did)

• Examples, if any, of failure to fulfill responsibilities, resulting in extended a problem or preventing/delaying an achievable outcome

• Overall rating of the HAP contractor

2. Survey of Tenants

Housing Environment/Demographics

• Years lived in your current rent-subsidized housing

• Years lived in any subsidized housing

• How often there is excessive noise around the property

• How often there is fighting and other disturbances is a problem

• Amount of theft and other property crime

• Level of safety from personal violence/harm

• Household size (number of children, number of adults)

• Respondent’s age

• Respondent’s race/ethnicity

Familiarity/Contact with PBCA

• Familiarity with the PBCA

• Contact with PBCA

Performance of PBCA

• Overall rating of the HAP contractor

• Rating of the HAP contractor on . . .

• respectful treatment

• responsiveness to questions and inquiries

• professional conduct of personnel

• ease of access by phone

• Satisfactory responses/actions to most recent contact

• Examples, if any, of exceptional service that was helpful to you

• Examples, if any, of poor service that did not help you with problem

The responses to questions based on these topics will be used in calculating customer service scores, which in turn will be used as metrics for determining the eligibility of PBCAs for incentives.

B. PERFORMANCE metricS

The key satisfaction questions are designed in multiple-choice format, using ordered-categorical response options in which a graduated series of categories is available to choose level of satisfaction. There are an equal number of positive statements (e.g., those used to express satisfaction) and negative statements (e.g., those used to express dissatisfaction). We have used the forced-choice option using four item categories. In this increasingly standard approach, a neutral response option (e.g., neither satisfied nor dissatisfied) is not offered, forcing the respondent to choose an either negative or positive response option. This is designed to counter the tendency of some persons to choose the middle option, if available. We have chosen to provide four response categories rather than six, as they present easily categorized choices that are unambiguous. Scales with six response options (e.g., very satisfied, satisfied, somewhat satisfied, somewhat dissatisfied, dissatisfied, very dissatisfied) may introduce more uncertainty rather than less, as the respondent tries to determine the difference between closely worded options (e.g., satisfied and somewhat satisfied). The selection of this response option was made to foster respondents’ understanding and to reduce uncertainty associated with the answers. Owners may be somewhat reluctant to “rate” PBCAs, especially given their reliance on PBCAs and the financial incentives associated with the ratings. Reducing ambiguity makes it easier for owners to answer the questions. Tenants may be uncertain about the role of the PBCA and have limited exposure to them. While we ask them about their familiarity, we still need to encourage a response on rating questions. Simple, unambiguous categories help to encourage a response.

In scoring the questions, we will treat the categorical ratings (e.g., very satisfied, satisfied, etc.) as ordinal data which are amenable to frequencies, percentages and, as we propose, means. Interval data, such as that in the bipolar 0 to 10 overall rating scale in the owner survey (where 0 represents very poor service and 10 represents excellent service) assumes an equal-interval between each numerical rating and thus provides additional analytic options. As appropriate we will combine the responses from several different categorical satisfaction items to express a summed rating. This may permit HUD to look at individual question scores as compared to anoverall summed score.

In addition to these general satisfaction “ratings”, there are some items used only in the owner’s survey that assess PBCA compliance with actions that are specified under the terms of the contract. These items are more amenable to yes/no questions. In addition, the owner instrument also collects background information on these actions and related topics, which may be useful in analyzing the data.

III. DATA COLLECTION METHODOLOGY

This section consists of an overview of the Insight team’s plans to ensure a sound and thorough customer-service survey of PBCAs. Effectively assessing the PBCA customer satisfaction with the different population groups with which the PBCA interacts (e.g., property owners and tenants) requires a multipronged study approach; this will enable us to assess the key satisfaction measures for these 2 groups. In this section we will:

• Describe our data collection plan. The selection of a data collection plan will take into account the kind of data needed (e.g., quantitative or qualitative), the target audience (e.g., are they Internet users, do they have access to a telephone?), the project schedule, and the data collection budget.

• Construct a timeline for carrying out the two surveys.

Below, we describe our proposed approach to each of the customer populations, based on the population and what we know about it and in consideration of the characteristics of each survey mode. Section A provides the plan for conducting the PBCA owner/manager satisfaction survey and Section B provides plans for conducting the tenant survey.

A. property owners: PBCA owner/MANAGER SATISFACTION SURVEY

A Web-based satisfaction survey will be conducted in each of the 11 States to obtain information on the satisfaction of owners/managers with the PBCAs. The nature of PBCA interaction with owners (and any management agents responding on behalf of owners) means that there will be ongoing communication and, presumably, sufficient opportunities for assessing satisfaction. While there is some risk that there is recall bias, in which recent events may seem more important than earlier ones, the retrospective nature of the survey, covering the past year, precludes attempts to counter it.

Insight recommends using a Web survey to obtain input from the owners and management agents regarding the customer service provided by the PBCAs. We believe that Web administration is the most efficient means to reach this group for a few reasons. First, Internet surveys are the most cost-effective data collection method and are ideally suited for persons in management and professional positions. Second, Internet surveys have the advantage of reducing measurement error and resulting bias. The absence of an interviewer reduces the possibility of obtaining socially desirable responses—the tendency on the part of the respondent to give what they interpret as the socially correct answer. Third, owners may be more motivated to respond because of their regular interactions with the PBCA. Last, we will be able to compile a listing of the owners with contact information, including email addresses.¹ All owners should have access to the Internet which is essential for their business interactions and communications with the PBCA and HUD.

Insight has subcontracted with ICF to conducts the surveys; ICF is a major provider of Web and telephone surveys for Federal government agencies. ICF uses SPSS Dimensions, an automated Web-based survey platform, to conduct Web surveys. This platform is linked to ICF’s complete sample management system, which includes automated email invitations and reminders. The survey will be posted to a Web site owned and maintained by ICF so that owners can submit survey responses electronically. This software suite allows the programmer to quickly and easily create professional-looking surveys that are easy to navigate and flexible enough to make quick modifications as needed. SPSS Dimensions has extremely high levels of customizability to allow for the programming of any skip patterns and survey logic. ICF’s Web servers are available 24 hours a day, 7 days a week, so respondents may complete the survey at any time that is convenient for them. Respondents who have questions about the survey are offered up to three methods of support: 1) Frequently Asked Question (FAQ) pages (which may be accessed from the questionnaire) and technical support to ensure that participants can access the system and complete the Web survey as intended; 2) a dedicated email address that is accessed by multiple team members to ensure timely response; and 3) a toll-free phone number (with voicemail) for participants to call to speak directly with a help-desk team member.

Insight will use proven methods to increase response rates for the survey, including multiple contacts, personalization, and guarantees of confidentiality. To make certain that owners do not ignore or accidentally delete the email, we will alert them to its imminent arrival with a prenotification letter for delivery about 1 week prior to the email. This letter will have the most impact if it comes directly from HUD and mentions that Insight is HUD’s authorized contractor for this study. We will prepare this letter on stationary with an official HUD logo obtained from HUD; we will then print and mail the letter in envelopes with a similar logo.

About 1 week after receipt of the prenotification letter, we will send an email to the owners with a customized link to the survey. Email notifications are used as an initial contact with potential respondents, to explain the purpose of the survey, to elicit cooperation, and to communicate a secure link as well as a unique ID and password that the respondent will use to complete the survey. The initial email will include the following items:

• The purpose of the survey,

• A statement of how the results will be used,

• A request for the respondent’s participation,

• A statement of promised confidentiality or anonymity,

• Detailed instructions for accessing the survey including a hypertext link address and a unique password for each respondent,

• The cut-off date for responses,

• Instructions to decline participation,

• A phone number and email address to use for technical support (provided by ICF Macro), and

• A phone number and email address for the HUD contact if the respondent has questions about the study’s validity.

Respondents may either click on the URL directly from the message or “copy and paste” the address into their Internet browser. The password feature serves several functions:

• It allows participants to begin the survey, suspend it, and re-enter the survey later at the point where they left off; the responses already entered will be saved.

• Similarly, it protects a participant’s data against power or network interruption, since responses are saved after each screen and participants can simply re-enter the survey using their password and begin where they left off.

• It ensures that only targeted respondents can complete the survey.

• Since each password can be used to complete only one survey, it safeguards the survey process against “ballot-box stuffing” by preventing multiple surveys.

Once participants reach the Web survey via the secure link contained in their email invitation, initial survey questions screen each respondent for eligibility. Those respondents who meet eligibility requirements continue to the main survey questions. Respondents may request another person in their organization to complete all or parts of the survey.

Reminder emails. If respondents do not complete the Web survey within a specified timeframe, additional reminder emails are sent. We typically recommend issuing an initial email invitation, followed by up to two email reminders to nonresponders, to maximize response rates. The reminder emails are very similar in content to the initial email invitation, restating the survey cut-off date and further emphasizing the importance of everyone’s participation. We will develop the appropriate email invitation text for a schedule of reminder emails that works with the overall timeline available for Web survey fielding. Any respondent who completes the survey via Web is identified as complete and removed from subsequent follow-up contact.

Telephone follow-up. If there is still no response within approximately 1 week of the second email, we will begin telephone follow-up. We will make up to three telephone calls to the owner or manager, requesting them to complete the survey. If we do not reach the sampled person in these attempts, we will leave a voicemail reminder. If we do reach the respondent live and can convince that person to complete the survey by telephone at that time, we will administer the survey by phone and enter the response into the Web survey form directly. If owners prefer to use the Web version, they will still have the option to do so. Insight estimates that approximately 30 to 40 percent of the owners will respond to the survey after the prenotification letter, an initial email, and a reminder email. The telephone follow-up should double the estimated response rate for a total response of 60 to 80 percent, a very good response level for a Web survey. With the anticipated high motivation among owners to rate the performance of the PBCA, we should come close to the 80 percent response rate target desired by OMB. If the 80 percent target is not reached, we will conduct non-response bias analysis to determine if there is any systematic difference (using available data) between respondents and nonrespondents.

Questionnaire design. The owner survey contains approximately 30 items, including background questions, and will take approximately 15 minutes to compete on the Web or by phone. The survey questions are customized to reflect their owners’ interests and perspectives and the nature of their interactions with the PBCA and the responsibilities of the PBCAs under the HAP contracts. Most questions are close-ended (as opposed to open-ended), with multiple-choice-type responses from which to choose. There are two open-ended questions at the end of the survey for any additional comments.

The survey will be designed to minimize response-time burden to the extent possible and thus maximize the response rate. Respondents to each of the surveys will be assured that their responses will be confidential and that as a result their responses will have no impact on their participation in the Section 8 rental subsidies program.

B. TENANTS: PBCA TENANT SATISFACTION SURVEY

A satisfaction survey will be conducted through the use of Computer Assisted Telephone Interviews (CATI) in each of the 11 PBCA coverage areas to obtain information on the satisfaction of tenants with the PBCAs. Efforts will be made to differentiate between tenant satisfaction concerning issues that may involve the PBCAs and issues that concern tenants but that are unlikely to require the attention of the PBCAs; for example, immediate maintenance requests.

Insight recommends using CATI to obtain feedback from tenants that may provide helpful input on the performance of HAP contracts and thus on the performance of the PBCAs themselves. Tenants may have little or no awareness of the PBCA, but they are in a position to provide feedback about their interactions with the owners/building managers and their operation of the buildings. Since the PBCAs provide standards for the contracts they administer, they are ultimately responsible for the performance of those they monitor.

CATI is an efficient way to reach a substantial number of respondents, where the sampling frame is sufficiently large and the contact information is adequate to provide a reasonably high response rate. Phone numbers are likely to be accurate given our assumption that owners/management agents maintain current telephone contact information for their tenants. Still, we acknowledge that cell-phone numbers, which may be the primary telephone of many of these tenants, are less stable and more likely to change than landlines. However, based on our experience with low-income populations; we find that those living in large, highly urban areas tend to change cell-phone numbers much more frequently than those living in the medium-to-small communities that dominate in these PBCA States. Thus, we plan to utilize standard locating and “skip tracing” procedures to verify the telephone numbers in the list and obtain additional numbers, where possible.

Training interviewers. Once the final survey instrument is approved, ICF will transform the survey document into a CATI instrument and conduct several rounds of testing. After testing is completed, ICF will provide Insight with a CATI “demo” for review. The CATI demo is a Web-based copy of the survey instrument, accessible only by designated individuals using a secure Web site. This provides an opportunity for project staff to review and navigate the survey exactly as the telephone interviewer would during data collection. ICF will develop a project-specific training course and training manual for this survey, which will cover background on the rental assistance program, the goals of the study, the survey instrument, how to deal with uncooperative respondents, scheduling of callbacks, referral procedures (if participants request validation or more information), and any special sensitivities for the study. This manual will include project-specific materials developed by Insight to familiarize the interviewers and project staff with the project and potential barriers to participation that could be overcome during refusal-conversion efforts. An Insight project representative will participate in the interviewer training. ICF will also conduct an operational pre-test with intensive monitoring from project management staff, quality assurance staff, and supervisors. This likely will reveal any issues that may require attention prior to full survey administration. If changes are necessary, Insight will modify the survey until it is correct.

Follow-up methods. We propose a multipronged strategy for ensuring strong response rates, including 1) obtaining the most current contact information from property administrative records; and 2) use of respondent telephone number retrieval/locating techniques as needed. Insight will design the materials and interview scripts to convince sample members that the survey feedback will provide some benefit to them as tenants and that their rental assistance benefits will not be affected by their responses. ICF call center interviewers are trained in refusal-conversion techniques and will utilize a wide range of methods to minimize nonresponse and maximize the complete data available for analysis. Procedures to maximize the response rate include the following:

• 7-attempt protocol on different days/at different times of day. Research shows that the incremental increase in response rates diminishes beyond seven calls. Messages will be left for recipients to call a toll-free number to complete the survey. After seven attempts to reach a number, a replacement number will be used if available.

• Call rotation and flexibility. The CATI system can schedule calls to rotate among various times throughout the day and evening during callbacks. The system allows respondents to call in to complete a survey or continue a survey over multiple sittings. Interviewers can also schedule appointments so that respondents can participate at a time convenient to them.

• Refusal conversion. ICF will work with Insight to implement refusal conversion appropriate to the needs of the project. The level of conversion will be communicated to interviewers as part of the training.

Questionnaire design. The survey is designed to collect data on customer service and satisfaction relevant to performance measures. For the tenant survey, we will use a screener to make certain that we are talking to an adult in the household who is involved in issues concerning the lease or maintenance request. The tenant survey contains approximately 23 items, including background questions, and will take approximately 10 minutes to compete by phone. Most questions are close ended (as opposed to open ended), with multiple-choice-type responses from which to choose. The survey will include two open-ended questions at the end to permit the respondent to provide any additional feedback as they wish.

C. TIMELINE

Under the terms of the contract, the surveys are scheduled for conduct between October 1 and October 31, 2012. While this timeframe is limited, it is sufficient to conduct both surveys. However, with a draft report due on November 30, 2012 this provides a very tight turnaround time for producing a data file, analyzing the results, and writing a draft report. To provide sufficient time for these critical deliverables, without extending the due dates of these deliverables, Insight would like to begin the data collection a full month earlier, no later than September 1, 2012 for the two surveys. Prenotification letters would be mailed to owners approximately 1 week prior to this date and training for the telephone survey of tenants would be completed by September 7 so that interviews could begin on September 10. Since OMB approval is expected to be received in July 2012, this change in the timeline would be doable, should HUD approve.

Key dates associated with this advance in the schedule are as follows:

• Obtain final list of owners from HUD with complete contact information: June 1, 2012 (If any input is needed from other sources, such as regional offices or PBCAs to ensure updated contact information, the process should be started approximately 2 weeks earlier.)

• Sample property owners: June 15, 2012

• Request list of properties for each sampled property owner: June 30, 2012

• Sample properties: July 1, 2012

• Request lists of tenants from owners for sampled property: July 2, 2012

• Initial deadline for receipt of tenant lists from owners: July 30, 2012

• OMB approval received: July 1, 2012

• Final deadline for receipt of clean/complete lists of tenants from owners: August 15, 2012

• Sample tenants: August 31, 2012

• Mail prenotification letters to owners: August 31, 2012

• Web survey of owners opens for use: September 4, 2012

• Owner survey telephone follow up: September 20, 2012

• Tenant survey training/pretests: September 7, 2012

• Tenant survey CATI telephone administration begins: September 10, 2011

• Survey period closes: October 15, 2012

• Editing: October 30, 2012

• Weighting: November 7, 2012

• Tabulation: November 15, 2012

The draft report would remain due as scheduled on November 30, 2012 and the final report and briefing would remain due as schedule on December 29, 2012.

D. CHALLENGES

Surveys of this type always have challenges. The design attempts to address each challenge and minimize its occurrence and any consequences.

Response Rate: Achieving a sufficient response rate is an issue intrinsic to surveys of all types, and may be especially challenging in satisfaction surveys. Owners have a direct interest in the PBCA and a history of contact with them that should help their motivation to respond. We have chosen the survey mode most appropriate for owners (Web) and included three efforts that in combination will encourage completion: a prenotification letter from HUD, email communication and email reminders, and telephone follow-up.

We also ask that HUD notify each regional office of the survey in an email introducing Insight and explaining the purpose of the study. It is important for local field offices to be aware of the owner and tenant surveys so that they can respond to any questions that are brought to the attention regarding the legitimacy of the study and any related questions.

Hesitancy of Rating the PBCA: While owners are familiar with the PBCA and should not have any trouble rating them, they may be reluctant to do so. To encourage their response we remind them that their responses are confidential and will not affect their HAP contracts in any way. Further, the survey instrument is structured with easier background questions up front, including those to assess level of contact/communication and a series of separate questions dealing with different topics of service on which to rate the PBCAs.

The issue with tenants is somewhat different because they may be less familiar with the PBCA. The instrument introduces the PBCA to them, assesses their familiarity/contact and then asks for their ratings. Furthermore, questions are clearly worded with few response options. Tenants, like owners, are also reminded that their responses are confidential and that their responses will not affect their Section 8 rental in any way.

Language: The surveys will be offered in both English and Spanish versions. Spanish translations will be provided following approval and acceptance of the English versions.

Compiling the Sampling Frame of Tenants. Having confirmed the availability of lists of owners/properties from HUD, we are confident the three-stage sampling procedure is appropriate and best suited to the study. The greatest unknown, and therefore challenge, is in obtaining the tenant lists from the owners. Clearly, owners will have such lists. There are 2 major challenges in obtaining these lists. First, the issue of their inclusiveness and accuracy is always a potential concern. We have addressed this to the extent possible by arranging for verification of telephone contact information and use of locating/skip-tracing services for additional phone numbers. A second concern is the owner’s willingness to share these lists and their timeliness in doing so. With HUD’s assistance—in the form of a signed letter on HUD letterhead that we can enclose with the request—and a timeline that provides sufficient time for the process, we can minimize these challenges. As noted, in the sample plan, we provide for “replacement” properties in the event we cannot obtain compliance of some owners in providing lists.

IV. SAMPLE DESIGN PLAN

The purpose of this section is to document the statistical procedures to be used for the HUD PBCA Customer Satisfaction Surveys. The sampling plan for the PBCA Satisfaction Surveys will be probability based so that study findings can be used to make statistically defensible inferences about the entire population of:

• Section 8 property owners (e.g., those who receive rental subsidies through the “project-based” Section 8 rental assistance program) that are administered under the 11 PBCAs awarded for FY2011.

• Section 8 tenants of these property owners.

Section 8 authorizes a variety of "project-based" rental assistance programs, under which the owner reserves units in a building for low-income tenants, in return for a Federal government guarantee to make up the difference between the tenant's contribution and the rent specified in the owner's contract with the government. Currently, nine States, (e.g., Iowa, Maine, Minnesota, Montana, New Hampshire, North Dakota, South Dakota, Vermont, Wyoming) Puerto Rico, and the U.S. Virgin Islands have awarded PBCA contracts in place, yielding a total of 11 PBCAs. The 11 PBCAs are:

• Iowa Finance Authority/EPS;

• Maine State Housing Authority;

• Minnesota Housing Finance Agency;

• Montana Department of Housing (PBS8 Housing);

• New Hampshire Housing Finance Authority;

• North Dakota Housing Finance Agency;

• Puerto Rico Housing Finance Agency;

• South Dakota Housing Development Authority;

• Vermont State Housing Authority;

• Virgin Islands: NTHDC (North Tampa Housing Development Corporation)/CGI;

• Wyoming: Housing Authority of the City of Cheyenne

The goal of designing the sample is to permit accurate statements regarding the overall satisfaction levels of building owners and tenants within each PBCA territory, so that results can be compared across the individual PBCAs. Thus, the sample design for this study will be based on the average satisfaction scores for each PBCA for the two key stakeholder groups, including both 1) owners and 2) tenants. We will also calculate, for both owners and tenants, the percent who are almost, or completely, satisfied with the PBCA.

The sample for the study was designed to achieve the following goals:

• The use of a three-stage sample design that produces a linkable analytical dataset of property owners, properties and their section 8 tenants.

• The development of owner customer satisfaction ratings with a 95-percent, two-tailed confidence intervals of between 5.9 and 9.8 percentage points by obtaining a sample size of 100 owners per PBCA.

• The development of tenant customer satisfaction ratings with a 95-percent, two-tailed confidence intervals of between 8.9 and 14.9 percentage points by obtaining a sample size of 150 tenants.

Steps involved in the sample design are briefly described below.

A. pROPERTY OWNERS: PBCA OWNER/MANAGER satisfaction SURVEY

1. Target Population

The target population for this study includes all property owners and Section 8 housing tenants residing in their properties in the 11 PBCA “States” who have received rental subsidies through the “project-based” Section 8 rental assistance program under the HAP Contracts that were administered by the 11 PBCAs in the current fiscal year (e.g., between October 1, 2011 and September 30, 2012).

2. Survey Eligibility

This study will include only those property owners and tenants who have received rental subsidies through the “project-based” Section 8 rental assistance program for which payment had been made under the HAP Contracts that were administered by the 11 PBCAs between October 1, 2011 and June 30, 2012 of the past fiscal year. Note: The lists of owners must be received by June 15, in order to select the sample and obtain the tenant sampling frame, and select the tenant sample, so that both surveys can be conducted during the month of October.

3. Sampling Frame

The sampling frame of property owners will include a list of all property owners who have received rental subsidies through the “project-based” Section 8 rental assistance program for which payment had been made under HAP Contracts administered by the 11 PBCAs between October 1, 2011 and June 30, 2012 (including both owners who have had direct contact and owners who have not had direct contact with the PBCA in the past year). The frame will include contact information for each property owner, including email address, and will indicate the date(s) of contact, if applicable, by a PBCA. The files will be obtained both from internal files maintained by the HUD regional offices and the PBCAs themselves. Note: Each property owner will be listed once on the frame, so that each owner will have the same probability of selection for the survey of property owners.

After sample selection, but prior to data collection, a pre-screening effort will be conducted identify the most appropriate respondent for the survey. The selected respondent should be the person most knowledgeable about the owner’s experience with the PBCA, thus, the most knowledgeable person could be a property manager or agent. A set of screener questions will be used to identify which person is the most appropriate respondent. Note: Managing agents may serve as proxies for the property owners if they have had the most contact with the PBCA.

4. Sample Design

The sample design will use stratified, sequential sampling to achieve the survey goals. The property owners will be stratified into eleven primary strata based on PBCA. Next, each of the eleven primary strata will be further divided into 3 substrata. The three substrata will be defined by whether the property owner has less than 75 Section 8 assistance units, 76 to 150 Section 8 assistance units or more than 150 Section 8 assistance units. This will yield a total of 33 substrata (11 PBCAs times 3 site categories), or three substrata per PBCA.

The sample size will be allocated equally among the PBCAs to achieve the desired precision of the estimates in each of the 11 PBCAs. Thus, with an overall sample size of 1,375, the sample size allocated to each PBCA would be 125 property owners.

Prior to selecting the sample within each PBCA, a sample allocation program will be run to determine the sample sizes within each of the substrata. Property owners will be allocated to each substratum in proportion to the size of that substratum (defined by the sum of all property owners in that substratum). The benefits of this procedure include the fact that all weights are exactly the same; as such, there is no ‘oversampling’ of certain strata causing variation in the weights. As a result, the variance of the overall satisfaction estimates for owners is smaller than would be otherwise. Note: If a PBCA has less than 125 property owners, all owners in the PBCA will be contacted for the study.

5. Sample Selection

Stage 1: Property Owners. After the sample size of 125 is allocated within each PBCA substratum, the property owners will be sorted within substratum by zip code and total number of Section 8 assistance units before sampling to ensure a representative sample geographically and within these groups. We will then perform systematic sampling within strata. This method involves numbering the property owners in the population from 1 to N (N= total records in population). To select a sample of n owners, we take an owner at random from the first k owners and every k^th owner thereafter until the appropriate number of property owners is achieved in the stratum. In this way, each property owner on the sampling frame will be given a known, nonzero probability of selection so that weighted inferences can be made about the entire population of property owners.

6. Sample Size Determination

A critical issue that any survey must confront is to determine the sample sizes necessary to accurately compute estimates from each PBCA and be able to detect differences across PBCAs. In this case, we plan to accurately estimate the percentage of owners satisfied with the PBCA’s performance.

The following table shows for various estimates, the size of the 95 percent confidence interval half-width to represent the sampling error. For example, assuming that the sample size is 100 and the percentage of owners satisfied is 50 percent, then using a 95 percent confidence interval; in 95 out of 100 samples like the one selected, the results should be no more than 9.8 percentage points above or below this figure. Note: We plan to obtain more than 100 completed cases to the extent it is feasible.

Table 1. 95 Percent Confidence Half-Widths

For Sample Percentages

	Estimated Percent Satisfied
Sample Size	90% or 10%	75% or 25%	50%
200	4.2%	6.0%	6.9%
150	4.8%	7.0%	8.0%
100	5.9%	8.5%	9.8%

Assuming an 80 percent response rate and a 100 percent eligibility rate among selected property owners, we plan to select approximately 1,375 property owners to participate in this survey (or 125 property owners per PBCA). Based on the anticipated response rate, this will yield 1,100 property owners in total (or 100 property owners per PBCA).

7. Weighting Procedures

Following data collection, sample weights for property owners will be prepared based on 1) the initial probability of selection, 2) adjusted to compensate for owner nonresponse, and 3) edited to remove multiple selection opportunities (if any arise). The end product will be final analysis weights suitable for use in analysis of property owners’ satisfaction scores. This weighting scheme inflates the respondents' data to represent the entire universe of property owners in the PBCAs.

B. TENANTS: PBCA TENANT satisfaction SURVEY

1. Target Population

The target population for this survey includes all Section 8 housing tenants residing in their properties in the 11 PBCA “States” that have received rental subsidies through the “project-based” Section 8 rental assistance program under the HAP Contracts that were administered by the 11 PBCAs in the current fiscal year (e.g., between October 1, 2011 and September 30, 2012).

2. Survey Eligibility

This study will include tenants who have received rental subsidies through the “project-based” Section 8 rental assistance program for which payment had been made under the HAP Contracts that were administered by the 11 PBCAs between October 1, 2011 and June 30, 2012 of the past fiscal year. Note: The tenant lists must be collected by August 15th, in order to conduct the surveys during the month of October.

3. Sampling Frame

The sampling frame for the tenant survey will be compiled in two separate steps. First, a list of individual properties will be compiled for each of the 1,375 selected property owners selected in the property owner survey. For each of the sampled property owners (identified in ‘Stage 1’ in Section A.5 above), we will obtain lists of each of the properties that they own for which they had received rental subsidies through the “project based” Section 8 rental assistance program for which payment had been made under the Housing Assistance Payments (HAP) Contracts that were administered by the 11 PBCAs between October 1, 2011 and June 30, 2012. These lists will contain property addresses and the approximate number of Section 8 tenants in each property. We plan to obtain these lists both from internal files maintained by the HUD regional offices and the PBCAs themselves. Note: This “intermediate” step is necessary for compiling a sampling frame for tenants, as it is not feasible to obtain tenant lists from all property owners in the 11 states. Each of the sampled owner’s properties will be listed on the sampling frame once; as such, owners may have multiple properties selected for the tenant sample (see sampling procedures in Section B.1.4 below).

Second, for each of the sampled properties (sampled under ‘Stage 2’ in Section B.4 below), we will obtain lists of all Section 8 tenants residing in each of the sampled properties. To collect these lists, property owners whose properties were selected in the second stage of sampling will be asked to provide a list of their Section 8 tenants. The lists of tenants must contain accurate and up-to-date contact information for each tenant.

4. Sample Design/Sample Selection

The sample design will use a 3-stage PPS sequential sampling procedure to achieve the survey goals. The second two stages of sampling are described below. Note: Stage 1, sampling of property owners, is discussed in Section A.5 above.

Stage 2: Properties. This procedure will be used to select a sample of 110 properties (a total of 10 properties in each of the 11 PBCAs) for a more detailed survey of Section 8 tenants. Within each BPCA, the properties will be selecting using Probability Proportionate to Size (PPS) sequential sampling procedures. Properties will be ordered and selected using the total number of Section 8 tenants in the property as the measure of size. PPS sampling methods will ensure that the larger properties will be included in the tenant sample. After sampling, we will run the same sampling program a second time to select a sample of replacement properties, where each replacement property is matched to a primary property. The replacement property will be substituted for the primary property if an owner refuses to provide a listing of the Section 8 tenants for sampling below.

Stage 3: Tenants. For the third stage of sample selection, within each PBCA, the tenants will be sorted by property before sampling to ensure a representative sample across all properties in the PBCA. To ensure that tenants have equal probabilities of selection across both Stage 2 and Stage 3, a sampling “rate” will be developed for each property, so that tenants in large properties will be selected at lower rates than tenants in smaller properties. This rate will be computed separately for each property as the desired sample size for tenants (in the PBCA) times the weighted proportion of tenants in the property (e.g., number of tenants times Stage 2 property weight over the sum of this weighted proportion for all sample properties). We will then randomly sample tenants within each property by selecting a tenant at random from the first k tenants and taking every k^th tenant thereafter until the appropriate number of tenants is achieved in the property. In this way, each tenant in the survey will be given an equal, nonzero probability of selection.

5. Sample Size Determination

In this case, we plan to accurately estimate the percentage of tenants satisfied with the PBCA’s performance. Assuming an 80 percent response rate from selected tenants and a 95 percent occupancy rate of section 8 tenants, we plan to select approximately 2,200 tenants to participate in this survey (or 200 tenants per PBCA; approximately 20 tenants per property.). Based on the anticipated response rate, this will yield 1,650 tenants in total (or 150 tenants per PBCA).

6. Weighting Procedures

Following the final sample weights for property owners, the sample weights for tenants will be prepared based on 1) the initial probability of selection for the owners, 2) the initial probability of selection for the properties, 3) the initial probability of selection for the tenants and 4) adjusted to compensate for tenant nonresponse. The end product will be final analysis weights suitable for use in analysis of tenant satisfaction scores. This weighting scheme inflates the respondents' data to represent the estimated universe of tenants across the PBCAs.

C. PRECISION/ RELIABILITY OF ESTIMATES

The estimates from a probability survey are commonly classified into two major categories, including 1) sampling errors and 2) nonsampling errors. This study is designed so that estimates of the magnitude of sampling error (and nonsystematic components of nonsampling error) can be computed from the sample data. The magnitude of the sampling error for an estimate, as indicated by such measures of variability as its variance or standard error (the square root of the variance), will be used to provide a basis for judging the precision of the sample estimates. Direct variance estimates that reflect the sample design will be computed for each analysis variable, and will be used in all analytic comparisons of final results.

To make survey comparisons between the PBCAs, estimates of satisfaction will be computed for 1) property owners and 2) tenants within each BPCA. Since the sample design for each component is different, different methods will be needed to compute the resultant variances for each group. Each is described below.

By obtaining a sample size of 100 completed owner interviews per PBCA, we plan to obtain PBCA property owner customer satisfaction ratings with a 95-percent, two-tailed confidence intervals of between 5.9 and 9.8 percentage points. For example, assuming that the respondent sample size is 100 for any one PBCA and the percentage of property owners who were satisfied with their communication is 50 percent, then using a 95 percent confidence interval, in 95 out of 100 samples like the one selected the results should be no more than 9.8 percentage points above or below this figure. Note: We plan to conduct a complete census of property owners in the Virgin Islands and Wyoming, which contain 12 and 59 properties respectively; as such, there will be no sampling variances for the survey of owners in these 2 PBCAs.

Because the proposed sampling design for tenants deviates from simple random sampling in the sense that it is a 3-Stage PPS sample design, the resulting variances need to be computed by taking the variance at each stage of the sampling process into consideration. For example, the loss of effectiveness by the use of cluster sampling of tenants (within properties), instead of simple random sampling of tenants across all properties in the PBCA, must be taken into consideration when calculating the variances. To estimate the approximate effect on variances, we calculated a design effect (e.g., the ratio of the actual variance, under the sampling method actually used, to the variance computed under the assumption of simple random sampling). The magnitude of the design effect depends on the size of the clusters and on the internal homogeneity of the clusters (intra-class correlation). For our calculations, we have assumed an average building size of 50 assistance units and an intra-class correlation of 0.05, yielding a design effect of approximately 3.45. If these conservative assumptions are correct, by obtaining a sample size of 150 completed tenant interviews within each PBCA, we plan to obtain PBCA tenant customer satisfaction ratings with a 95-percent, two-tailed confidence intervals of between 8.9 to 14.9 percentage points.

V. DATA ANALYSIS PLAN

The data analysis plan ensures that the study results in effective PBCA customer-satisfaction ratings. The satisfaction ratings build on the metrics discussed in Section II.B., Performance Metrics.

Various descriptive statistical techniques will be used to analyze the data, including cross tabulations and frequency distributions, and appropriate t-tests. After receiving the final survey data files, Insight will edit the data, perform weighting, and will initiate three basic analyses: 1) univariate summaries (tables analysis), 2) multi-way frequency tables (crosstabs analysis), and 3) group means differences. The type of analysis used is based primarily on the measurement level (i.e., nominal, ordinal, or scale) for each defined variable. For example, a nominal measure (e.g., non-directional categories, like race, sex, region, store) will produce only counts and percentages, whereas an ordinal measure (e.g., directional categories, like strongly agree to strongly disagree) will produce counts, percentages, and an overall mean for the variables; a scale measure (e.g., a numerical value, such as age, income, size, weight, speed, costs, etc.) will produce a mean, median, standard error, 25^th and 75^th percentile, or other customized summary statistics. All results will be presented in customized table shells.

For each survey question/item, we plan to compute 1) the percent of respondents in each satisfaction category and 2) an index/satisfaction score (e.g., 1 to 4) for the question. Key items can be ‘rolled up’ into an average satisfaction score for the overall respondent. Note: we can compare these rolled up scores to the respondents’ answer to the overall satisfaction question on the questionnaire.

The analysis will be conducted separately for the owner survey and the tenant survey for a given PBCA. We do, however, plan to explore with HUD the possibility of an overall combined owner/tenant PBCA score, in which results from the owner survey contribute some yet-to-be-determined percentage of the total score (for example, 50 percent, 66.7 percent, or 75 percent) and results from the tenant survey contribute the remaining portion of the score.

After each survey estimates are completed and the table shells have been populated, the PBCAs will be ranked in various orders depending on the satisfaction scores received on key items. For example, we will rank the PBCAs in terms of 1) the owner scores to PBCA responsiveness, 2) the tenant scores to health, 3) the tenant scores to safety and 4) the tenant scores to maintenance, etc. This will allow HUD to select various potential cutoff points to determine whether a PBCA is eligible for an incentive fee or not. For example, HUD may select the 90^th percentile or 95^th or any other point as the cutoff for determination of an incentive. This provides HUD with the option of selecting these cutoffs to meet programmatic and budgetary realities.

Although data management and simple cross-tabulations will be conducted using SAS v9.2, we will use SUDAAN v9.0.1 for standard errors and tests of significance. SUDAAN provides the correct computations for the standard errors by accounting for the design of the sample. Direct variance estimates that reflect the sample design will be computed for each analysis variable, and will be used in all analytic comparisons of final results.

VI. FINAL REPORT

In this step, we will prepare a comprehensive final report to address the study objectives. The draft report will include charts and tables of findings by PBCA to illustrate major findings. The report will contain descriptions of each of the PBCAs and their activities, the metrics used to develop the satisfaction measures and the satisfaction ratings (for tenant vs. owner/manager). We will write the report for HUD as well as a broader targeted audience of PBCAs and stakeholder groups. If sample sizes allow, we also plan to determine whether the following variables affect differences in customer-service ratings: size and age of properties [for tenants], number of properties owned [for owners], etc. Insight’s professional editor is fully cognizant of the requirements of the GPO Style Manual. Following HUD’s review of the draft report, Insight will revise the report according to the feedback received. If necessary, additional rounds of submittals and comments can then be conducted.

APPENDIX A:

SURVEY OF OWNERS

A. Introduction and Instructions

Welcome to the U.S. Department of Housing and Urban Development’s Section 8 Survey of Owners and Managers. Thank you for agreeing to participate!

The results of this survey will be used to help evaluate the agency that administers your HAP contract. Their name is [INSERT NAME OF CONTRACTOR]. You responses, in combination with those of other owners, will help HUD to determine the effectiveness of this contractor.

PRIVACY ACT: Before we begin, I want to let you know that the information gathered from this survey is protected by the Privacy Act and the results will be reported in summary form only. No responses will ever be associated with a specific individual.

This survey asks about your experiences with (FILL CONTRACTOR), the Housing Authority for your state.

IF STATE=IA THEN CONTRACTOR= Iowa Finance Authority/EPS

IF STATE=ME THEN CONTRACTOR= Maine State Housing Authority

IF STATE=MN THEN CONTRACTOR= Minnesota Housing Finance Agency

IF STATE=MT THEN CONTRACTOR= Montana Department of Housing (PBS8 Housing)

IF STATE=NH THEN CONTRACTOR= New Hampshire Housing Finance Authority

IF STATE=ND THEN CONTRACTOR= North Dakota Housing Finance Agency

IF STATE=SD THEN CONTRACTOR= South Dakota Housing Development Authority

IF STATE=VT THEN CONTRACTOR= Vermont State Housing Authority

IF STATE=WY THEN CONTRACTOR= Wyoming Housing Authority of the City of Cheyenne

IF STATE=PR THEN CONTRACTOR= Puerto Rico Housing Finance Agency

IF STATE=VI THEN CONTRACTOR= Virgin Islands: NTHDC (North Tampa Housing Development Corporation)/CGI

1. Have you been responsible for overseeing all or part of a Section 8 Housing Assistance Payments (HAP) contract in the past year that was administered by (FILL CONTRACTOR)?

Yes  GO TO “INSTRUCTIONS”

No

Is there someone else in your company who might have been contacted by (FILL CONTRACTOR) about a HAP contract in the past year?

Yes  GO TO Q1a

No  GO TO Q1b

1a. Please provide the name, phone number to call to reach that person (and an email address, if available):

Name: ________________________________________________________

Phone: ________________________________________________________

E-mail address: ________________________________________________

Thank you for your assistance! (END OF SURVEY)

1b. Thank you for your time. (END OF SURVEY)

Instructions

The survey takes about 15 minutes. If you need to stop, you may resume and complete the survey later by re-entering the password sent to you.

If you have any questions about the survey or encounter problems taking it, you may contact [NAME AND PHONE NUMBER] or by email at: [EMAIL ADDRESS].

NAVIGATION: When you are finished with the questions on a page, simply click on the Next button at the bottom to proceed to the next page. To return to the previous page, click the Prev button. (You might need to scroll down to the bottom of the page to see these buttons on your screen.)

Although not all questions require a response in order to proceed, please try to answer all of them. If you are unable to recall the exact answer, please give your best approximation.

Section A. Background Questions

The first several questions ask about background characteristics.

1. How many properties do you own or manage that currently participate in the Section 8 program (units subsidized through government rental vouchers)?

One

Two

Three

Four

Five or more

2. How many Section 8 units do you own or manage in all of your properties?

Less than 50

50 – 99

100 – 199

200 – 299

300 – 499

500 – 999

1,000 or more

3. How long have you or your company participated in the Section 8 Housing Assistance Payments (HAP) program?

Less than one year

1 – 2 years

3 – 4 years

5 – 9 years

10 years or longer

4. How familiar are you with the Section 8 contract responsibilities of the (FILL CONTRACTOR) which administers the contract for [NAME OF SAMPLED HOUSING DEVELOPMENT]?

Very familiar

Fairly familiar

Slightly familiar

Not familiar

5. How many times did you have direct contact (in-person, phone conversation) with (FILL CONTRACTOR) personnel during the last 12 months on matters concerning HAP contracts?

6 or more times

3 – 5 times

1 – 2 times

0 times

6. How many times during the last 12 months did you and (FILL CONTRACTOR) communicate by sending letters or email on matters concerning HAP contracts? Count each matter you corresponded about once.

6 or more times

3 – 5 times

1 – 2 times

0 times

Section B: Statements About (FILL CONTRACTOR)

For the next few questions, I am going to read you some general statements about (FILL CONTRACTOR). After each statement, please tell me how much you agree or disagree with the statement, by choosing strongly agree, agree, disagree, or strongly disagree.

7. HUD’s decision to use (FILL CONTRACTOR) in administering the Section 8 HAP contracts was fair and reasonable.

Strongly agree

Agree

Disagree

Strongly disagree

8. (FILL CONTRACTOR) makes a good faith effort to work with property owners to resolve billing and payment issues.

Strongly agree

Agree

Disagree

Strongly disagree

9. (FILL CONTRACTOR) provides competent oversight of Section 8 HAP contracts.

Strongly agree

Agree

Disagree

Strongly disagree

10. (FILL CONTRACTOR) has been helpful working with us on health, safety, and maintenance issues.

Strongly agree

Agree

Disagree

Strongly disagree

Section C. Satisfaction with (FILL CONTRACTOR) ‘s Customer Service

The next few questions ask you to rate your satisfaction with (FILL CONTRACTOR)’s performance and customer service.

11. Thinking about all the interactions you had with (FILL CONTRACTOR) during the past year, how satisfied have you been with their performance overall?

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

12. How satisfied are you with the level of knowledge that (FILL CONTRACTOR)’s representatives have about Housing Assistance Payment issues?

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

13. Thinking about the representatives from (FILL CONTRACTOR) with whom you have communicated, how satisfied are you with how responsive they have been in addressing and resolving your questions and inquiries?

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

14. How satisfied are you with the level of professionalism and courtesy of (FILL CONTRACTOR)’s representatives that you have communicated with?

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

15. How satisfied are you with their ease of access by phone—that is, how easy or hard it has been in the last 12 months to reach someone able to answer your questions or address your concerns?

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

Section D. (FILL CONTRACTOR)’s Actions During the Last 12 Months

Next, some questions about your experiences with (FILL CONTRACTOR) regarding specific services that you may have received over the past year.

16. In the last 12 months, how many times do you remember asking questions or making requests of (FILL CONTRACTOR)—either in person, by phone, by postal mail or email? (If not sure, please make your best estimate.)

___ times (IF 0, GO TO Q17)

16a. (ASK IF NOT 0:) Of these [INSERT ANSWER TO PREVIOUS QUESTION] inquiries or requests, how many times, if any, did you experience significant delays in getting responses? (If not sure, please make your best estimate.)

___ times

17. How often during the last 12 months has the (FILL CONTRACTOR) sent your Housing Assistance Payments on time?

Always

Almost always

Sometimes on time, sometimes not

Mostly not

18. (FILL CONTRACTOR) is supposed to send you a report—HUD-9834— following their on-site Maintenance and Occupancy Review. Did you receive this report within 30 days of their most recent Review?

Yes

No

Do not remember

19. Did you feel that the Maintenance and Occupancy Review was thoroughly conducted and fairly reported?

Yes

Yes, with minor reservations

No

Do not remember or did not read

20. Did you send a response to (FILL CONTRACTOR) after receiving the most recent Maintenance and Occupancy Review report?

Yes

No  (GO TO Q21)

Do not recall receiving a Maintenance and Occupancy Review report in the last 12 months  (GO TO Q21)

Do not remember if I sent a response to the report I received  (GO TO Q21)

20a. Did (FILL CONTRACTOR) reply to your response within 30 days?

Yes

No

Do not remember

21. How many times during the last 12 months did you request rent adjustments from (FILL CONTRACTOR)?

___ times (IF 0, GO TO Q22)

21a. (IF 1 OR MORE RENT ADJUSTMENT REQUESTS:) How many times did (FILL CONTRACTOR) process those rent adjustments within 30 days? (If not sure, please make your best estimate.)

___ times

21b. (IF 1 OR MORE RENT ADJUSTMENT REQUESTS:) How many of these times did (FILL CONTRACTOR) notify you in writing of their rent adjustment decision(s)? (If not sure, please make your best estimate.)

___ times

21c. (IF 1 OR MORE RENT ADJUSTMENT REQUESTS:) How many times within the last 12 months did you appeal a rent adjustment decision?

___ times

21d. (IF 1 OR MORE RENT ADJUSTMENT APPEALS WERE LODGED:) How many times did (FILL CONTRACTOR) respond to those appeals within 30 days? (If not sure, please make your best estimate.)

___ times

21e. (IF 1 OR MORE RENT ADJUSTMENT REQUESTS:) Did you understand the reasons given in deciding your rent adjustment requests, even if you did not agree with them?

Yes

Partly

No

22. How many times during the last 12 months did you request utility adjustments from (FILL CONTRACTOR)?

___ times (IF 0, GO TO Q23)

22a. (IF 1 OR MORE UTILITY ADJUSTMENT REQUESTS:) How many times did (FILL CONTRACTOR) process those utility adjustments within 30 days? (If not sure, please make your best estimate.)

___ times

22b. (IF 1 OR MORE UTILITY ADJUSTMENT REQUESTS:) How many times within the last 12 months did you appeal utility adjustment decisions?

___ times

22c. (IF 1 OR MORE UTILITY ADJUSTMENTS WERE LODGED:) How many times did (FILL CONTRACTOR) respond to those appeals within 30 days? (If not sure, please make your best estimate.)

___ times

22d. (IF 1 OR MORE UTILITY ADJUSTMENT REQUESTS:) Did you understand the reasons given in deciding your utility adjustment requests, even if you did not agree with them?

Yes

Partly

No

23. How many times in the last 12 months, did you submit a claim to be reimbursed for unpaid rent?

___ times (IF 0, GO TO Q24)

23a. (IF 1 OR MORE RENT REIMBURSEMENT CLAIMS:) How many times did (FILL CONTRACTOR) respond to those unpaid rent reimbursement claims for unpaid rent within 30 days?

___ times

24. How many times in the last 12 months, did you submit a claim to be reimbursed for damage caused by residents?

___ times (IF 0, GO TO Q25)

24a. (IF 1 OR MORE DAMAGE CLAIMS:) How many times did (FILL CONTRACTOR) respond to those reimbursement claims for resident damage within 30 days? (If not sure, please make your best estimate.)

___ times

25. How many times in the last 12 months, did you submit a claim to be reimbursed for vacancy loss?

___ times (IF 0, GO TO Q26)

25a. (IF 1 OR MORE VACANCY LOSS CLAIMS:) How many times did (FILL CONTRACTOR) respond to those reimbursement claims for vacancy loss within 30 days? (If not sure, please make your best estimate.)

___ times

26. (IF 1 OR MORE REIMBURSEMENT OR OTHER CLAIMS:) Did you understand the reasons given in deciding your reimbursement claims, even if you did not agree with them?

Yes

Partly

No

27. During the last 12 months, were you asked to make corrections on any vouchers submitted for payment?

Yes

No  (GO TO Q27)

27a. (IF YES:) Were you notified in writing that corrections had to be made?

Yes

Sometimes yes, sometimes no

No

28. During the last 12 months, did you apply to have your Housing Assistance Payment program contract renewed?

Yes

No  (GO TO Q28)

28a. (IF YES:) Did you receive a copy of the original, signed contract from (FILL CONTRACTOR)?

Yes

No

29. During the last 12 months, did any of your Section 8 residents contact (FILL CONTRACTOR) directly with inquiries or concerns about health, safety, or maintenance?

Yes

No  (GO TO Q30)

Not sure  (GO TO Q29)

29a. (IF YES:) Did (FILL CONTRACTOR) promptly contact you about these inquiries or concerns?

Yes, in each case

No, in none of the cases

Sometimes yes, sometimes no

Not sure

30. Can you think of any instances during the last 12 months in which (FILL CONTRACTOR) provided exceptional service or expertise in helping achieve a notable success or alleviate a serious problem at your property? (IF YES, BRIEFLY DESCRIBE)

________________________________________________________________________

________________________________________________________________________

31. Were there any instances during the last 12 months in which (FILL CONTRACTOR) failed to fulfill its responsibilities, resulting in extended a problem or preventing/delaying an achievable outcome? (IF YES, BRIEFLY DESCRIBE:)

________________________________________________________________________

________________________________________________________________________

Section E: Overall Rating

The last question asks your overall rating of the (FILL CONTRACTOR).

32. On a scale from 0 to 10, please rate the overall level of service you feel that (FILL CONTRACTOR) has provided, with 0 representing very poor service and 10 representing excellent service.

0 Very Poor Service

1

2

3

4

5 Average Service

6

7

8

9

10 Excellent Service

Thank you for taking the time to complete the survey!

APPENDIX B:

SURVEY OF TENANTS

A. Introduction

Hello, this is [INTERVIEWER NAME] calling on behalf of the U.S. Department of Housing and Urban Development to ask a few questions about your experiences at [NAME OF HOUSING DEVELOPMENT/PROPERTY]. I need to speak to an adult 18 or older who has rented this unit at [SAMPLED ADDRESS].

A. Have I reached you at that address?

Yes

No —> (VERIFY THAT THIS IS NOT THE SAMPLED ADDRESS AND TERMINATE)

B. Are you a resident here 18 or older who would contact the property management about maintenance or safety issues?

Yes —> (CONTINUE)

No —> May I please speak with an adult living at this address who would contact property management about maintenance or safety issues?

(IF ELIGIBLE RESPONDENT IS BUSY OR NOT AVAILABLE:) What would be a good time to call back?

RECORD: _______________________________

(WHEN AN ELIGIBLE RESPONDENT IS REACHED, REPEAT INTRODUCTION AS NECESSARY . . . You were randomly selected to take this survey that will take less than 10 minutes. The results will be used to improve the quality of life at [NAME OF HOUSING DEVELOPMENT].)

(IF RESPONDENT HESITATES: All information you provide in this survey will be confidential and analyzed only in combination with all other residents’ responses. Your opinions are very important. May we begin this short survey?) (IF “NO” OR FURTHER HESITATION: If this is not a good time, we can schedule the interview for another time. What would be a good time to call back?)

PRIVACY ACT: Before we begin, I want to let you know that the information gathered from this survey is protected by the Privacy Act and the results will be reported in summary form only. No responses will ever be associated with a specific individual.

B. Housing Background/Environment

Okay, let’s begin.

1. How many years have you lived at your current address? (IF NECESSARY, PROBE: Would you say less than 1 year, 1 year, 2 years, or how long?)

___ years

2. How many years in all have you lived in ANY housing, including your present unit, where the rent was subsidized—that is, part of it paid by the housing authority? (IF NECESSARY, PROBE: Would you say less than 1 year, 1 year, 2 years, or how long?)

___ years

Next, I have a few questions about living conditions at [NAME OF HOUSING DEVELOPMENT].

3. How often are you bothered by excessive noise at [NAME OF HOUSING DEVELOPMENT]? (READ RESPONSES; ORDER ROTATED)

Rarely or never

Occasionally

Fairly often

Very often

4. How often do you hear or see fighting or other serious disturbances occurring at [NAME OF HOUSING DEVELOPMENT]? (READ RESPONSES; ORDER ROTATED)

Rarely or never

Occasionally

Fairly often

Very often

5. How much of a problem is theft and other property crime at [NAME OF HOUSING DEVELOPMENT]? (READ RESPONSES; ORDER ROTATED)

A very serious problem

A fairly serious problem

A minor problem

Not a problem

6. How safe do you feel from violent crime at [NAME OF HOUSING DEVELOPMENT]? (READ RESPONSES; ORDER ROTATED)

Not at all safe

Somewhat unsafe

Mostly safe

Completely safe

(DO NOT READ:) Refused

C. Familiarity/Contact With the HAP CONTRACTOR

7. Have you heard of [CONTRACTOR]—the agency you may contact if you experience problems that the property management here does not resolve?

Yes

No -> (THANK AND TERMINATE INTERVIEW)

Not sure

8. How familiar are you with what [CONTRACTOR] does in relation to [NAME OF HOUSING DEVELOPMENT] and the residents living there? (READ RESPONSES)

Very familiar

Fairly familiar

Slightly familiar

Not familiar

9. How many times did you have direct contact (in-person or phone conversations) with [CONTRACTOR] personnel during the last 12 months?

6 or more times

3 – 5 times

1 – 2 times

0 times

10. How many times, if any, did you have mail or email contact with [CONTRACTOR] personnel during the last 12 months? (CLARIFY IF NECESSARY: Consider something one contact each time you sent a letter or email about some concern, regardless if you received a response.)

6 or more times

3 – 5 times

1 – 2 times

0 times

D. Satisfaction With [CONTRACTOR’S] Performance

The next few questions ask are about various aspects of [CONTRACTOR’S] service and performance. Please answer as best as you can. If you are totally unable to answer a question or it does not apply to you, say “don’t know” or “doesn’t apply”.

11. Thinking about the contacts you’ve had with [CONTRACTOR] during the past year, how satisfied have you been with their performance overall? (READ RESPONSES) (ORDER OF RESPONSE OPTIONS ROTATED: HALF START WITH “VERY DISSATISFIED;” HALF WITH “VERY SATISFIED”)

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

(DO NOT READ:) Don’t know

(DO NOT READ:) Does not apply

12. How satisfied are you with how respectfully they treated you? (READ RESPONSES)

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

(DO NOT READ:) Don’t know

(DO NOT READ:) Does not apply

13. Thinking about the representatives from [CONTRACTOR] with whom you have communicated, how satisfied are you with how responsive they have been in addressing and resolving your questions and inquiries? (READ RESPONSES)

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

(DO NOT READ:) Don’t know

(DO NOT READ:) Does not apply

14. How satisfied are you with the level of professionalism of [CONTRACTOR]’s representatives that you have communicated with? (READ RESPONSES)

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

(DO NOT READ:) Don’t know

(DO NOT READ:) Does not apply

15. How satisfied are you with their ease of access by phone—that is, how easy or hard it has been in the last 12 months to reach someone able to answer your questions or address your concerns? (READ RESPONSES)

Very dissatisfied

Dissatisfied

Satisfied

Very satisfied

(DO NOT READ:) Don’t know / Didn’t try to call them

(DO NOT READ:) Does not apply

16. During the last 12 months, did you contact the [CONTRACTOR] with inquiries or concerns about health, safety, or maintenance? This does NOT include any contacts you might have had only with the local property manager at [NAME OF HOUSING DEVELOPMENT].

Yes

No —> (GO TO Q18)

17. Thinking about the most recent contact, did the [CONTRACTOR] answer your questions or, if you had requests or complaints, provide help in resolving the problem?

Yes

Partly

No

Not sure

Not yet – still pending

18. Can you think of any instances during the last 12 months in which the [CONTRACTOR] provided exceptional service to you or helped you with a difficult problem? (IF YES, BRIEFLY DESCRIBE:)

________________________________________________________________________

________________________________________________________________________

19. Were there any instances during the last 12 months in which you feel the [CONTRACTOR] provided poor service to you or did not help you with a real problem? (IF YES, BRIEFLY DESCRIBE:)

________________________________________________________________________

________________________________________________________________________

E. Demographics

Finally, a few background questions about you and your family.

20. How many adults 18 and older including yourself live at your address?

One (respondent only)

Two

Three

Four or more

(DO NOT READ:) Refused

21. How many persons under 18, including any infants, live at your address?

None

One

Two

Three

Four or more

(DO NOT READ:) Refused

22. Which of the following best describes your race or ethnicity? (READ RESPONSES)

White – not Hispanic

African-American

Hispanic

Asian-American or Pacific Islander

(DO NOT READ:) Other/Mixture

(DO NOT READ:) Refused

23. One last question. Please stop me when I read the range that includes your age (READ RESPONSES):

Under 18

18 - 29

30 – 39

40 – 49

50 – 59

60 – 69

70 or older

(DO NOT READ:) Refused

Thank you for taking the time for this survey!

1 Insight has located a copy of the Section 8 Contracts Date Element Dictionary on the HUD Web site. There are fields for email addresses. HUD has confirmed its ability to provide relevant data files to Insight.