12-20-18_HUD TA Supporting Statement B

Supporting Statement for Paperwork Reduction Act Submissions

HUD Technical Assistance Assessment

(OMB # 2528-XXXX)

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the

Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Urban’s respondent universe will include 13 Technical Assistance (TA) providers total (8 in the pilot and 5 in the actual data collection) and 14 TA customers (4 in the pilot and 10 in the actual data collection), the representativeness and selection methods for which are provided in Table 1 and discussed below.

Table 1 - Summary of Interviews by Respondent Type

Organization Type	Completed (Pilot)	Remaining (Planned)	Total Sampled	Total Populationa
TA Providers	8	5	13	25
TA Customers	4	10	14	At least 286

^aTA providers represent an unduplicated count of all TA provider organizations awarded funds in FY14, FY15, and FY16. There is no known total number of customers served since not all TA is reported in the TA Portal database that collects TA workplan data, and customer information is missing for some TA types.

The respondents for this collection’s group interviews will come from the following sources:

• TA Providers – Of the 25 organizations providing TA for U.S. Department of Housing and Urban Development (HUD) grantees, the Urban research team will contact 13 organizations total (8 as pilot runs prior to Office of Management and Budget (OMB) approval and 5 in the actual data collection). These will represent 52% of the full universe of TA providers.
  
  

• TA Customers – There is no total number of customers served by TA providers since not all TA is reported in the TA Portal and customer information is missing for some TA types. However, we do know the spread of types of TA customers available through the Portal (see Table 2 below). Out of the known 286 TA customers served during the project period, Urban’s research team will contact 14 organizations total (4 for the pilot sessions prior to OMB approval and 10 in the actual data collection) to conduct interviews. Urban’s selection of TA customers represents 4.9% of all known TA customers.

Table 2 – Count of Listed TA Customers

TA Type	TA Portal Work plans with a Listed Customer Organization	Percent of all TA Portal Work plans
Direct	260	90.6%
Needs Assessment	8	2.8%
On-Call	18	6.3%
Training Delivery	1	0.4%
TOTAL	286	100%

We expect to reach our total sample number of TA provider organizations and TA customers targeted for interviews. We will replace an organization that declines to participate with another organization with similar characteristics from the total population.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Each organization will only be interviewed once with a group interview, if necessary. These group interviews will be conducted with multiple people serving different, but complementary functions delivering or receiving TA in the organization.

The sample of organizations will be stratified based on TA provider and customer type, and to the greatest extent possible will be proportional. Selection will be based on the following criteria for each group:

• TA Providers: We will draw the sample of provider organizations from those that received Community Compass TA awards between 2014-2016, inclusive. We will select organizations with a range of organizational size, funding levels, urban/rural concentration, TA type, and subject matter of TA delivered. This collection will target organizations that received three or fewer years of funding. We will ensure we have interviewed at least one provider for each of the programs covered under Community Compass.

• TA Customers: TA customer interviewees will be selected starting with the list of Community Compass work plans available from the TA Portal where we have at least one recipient organization listed. The current count of work plans with listed customer organizations is shown in Table 2 above. We will sample customers from those who received one or more of the three targeted types of TA: direct TA, needs assessment, or on-call TA. We will not select based on receipt of other TA types due to lack of data, however, we will include questions in the interview guides to review with each customer if they have received another TA type and their reflections on their experiences.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Our sample of TA providers comes from a known and contactable pool. All but one contacted for the pilot agreed to participate. We expect continued high response rates. Non-responses will be handled by randomly selecting other providers with similar characteristics for study recruitment.

For TA customers, we face a key limitation. While there are approximately 1,500 work plans in the HUD TA Portal, only about half list a recipient organization, and most of these involve only three TA types. The work plans do not provide contact information for those organizations and while we will pair this information through other means, this limitation could affect our ability to contact customers and ensure the customers cover different types of TA. To combat this limitation and ensure a broad and more representative sample, we will oversample customers by each type, supplementing our use of the TA Portal with snowball sampling through interviews with HUD TA program staff, field office staff, and providers. Non-responding customers will be contacted up to two times within a four-week period and then will be replaced by moving to the next customer within the same category of TA type. Since participation is voluntary and the full universe’s characteristics are unknown, we will be unable to test for differences between those who participate in the interviews and those who do not. However, as a qualitative research method, conducting multiple group interviews across TA customers provides reliable data about the breadth and depth of experiences and opinions of TA program participants to the degree necessary for effective TA program evaluation.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

As noted in Question 2, the interview protocols were pre-tested in a pilot with 8 TA provider organizations and 4 TA customer organizations. The pilot organizations were chosen to exemplify the range of experiences and TA types in existence.

Pilot tests showed that TA providers are responsive to requests for participation at high rates, while TA customers are less responsive to requests for participation. Since TA providers are contracted to provide TA, and customers generally volunteer to participate in TA, this is as expected. We have added additional methods of identifying possible customers for participation, including asking HUD program staff, field office staff, and providers for customers they may recommend for participation based on their knowledge of the types of TA that customer has received. We will ask for multiple recommendations and recommenders will not be aware of which customers are ultimately invited to participate or agree to do so.

Pilot testing largely confirmed the efficacy of the interview protocols for assessing the key substantive areas and themes addressed by the research design. They did indicate a need to use group interviews. As a result of pilot testing, adjustments were made to interview structures and procedures to better reflect the reality of TA provision when the responsibilities are spread among staff members and subcontractors that perform different roles within the TA process. Group interviews ensure that all questions can be answered in a single setting by the group member responsible for each specific part of the process. Additional changes to improve the content, flow, and validity of interview questions and responses are reflected in the submitted interview protocols in the appendices.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Joseph Schilling, Urban Institute, 202-261-5982