3-29-18_ROSS SC_ Part B

^{Supporting Statement} Part B for Paperwork Reduction Act Submission Resident Opportunity and Self-Sufficiency – Service Coordinator (ROSS-SC) program

OMB # 2528-XXXX

Part B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Surveys:

All organizations receiving a ROSS-SC grant in FY2014, FY2015 or FY2016 will be contacted to collect names, phone numbers and e-mail addresses of the current ROSS Service Coordinators. All current service coordinators for whom we have contact information will be contacted and asked to complete the online survey. There will be no sampling. Non-response will include service coordinators for whom we cannot get good contact information, as well as refusals. Service coordinators have not been previously surveyed. Response rate estimates are based on the target population having a natural incentive to contribute to the success of the program, encouragement from grantee organizations and HUD, and efforts to reduce non-response detailed in Section 3.

Respondent category	Universe	Number covered by collection	Number of responses expected	Response rate
Grantee contact	330	330	330	100%
Service coordinators	1,200	1,200	840	70%

Site Visits:

Of the 104 organizations receiving a ROSS-SC grant in FY2014, the Urban research team will contact 10 total to conduct site visits, three as pilot sites prior to OMB approval and the remaining seven after receiving approval. HUD and Urban will collaborate to select organizations for site visits based on grantee characteristics including population served (elderly, family, or both), grantee type (housing authority, resident council, or nonprofit agency), and geographic location (NE, SE, Midwest, South, Mountain, SW, NW). The sample will account for approximately 10 percent of the universe of ROSS-SC grantees. The sample of organizations will be stratified based on grantee type, but will not be proportional; smaller entities, such as resident councils or nonprofit agencies, will be over-sampled. Each site visit will include a focus group with residents (see below) and interviews with staff. Interviews will be conducted with the ROSS grant manager, staff responsible for data collection (if separate from the grant manager), at least one ROSS Service Coordinator, and a small sample of partnering agencies (2-3 nonprofits, government agencies, or service providers).

Focus group participants will be recruited via flyer from the pool of active program participants. Eligible individuals must be 18 or older, a current ROSS participant, and conversant in English. Only one adult per household can participate. To participate, eligible individuals must call a toll-free number to be screened for eligibility and register. We will target 10-15 participants to attend each focus group out of the seven conducted post-OMB approval, and no more than 9 participants per pilot focus groups prior to OMB approval. Each pilot focus group differed by program type (elderly/disabled versus family) and by grantee type (public housing authority versus nonprofit); questions asked were different for each program/grantee type. We will slightly oversubscribe during recruitment to account for attrition between registration and the focus group date. Participation in focus groups will be voluntary.

Respondent category	Universe	Number covered by collection	Number of responses expected
ROSS-SC Grantee Organization	104 grantees	10 grantees/sites	10 grantees/sites
Grantee participants (staff/partners)	Unknown	4-8 staff/partners	4-8 staff/partners per site
Grantee participants (residents)	Unknown	20-30 residents per site	residents per site

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Surveys:

As noted above, all service coordinators for whom we have contact information will be contacted and asked to complete the online survey (i.e., the entire population for which we have available contact information). A sampling methodology and estimation procedure are not being used. Therefore, methodology details related to sample collection do not apply. The survey will be a one-time survey of the population of ROSS Service Coordinators. There are no unusual problems requiring specialized sampling procedures. There is only one survey response requested from the service coordinators for the entire study so there is no data collection cycle and no way to reduce burden.

Site visit:

The sample of organizations will be stratified based on grantee type, but will not be proportional; smaller entities, such as resident councils or nonprofit agencies, will be over-sampled. Therefore, methodology details related to statistical methodology for sampling/stratification procedures and estimation do not apply.

Site visit selection will be based on the following formal factors:

1. Grantee Type (Housing Authority, Non-Profit, or Resident Council)

2. Tribal Status (at least one site is to be a tribal entity or primarily serving native communities)

3. Region (Northeast, Southeast, Midwest, Northwest, Southwest, or Outlying, i.e., Alaska and Hawaii)

4. Size of the PHA, measured by the number of housing units managed

Efforts will also be made to ensure rural communities are represented. This sample is intended to be purposive: selection is intended not as a precisely representative sample of FY 2014 grantees, but to reflect the diversity of ROSS programs in the country. As noted above, the sample will account for approximately 10 percent of the universe of ROSS-SC grantees. There are no unusual problems requiring specialized sampling procedures. There is only one site visit requested for the entire study.

3. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Surveys:

Construction of an up-to-date contact list for the service coordinators employed by each grantee is critical to the service coordinator survey. Every grantee will receive a letter on HUD letterhead urging them to respond to the on-line survey, and explaining how important the evaluation is to the program (a draft letter is included in Appendix E). The survey will be an easy to complete online form administered through Qualtrics. The contact survey will collect no information other than the service coordinator contact information. Reminder emails will be sent to grantees after two weeks and four weeks. Based on the light burden of the survey and the value of notification from HUD, we expect a very high response rate for this effort.

Service coordinators will receive an email explaining the evaluation effort, and that their responses will be kept confidential to the extent permitted by law (a draft of this email is included in Appendix F). They will be invited to participate by using a provided link to a secure Qualtrics website. The survey is designed to take about 30 minutes to complete, and does not have to completed in one sitting. The invitation will stress the value of their responses to the ROSS-SC program, and the relative brevity of the survey. The Qualtrics online surveys will be designed to work on a variety of electronic devices, including personal computers, tablets, and cell phones, allowing respondents to complete the survey in the manner most convenient to them.

Reminder emails will be sent to service coordinators that have not responded within two weeks and four weeks of the initial invitation to participate.

We expect on overall high response rate for this survey. However, to control for possible bias introduced through non-participation, we will stratify the universe by available administrative data on grantee size, geographic region and service coordinator area (“family” or “elderly”); if after four weeks particular strata have not yet achieved the target response rate, up to 30 service coordinators in each affected strata will be randomly sampled for direct calls to encourage participation, creating a pool of representative non-respondents. These direct calls will not be used to administer the survey, but instead will request completion through the on-line survey. Completed surveys from the late-to-complete cohort will be compared with all other completed surveys to test for significant differences.

Site Visits:

Three grantees received invitations to participate in HUD’s Evaluation of the ROSS-SC program as pilot sites prior to OMB clearance. Those grantees, individual interviewees, and focus group participants were informed regarding the guaranteed privacy to the extent permitted by law of the information they provide prior to agreeing to participate or being administered informed consent. These three site visits were completed in 2017, and included SET Ministries in Milwaukee, WI, Home Forward in Portland, OR, and the New Jersey Institute for Disabilities in Edison, NJ.

Seven subsequent grantees will receive invitations to participate in HUD’s Evaluation of the ROSS-SC program after OMB clearance is received. Those remaining grantees will be informed regarding the privacy to extent permitted by law of their information prior to their informed decision to participate; individual interviewees and focus group participants will be informed of their guaranteed privacy to the extent permitted by law of the information they provide prior to agreeing to participate or being administered informed consent. Those seven site visits are planned for 2018, following OMB clearance. For those seven site visits we have a preliminary list of seven sites plus four alternates; confirmation of the remaining seven sites is pending OMB approval and finalization of the site selection with HUD. Four additional alternate sites have been selected in the event that one or more of the seven planned sites declines to participate. In the event that an original site elects not to participate, then the research team will select an alternate site to match the non-participating site as closely as possible in terms of type of grantee, population targeted, and geographic location.

Focus groups will be limited to 10-15 participants (of 9 or fewer at pilot sites prior to OMB approval) per standard methodological practice emphasizing rich, detailed responses through small group interaction and discussion. We anticipate having to approach 20-30 residents per site to recruit a final single sample of 10-15 active participants (comprising one focus group) per site. Only individuals who actively consent to participate in the focus group by registering will be considered focus group participants. We will ask all participants to keep shared information private, and Urban itself will only report information summarized in aggregate and/or with the specific details masked/changed to protect their identity. Participants who attend the focus group for any length of time will receive a $25 token of appreciation to account for incidental costs of their participation. Since participation is voluntary, and full population characteristics are unknown, we will not be able to test for differences between those who register and those who do not. We will also not be able to test differences in perspective between those who register and attend and those who register but do not attend. However, as a qualitative research method, conducting multiple focus groups across multiple sites provides reliable data about both the breadth and depth of experiences and opinions of program participants to the degree necessary for effective program evaluation.

3. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

Surveys:

Pretesting has been conducted with four service coordinators at three sites. We interviewed the testers after they completed the survey to assess their interpretation of the questions and the reasoning behind their responses. Based on feedback received, we have changed the wording of questions to improve clarity and eliminated several questions to shorten the survey. Changes are reflected in the new draft submitted.

After surveying is complete, we will compare response rates across grantee geographic region and service coordinator area, and will apply a weighting adjustment if necessary to correct the responses to be representative of the overall population of service coordinators, which will reduce the possible nonresponse bias. Weights, if necessary, will be to ensure the weight-corrected proportion of completed surveys in each grantee stratum is equivalent to the proportion of service coordinators in that stratum compared to the universe. While we do not yet have complete information on the FY2014, FY2015, and FY2016 grantee universe, the inverse probability stratification design will follow the table below. Some strata will likely be combined to avoid very small proportions.

We will also test for bias in the completed survey based on the results of the nonresponse test. Testing for bias will use T-Tests comparing results from the late-to-complete cohort with all other surveys and comparing responses. If results for respondents in the late-to-complete cohort are significantly different from the rest of the respondents, implying bias in non-participants, additional adjustments to the estimates through weighting may be used. Late-to-complete responses would be added as a post-stratification element; this will be a viable strategy only if sufficient late responses are received to detect significant differences by stratum. Results of non-response bias testing and any methods used to compensate for it will be described in our reporting.

Site Visits:

The focus group protocol was pre-tested at 3 pilot site visits at two different types of grantees. Based on prior Urban Institute staff experience on housing-based programs and policies, we hypothesize that different types of grantees run their programs differently (e.g., public housing authority, resident council, or nonprofit organization). Although not a representative sample of ROSS-SC sites, the pilot sites were chosen to exemplify the range of experiences of ROSS-SC sites. Pre-testing was comprised of two nonprofit grantees and one public housing authority grantee. Pre-testing grantees also varied by geography and type of resident served: At least one focus group was with participants in a family service coordination program and at least one was with participants in an elderly/disabled service coordination program. The pre-tests were capped at 9 participants per focus group at each of the 3 pilot sites.

Pilot testing largely confirmed the efficacy of the site visit protocols for assessing the key substantive areas and themes addressed by the research design; the pilot site visit protocols were deemed appropriate for implementation for the seven site visits to be conducted in 2018, with small adjustments. As a result of pilot testing, adjustments were made to site visit procedures to better reflect the reality of on-the-ground organizational structures of host organizations. Changes based on site visits conducted at the 3 pilot sites are reflected in the submitted site visit protocols (Appendices C and D).

3. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractors, grantees, or other person(s) who will actually collect or analyze the information for the agency.

Survey: Christopher Hayes, Urban Institute (202-265-5650)

Site Visits: Elsa Falkenburger, Urban Institute (202-261-5512)

1