Supporting Statement for
Paperwork Reduction Act Submission
Revised per OMB Comments received June 7, 2011
Strengthening Communities Fund (SCF)
Performance Management and Evaluation Support
Contract #: HHSP23320095654WC
Order #: HHSP23337007T
Urban Institute Project #: 08350-G-007
PART B
June 23, 2011
Prepared for
Administration for Children and Families
Office of Planning, Research, and Evaluation
Prepared by
Urban Institute
Contents
Part B: Collection of Information Employing Statistical Methods 3
B.1 Respondent Universe and Sampling Methods 3
B.2 Information Collection Procedures 3
B.3 Methods to Maximize Response Rates 5
B.5 Individuals Consulted on Statistical Aspects of Design 6
Part B: Collection of Information Employing Statistical Methods
Our strategy for selecting our universe of respondents consists of the following approaches:
For the telephone interviews, we plan to interview all 84 SCF grantees (35 nonprofit grantees and 49 state, local and tribal government grantees).
For the web-based surveys, we plan to survey all 84 SCF grantees and either all or a sample of FBCO subgrantees and groups that received intensive training and technical assistance, depending on the number of subgrants awarded or groups receiving intensive training/TA. (Intensive training and technical assistance is defined as 10 or more hours of training and TA.)
Based on the grantee’s initial work plans, we estimate the pool of eligible FBCOs to be no greater than 1,400 organizations. Our budget, however, calls for a total sample of 800 to 1,000 FBCOs. If the number of subgrants/eligible groups with training/TA exceeds this range, we will draw a sample, but we expect our sample will include over 70 percent of all eligible organizations.
Part of the sample will consist of FBCO subgrantees selected proportionate to the size of the subawards. This will ensure that the evaluation collects information from the subgrantees who receive the largest amount of funds. Because we will know all groups that receive subawards, if necessary, our final sample will be weighted by key organizational characteristics to ensure that we end up with a representative sample.
The other part of the sample will consist of FBCOs receiving intensive training/TA. These groups will be drawn as a simple random sample. At this time, we do not know how much information will be available on the characteristics of FBCOs receiving intensive training/TA nor if the sampling frame (i.e., the universal list) will be complete. Given these uncertainties, there is no compelling characteristic upon which to stratify the sample of FBCOs receiving intensive training/TA. If the sampling frame is incomplete, we will have a convenience sample, not a representative sample. In this case, we will analyze this group of FBCOs separately from those with subgrants. We believe that much can be learned from this group of FBCOs that can inform the overall evaluation.
Lists of grantees, subgrantees, and FBCOs with intensive training/TA will be obtained from the PPR forms or other lists provided by the Department of Health and Human Services.
Document Review
A thorough review of existing documents will serve as a starting point for identifying information already available and avoid duplication of data collection efforts. We intend to draw data from the following documents as part of our evaluation:
Grantee documents: applications, work plans, outcome measurement plans, SCF Performance Progress Reports (PPR), evaluation plans or summaries produced by grantees.
Externally produced documents and data: published data on topics such as the local economy and labor market from the Bureau of Labor Statistics or other sources, and data on nonprofit and faith-based organizations available from the Urban Institute’s National Center for Charitable Statistics (i.e., IRS Forms 990).
We expect to draw on grantee documents for information on program design, operating strategies, the political and policy context, descriptive information about grantees and programs, geographic target areas, approach to delivering capacity building services, and early outcomes, such as hours of training provided or number of nonprofit organizations that completed training. The external documents and data sources will provide information on unemployment rates, benefits coverage, organizational finances, etc.
We will develop summary spreadsheets or templates for systematically reviewing the various document sources. These data will contribute to a descriptive profile of grantees and inform the interviews with grantees.
Web-based Surveys and Telephone Interviews
The study will use both quantitative and qualitative methods. The data will be collected through surveys and telephone interviews.
Data collection will begin after we receive approval from OMB. We will administer the surveys through a web-based application. Web-based applications are generally less of a burden on respondents because their automated skip patterns limit the number of questions respondents need to read or answer. Another key advantage of web-based surveys is that they facilitate regular tracking of responses. The Urban Institute uses CheckBox, a software application that is easy to format for web-based application and easy for respondents to use. For organizations without email addresses, we will send questionnaires via the U.S. postal service. The email invitation to the web survey and the mailing packet will include a cover letter that explains the purpose of the survey. We will assure all respondents that their answers will be protected within the limits of the law, and information from the survey will be reported as aggregated statistics to protect the organization’s identity.
The surveys primarily use closed-ended questions. The survey for SCF grantees is estimated to take 15 minutes to complete and the survey for FBCOs, 30 minutes. Longer surveys are likely to adversely affect response rates, especially for FBCOs.
Survey procedures are designed to ensure high response rates among both types of respondents – namely SCF grantees and FBCOs. Initially, we will contact respondents for each of the surveys by mail or email. SCF grantees (84) will receive a letter (or email) from ACF/OCS explaining the purpose of the survey and introducing Urban Institute as the survey contractor. A subsequent letter (or email) from the Urban Institute will include the web link, login name, and password so respondents can complete the survey on the web. For FBCOs (800), the Urban Institute will send a letter that explains the study and provides instructions for accessing the web-based survey. This process will enable UI to more quickly and efficiently monitor returns and reduce burden on ACF/OCS staff. Upon request, we will send a Microsoft Word version or paper copy of the survey for those not able to complete the survey on-line. The contents of the surveys will be the same in either mode. Samples of the letters are in Appendix F of this document.
Electronic reminders to complete the survey will be generated only to those who have not responded. Reminder postcards will be sent to those without email addresses. After several attempts have been made to remind SCF grantees/FBCOs to complete the survey, team members will make reminder telephone calls to those who have not responded. To be cost effective, we will make reminder phone calls to a random stratified sample of nonrespondents and continue to send email messages and postcards to the remainder of nonrespondents. The stratification procedures will be determined after reviewing early returns for any potential response bias. The goal will be to obtain a representative group of respondents. In addition, we will offer to mail or fax a hard copy of the questionnaire or offer to administer the survey over the telephone at the respondent’s convenience.
We will use telephone interviews to obtain in-depth qualitative data that provides context and nuance for the quantitative data collected in the web-based surveys. Interviews will be conducted with at least one key informant in each of the 84 grantee organizations. The key informant is likely to be the executive director, program manager of the SCF-ARRA grant, or other individual knowledgeable about the SCF-related activities. Our HHS program officer will notify all grantees of the telephone interviews. Once this notification has been issued, we will contact grantees to schedule a convenient time to conduct the interview. To facilitate the interviews, we will send an advance copy of the questions that may require the respondent to look up information or consult with (or involve) other staff. This approach enables the respondent to reflect on the topic rather than provide a superficial or partial response because they had not anticipated a question.
The Urban Institute is aware of the increasing challenges associated with achieving high response rates. We cannot guarantee a specific response rate, but we will use standard and proven survey techniques to ensure the highest response rate possible. We anticipate that these steps are likely to produce response rates of 75 to 80 percent (or more) among SCF grantees and perhaps 30-40 percent among FBCOs. ACF/OCS’s expressions of strong support for the project are likely to have a favorable impact on grantee respondents. Response rates for similar surveys of FBCOs are generally 25-30 percent, but experience shows that with the support of the grantees and ACF/OCS, the response rate should increase. We understand that grantees are responsible for ensuring that FBCOs that receive financial support comply with the national evaluation efforts and we will work through the COTR to remind grantees of this responsibility. However, we also recognize that the grantees may have limited capacity to bring about this cooperation. Our previous experience in conducting surveys of FBCOs has been that faith-based organizations are less likely than community-based nonprofits to respond to surveys and therefore tend to lower the overall response rate. To achieve a high response rate, repeated follow-up reminders and phone calls will be needed to encourage participation. The survey returns will be continually monitored and the list of FBCO respondents updated to avoid sending remainders to those that have already replied. By drawing a random sample of FBCOs, we will provide more tailored and concentrated follow-up effort, which generally leads to higher response rates.
Pre-Test
We worked closely with our COTR to identify eight SCF grantees and eight FBCOs to participate in a pretest of the instruments. Pretest respondents generally found the surveys to be simple to use and did not have difficulty with the wording of questions.
Survey Administration Procedures
Survey procedures are designed to ensure high response rates among both types of respondents – namely, SCF grantees, and FBCOs. Initially, we will contact respondents for each of the surveys by mail. Respondents will receive a letter (or email) from ACF/OCS explaining the purpose of the survey and introducing Urban Institute as the survey contractor. A subsequent letter (or email) from the Urban Institute will include the web link, login name, and password so respondents can complete the survey on the web. Upon request, we will send a Microsoft Word version or paper copy of the survey for those not able to complete the survey on-line. The contents of the surveys will be the same in either mode.
Electronic reminders to complete the survey will be generated only to those who have not responded. Reminder postcards will be sent to those without email addresses. After several attempts have been made to remind SCF grantees/FBCOs to complete the survey, team members will make reminder telephone calls to those who have not responded. To be cost effective, we will make reminder phone calls to a random stratified sample of nonrespondents and continue to send email messages and postcards to the remainder of nonrespondents. The stratification procedures will be determined after reviewing early returns for any potential response bias. The goal will be to obtain a representative group of respondents. In addition, we will offer to mail or fax a hard copy of the questionnaire, or offer to administer the survey over the telephone at the respondent’s convenience.
The agency responsible for funding the study, determining its overall design and approach, and receiving and approving contract deliverables is:
ACF Reports Clearance Officer
Administration for Children and Families
Office of Planning, Research and Evaluation
U.S. Department of Health and Human Services
370 L’Enfant Promenade SW
Washington, DC 20447
Email address: [email protected].
The Urban Institute is the contractor for this study. It is responsible for implementing the overall design of the study and development of the data collection instruments. It will field the survey and conduct telephone interviews using its own staff, and will have responsibility for all data analyses obtained through the web-based survey and telephone interviews.
The Urban Institute
2100 M Street, NW
Washington, DC 20037
(202) 833-7200
Persons Responsible: Carol De Vita and Elaine Morley, Co-Principal Investigators
Timothy Triplett, Survey Associate and Statistical Expert
Direct Contact Information:
De Vita (phone: 202-261-5232; email: [email protected])
Morley (phone: 202-261-5629; email: [email protected])
Triplett (phone: 202-261-5579; email: [email protected])
| File Type | application/msword |
| File Title | Grantee Early Site Visit Discussion Guide |
| Author | admined |
| Last Modified By | Department of Health and Human Services |
| File Modified | 2011-06-27 |
| File Created | 2011-06-27 |