OMB Part B

Corporation for National and Community Service

Volunteers in Service to America (AmeriCorps*VISTA)

VISTA Assessment of project sustainability

Office of Management and Budget

Clearance Package Supporting Statement

for Paperwork Reduction Act Submissions

Part B

Collections of Information Employing Statistical Methods

Table of Contents

Page

INTRODUCTION..……………………………………………………………………………1

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods …………………………………9

B.2 Procedures for the Collection of Information …………………………………11

B.3 Methods to Maximize Response Rates ………………………………………..14

B.4 Tests of Procedures or Methods ……………………………………………….15

B.5 Consultations on Statistical Aspects of the Design ……………………………16

Corporation for National and Community Service

AMERICORPS*vista

VISTA Assessment of project sustainability

B. Collection of Information Employing Statistical Methods

B1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The 2008 Assessment of VISTA Project Sustainability design provides for selecting a diverse and nationally representative sample of VISTA projects for the evaluation. The unit of sampling (as well as the primary unit of analysis) is defined to be the specific VISTA project within a VISTA-sponsoring organization. Organizations with multiple projects will have multiple chances of selection. The following two major groups of projects are included in the sample design:

(2) VISTA Projects that have been closed for at least 2 years.

The sampling frame consists of 984 closed VISTA projects. The stratification variables we plan to use are: active/closed status of a project, region, and average number of VISTAs per year. Table B1-1 represent the number of active VISTA projects by region and Table B1-2 presents the closed VISTA projects by region.

Table B1-1. Active Projects by Region

Region	Number of Projects	Percent	Cumulative Frequency	Cumulative Percent
Atlantic Region	115	29.04	115	29.04
North Central Region	88	22.22	203	51.26
Pacific Region	78	19.70	281	70.96
Southern Region	66	16.67	347	87.63
South Western Region	49	12.37	396	100.00

Table B1-2. Closed Projects by Region

Region	Number of Projects	Percent		Cumulative Frequency		Cumulative Percent
Atlantic Region	209		21.24		209		21.24
North Central Region	147		14.94		356		36.18
Pacific Region	178		18.09		534		54.27
Southern Region	195		19.82		729		74.09
South Western Region	255		25.91		984		100.00

Operationally, a project’s start date will be defined as the month and year of the project’s Master Agreement (MA). A project’s end date will be defined as the last month and year in which a VISTA was active in the project. Specifically, active projects are eligible for the study if and only if their start date is more than 2 years (2*365 days) prior to a specified reference date. Closed projects are eligible for the study if their end date is at least 2 years (2*365 days) and at most 5 years (5*365 days) prior to a specified reference date. The 5-year restriction on closed projects is imposed for practical reasons of data collection. The reference date will be determined by the Corporation for National and Community Service (CNCS) VISTA program as part of their process of preparing a list of eligible projects as described under Sampling Frame below.

Table B1-3. Average Number of VISTAs in Active Projects

Average Number of VISTAs	Number of Projects	Percent	Cumulative Frequency	Cumulative Percent
0 – 12 VISTAs	308	77.78	308	77.78
More than 12 VISTAs	88	22.22	396	100.00

Table B1-4. Average Number of VISTAs in Closed Projects

Average Number of VISTAs	Number of Projects	Percent	Cumulative Frequency	Cumulative Percent
0 – 6 VISTAs	830	84.35	830	84.35
More than 6 VISTAs	154	15.65	984	100.00

The distribution of the average number of VISTAs per year is different for active and closed projects as indicated in Tables B3 and B4.  The active projects tend to have higher average VISTAs numbers as the number of VISTAs tapers out over the final few years of a project, i.e., by the time the project has closed the average number of VISTAs per year may decrease. Under the assumption that the average number of VISTAs per year may be correlated with some of the project characteristics of interest (e.g., success) different groupings were used for active and closed projects.

Expected Response Rates

We will sample 250 closed projects. The size of the samples to be initially selected will be adjusted to compensate for losses due to nonresponse, inability to contact project personnel, and missing or problematic administrative data. We plan to over-sample the closed projects by 15–20 percent (i.e., an initial sample of approximately 290 projects will be selected). The expected response rate for the closed projects is 250/290 = 86%.

Previous Data Collections and Response Rates

This is the first data collection of this type and therefore there are no previous response rates to report.

B2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Methodology for Stratification and Sample Selection

The samples will be allocated to the substrata in a manner designed to provide acceptable precision for a range of statistics to be derived from the study. Since costs of data collection are not expected to vary significantly, such factors need not be considered. In general, a stratified sample design may use disproportionate allocation so that more heterogeneous strata are sampled at higher rates. However, since little is known about variability in project effectiveness for different substrata, the application of Neyman (optimal) allocation is problematic for this study. If, for example, there is general agreement that small projects are more variable in terms of their success than are large projects, small projects will be sampled at higher rates. On the other hand, if it is important that survey estimates for large projects have adequate precision, the design must ensure that an adequate number of large projects are included in the sample. In the absence of justified reasons (such as the aforementioned) for over- or under-sampling of certain substrata, proportional allocation will be used.

A stratified sample design will be used to ensure that the sample of projects is nationally representative with respect to project status, size, and location. Project status – active or closed – will be used as the primary stratification variable. Based on the criteria specified in Section 6.2 (i.e., more than 2 full years of activity for current projects, and 2–5 years since closure for closed projects), the sampling frame is expected to contain about 450 active projects and 982 closed projects.

Next, substrata based on region (i.e., the five CNCS clusters) and size of project will be defined within the major strata. Project size will be measured by the average number of VISTAs active per year. The exact values of project size that define the substrata boundaries (e.g., 1, 2–4, 5–9, 10 or more) will be determined upon review of the distribution of project size in the sampling frame. While it is also desirable to select a sample that is representative in terms of project length or duration, the number of projects on the sampling frame is unlikely to support the use of additional explicit substrata. Thus, implicit stratification will be employed by taking a systematic sample of projects ordered according to their duration within substrata. Duration will be defined as the length of time between the project’s start and end dates (for closed projects) or between the start and reference dates (for current projects).

Estimation Procedure

We will collect qualitative and quantitative information. The qualitative information will be used to describe the context and content of the projects, as well as the steps along the way to project completion. Quantitative information will be used to develop predictive models of success in sustaining project activities. Multivariate methods will be applied to the sample of completed projects to identify variables that are potentially predictive of program success, with program success defined on the basis of selected outcome measures as a categorical variable indicating varying levels of success.

Degree of accuracy needed for the purpose described in the justification

Table B 2-1 summarizes the degree of accuracy to be expected under the proposed design for estimates of percentages ranging from 30 percent to 50 percent (or, alternatively, 70% to 50%). The entries correspond to 95-percent confidence bounds and are given for subgroup sample sizes ranging from 100 to 250, separately for the two major analytic groups. An estimated design effect of 1.2 was used in the calculations for both groups to reflect any increased variation in sampling weights that may arise due to disproportionate allocation of the sample to substrata. In addition to the design effect, appropriate finite population correction (fpc) factors, which dampen sampling errors, are reflected in the confidence bounds. Since the population of current projects is smaller than the population of closed projects, confidence bounds for a sample of active projects will generally be smaller than the corresponding bounds for a similar-size sample of closed projects. For the sample sizes given in Table 6-1, the margins of error (95% confidence bounds) around an estimated percentage can be expected to range from ± 4 percent to ± 7 percent for the sample of current projects and from ± 5 percent to ± 9 percent for the sample of closed projects. For statistics derived by combining the samples of active and closed projects, the margins of error based on the combined sample would be approximately 0.92 times the entries shown in the table under current projects.

Table B 2-1. Approximate 95-percent confidence bounds for estimated percentages and selected subgroup sample sizes

Active projects				Closed projects
Sample size	Estimated percentage			Sample size	Estimated percentage
	30/70%	40/60%	50%		30/70%	40/60%	50%
250	± 4.1	± 4.4	± 4.5	250	± 5.4	± 5.8	± 5.9
200	± 4.6	± 5.0	± 5.1	200	± 6.0	± 6.4	± 6.6
150	± 5.4	± 5.7	± 5.8	150	± 7.0	± 7.4	± 7.6
100	± 6.6	± 7.0	± 7.2	100	± 8.5	± 9.1	± 9.3

Unusual problems requiring specialized sampling procedures

We do not foresee any unusual problems that would require specialized sampling procedures.

Use of periodic data collection cycles to reduce burden

This will be a one-time data collection and currently there are no plans to conduct periodic data collections.

B3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

The research staff working on this project will be carefully trained in data abstraction and conducting the telephone interview. We expect that the CSO staff will cooperate with the study. The following procedures will be used to maximize the completion rates for the telephone surveys:

• Provide information about the study to the sampled projects before the telephone interview.

• Use a core of research staff who will conduct the telephone interviews that have experience working on telephone surveys and have proven their ability to obtain cooperation from a high proportion of sample members.

• Require all interviewers to successfully complete training specific to the study, including understanding and getting familiar with the goals of the project based on the background information developed during document abstraction.

• Make every reasonable effort to obtain an interview at the initial contact, but allow respondents flexibility in scheduling appointments to be interviewed.

• Closely supervise interviews during data collection.

• Conduct silent mentoring of interviews to identify and promptly correct behaviors that could be inviting refusals or otherwise contribute to low cooperation rates.

• Leave a message on answering machines in order to let the respondent know the call was not a marketing effort but a research study.

• Provide a toll-free number or respondents to call to verify the study’s legitimacy or to ask questions about the study.

• Implement refusal conversion efforts for first-time refusals and use interviewers who are skilled at refusal conversion and will not unduly pressure the respondent.

This data collection effort will yield reliable data that can be generalized to the universe studied.

B4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

With the assistance of the CNCS, four respondents were identified from among sponsoring organizations with completed VISTA projects continuing for at least 2 years past the VISTA’s departure and located nearby in the Washington metropolitan area. The pilot process began with obtaining information from the sampling frame and document abstraction, followed by interviews with staff from sponsoring organizations. We examined the usefulness of the abstraction forms using documents provided by CNCS. We interviewed these project directors over the telephone and, following the interview, asked a series of questions on their opinions of specific questions and of the instrument as a whole. The site visit protocol was tested by senior staff members who visited two of the organizations. If feasible, we will choose one with a successful project, and one with an unsuccessful project. (Will edit previous sentence following pre-test.)These site visits enabled us to sharpen the focus of the questioning and refine the items.

Pretest Procedures

All instruments were pretested following approval of final drafts by CNCS. A purposive sample of four nearby closed projects was with the concurrence of Headquarters. Documents were requested from eGrants and, when necessary, the State Offices, and were be abstracted using the final abstraction form. We reviewed applications and progress reports before in preparing a draft abstraction form and found that the documents would probably supply less information than originally assumed. The pilot test verified/modified that finding. (Well edit previous sentence following pre-test.)

The entire pilot test was completed before the submission of the Federal Register Notice.

B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Name	Affiliation	Telephone Number
Cynthia Thomas	Westat	301-251-4364
Adam Chu	Westat	301-251-4326
Andrea Piesse	Westat	240-453-2907
Jianzhu Li	Westat	240-314-2566
Carol Rogers	VISTA, Corporation for National and Community Service	202-606-6815
Craig Kinnear	VISTA, Corporation for National and Community Service	202-606-6708
Esayas Mekonnen	VISTA, Corporation for National and Community Service	202-606-7557