PART A
JUSTIFICATION
Part A of this narrative provides the justification for the proposed data collection. Part B provides detail on the statistical methods for the proposed data collection.
The Corporation for National and Community Service (the Corporation) is seeking to make its performance measurement and management systems more accountable and results oriented. To do so, the Corporation will continue to measure outcomes for its programs—including the AmeriCorps Volunteers in Service to America (VISTA) program, the AmeriCorps National Civilian Community Corps (NCCC) program, and the AmeriCorps State and National programs.
Certain types of outcomes can only be assessed by surveying appropriate individuals, nonprofit organizations, and institutions. Therefore, to do the latter, instruments have been developed and samples will be drawn for use in systematically surveying: (a) AmeriCorps members1; (b) former members; and (c) programs or project staff representing AmeriCorps sub-grantees and other organizations that deliver services.
The Corporation intends to conduct such surveys annually and use the information collected, along with other outcome data, to prepare report materials (such as required by the Government Performance Results Act (GPRA) of 1993), to assess program performance, to link program performance with the budget process, and to improve the Corporation’s programs and management. In addition, it is anticipated that the surveys will serve as modes that the Corporation’s grantees can adopt to assess the performance of their own programs and projects.
A2 How and By Whom the Data Will Be Used?
This section presents an overview of the project, discusses the purpose of the data collection activities, indicates who will be the primary users of the information to be collected, and provides a justification for each of the survey items covered by this submission.
A2.1 Project Overview
Created in 1993, the Corporation is a public-private partnership that oversees three national service initiatives:
AmeriCorps, which is designed to engage Americans of all ages in full- or part-time service and includes AmeriCorps*VISTA, AmeriCorps*NCCC, and AmeriCorps*State and National;
Learn and Serve America, which integrates service and learning from kindergarten through post-secondary education; and
The National Senior Service Corps, which includes the Foster Grandparent Program, the Senior Companion Program, and the Retired and Senior Volunteer Program.
The Corporation was established to strengthen the ability of existing private and public agencies to expand service through partnerships, grants, training, and assistance, and to give Americans of diverse backgrounds an opportunity to serve. It has several basic goals:
To provide opportunities for all citizens (including members2 and community volunteers) to serve others in communities throughout the United States;
To afford such citizens (including members and community volunteers) who provide help to others with meaningful, valuable, and enriching experiences (such as through leadership training, technical assistance, and citizenship training development), and to support a continued ethic of volunteer service;
To improve the lives of citizens and their communities through the services of AmeriCorps members, such as by addressing critical needs of citizens and their communities in such areas as education, health, safety, environmental protection, and homeland security; and
To strengthen the capacity of private nonprofit and faith-based organizations as a means of promoting one or more of the other missions.
Because of the Corporation’s distinct goals, its diverse program structure, and the wide range of service activities in which its members are involved, measuring the performance of the Corporation’s programs requires collecting a variety of different kinds of information. While some performance information is maintained in Corporation administrative records, other necessary information must be gathered through contact with relevant members of the public. For the Corporation’s AmeriCorps programs, this can be accomplished, with least burden, through brief sample surveys of:
Current and former AmeriCorps members serving with VISTA, NCCC, and State and National, and
Program/project staff representing service delivery organizations in which (a) VISTA or State and National are currently serving .
A2.2 Purpose of the Data Collection
The first and primary purpose of the data collection is to provide reliable and valid outcome data with which the Corporation can respond to the GPRA outcome measurement requirements as well as OMB’s requirements, such as those pertaining to the Program Assessment Rating Tool (PART). Major portions of the data to be collected will provide a basis for the Corporation to report the outcomes of its AmeriCorps VISTA, NCCC, and State and National programs
A second purpose, also of major importance, is to help the Corporation, its AmeriCorps programs, and its subgrantees and host sites to improve services. A considerable amount of the data to be collected will provide an important basis for assessing those aspects of service delivery that are doing well and those that can be improved and, therefore, provide opportunities for service and program enhancement.
A third purpose of the data collection is to use the data (beginning FY2004) to assess the extent to which progress has been made. Similarly, this outcome data will provide an important basis for setting future targets, required by the OMB budgeting process.
Based on the authority provided by the National and Community Service Act Of 1990 (the Act), as amended through December 17, 1999 (P.L. 106-170), the Corporation is charged with utilizing its service programs, including AmeriCorps, to:
Address the unmet human, educational, environmental, and public safety needs of the United States
Renew the ethic of civic responsibility and the spirit of community throughout the United States; and
Encourage citizens of the United States . . . to engage in full‑time or part‑time national service.
Accordingly, the Corporation and its programs provide opportunities for service participants and volunteers, working side-by-side, to serve one-on-one as tutors and mentors to young people, provide intensive educational services to children and adults, respond to natural disasters, help to build the capacity of local nonprofit organizations working to solve community issues, and conduct other service-related activities.
The Act also requires the Corporation to identify measurable performance goals for its programs and evaluation methods, including conducting surveys of program participants, to determine the impact of the program. Likewise, the Corporation was directed to report to Congress in the fall of 2004 on its performance measurement enhancement efforts. Finally, the President's Executive Order on National and Community Service (released in February 2004) further affirms that the Corporation is engaging in performance measurement efforts and mandates that the Corporation’s programs develop performance measures to ensure accountability.
The need to report on the impact of the Corporation’s programs has become even more critical since the signing of 2004 Omnibus Appropriations Bill, which authorized the Corporation to support a record 75,000 AmeriCorps members. The Corporation must be able to demonstrate clearly to Congress, the Administration, and the public that its programs are accountable and effectively meeting community needs. In addition, an outcome-based performance measurement system will allow the Corporation to report verifiable program outcomes and outputs to the OMB and Congress in the Performance and Accountability Report (PAR), as required under GPRA, and provide the Corporation with information that will help to improve Corporation services to organizations, members, and the communities that they serve.
A2.3 Who Will Use the Information?
A primary category of users of the outcome information to be obtained and reported are external to the Corporation, including OMB, the President, and the Congress—and, ultimately, the public (when the data and analysis are reported on the Corporation’s web site). These include readers of the Corporation’s GPRA report and related accountability reports, and the Corporation’s budget justification materials.
A second major category of users consists of AmeriCorps program (and other Corporation) officials. Corporation officials are not only vitally interested that the collected data be reported externally but are also interested in utilizing such information to help identify where service improvements are necessary. A significant portion of the outcome data being collected will provide an insight to issues and problems (such as the adequacy of training provided to AmeriCorps members and to the service organizations that supervise AmeriCorps members). This includes not only Corporation headquarters staff but also AmeriCorps field officials, such as those in Corporation State Offices and NCCC campuses.
A third category of users consists of organizations that receive AmeriCorps resources. These include the State Commissions that oversee the AmeriCorps State program and the many nonprofit and faith-based organizations, educational institutions, and public agencies that involve AmeriCorps members in the services they provide to the general public. While most data will be aggregated such that it will not provide direct information on individual programs, such data (like feedback from end beneficiaries and AmeriCorps members) may indicate avenues for examining and helping to improve policies, practices and procedures, even at the state and local level.
The survey modules are:
Current AmeriCorps members (to be surveyed separately for the AmeriCorps*State and National Direct programs, the AmeriCorps*VISTA program, and the AmeriCorps*NCCC program;
Former AmeriCorps members (surveyed separately for the AmeriCorps*State and National Direct programs, the AmeriCorps*VISTA program, and the AmeriCorps*NCCC program. The former member surveys also include surveys of former members who dropped out of service ; and
Representatives/program or project staff of the service-providing organizations that supervise AmeriCorps members or receive services from members.
A3 Use of Improved Technologies
Surveys of current AmeriCorps members and potentially dropouts will be conducted via the My AmeriCorps Portal, web tool used to exit members from service, and track the receipt and use of their education award. The other surveys will be conducted utilizing the latest methodological and technical developments in telephone interviewing.
A4 Efforts to Identify Duplication
Corporation staff reviewed current program data collection efforts to gage duplication. Although the Corporation collects a limited amount of information from members at the point of their exit from service, the proposed data collection is the first to be conducted by the Corporation (or anyone else) that is designed to provide information for measuring program outcomes for members, former members and organizations consistently across AmeriCorps programs
A5 Involvement of Small Entities
Some respondents to the outcome indicators surveys will be AmeriCorps sub-grantees, which include small nonprofit organizations. Therefore, in the development of procedures and instruments to be used in the surveys, special effort was made to minimize the data collection burden on such entities.
First, information will be collected, whenever possible, online. Most of the organizations to be surveyed already provide information on their Corporation grants electronically, and are familiar with responding in this manner. Second, survey procedures designed to minimize burden involve: voluntary participation; notification by the Corporation to officials of State Commissions and Corporation State Offices; and pre-notification and explanation by letter that a survey is being undertaken. Third, the sample includes only those organizations that are either the recipients of current AmeriCorps grants or have utilized AmeriCorps members within the preceding year, to minimize problems of recall or the necessity of respondents having to refer to records to answer questions. Fourth, the survey is brief, averaging approximately 10 minutes . Finally, the survey has been designed to require no reference to records or activity other than the time it takes to answer the questions. Should any respondent choose to consult records, it is anticipated that this would require minimal effort. This data collection, therefore, is not considered to have a significant economic impact on small entities.
A6 Consequences of Less Frequent Data Collection
Failure to collect the data outlined in this submission on an annual basis will result in insufficient and incomplete information with which to assess and report priority outcomes of the Corporation’s AmeriCorps programs—those outcomes that are observable only through collection of information directly from AmeriCorps members, andsub-recipient organization. Reporting is required to meet the Corporation’s annual GPRA and OMB budget justification responsibilities. Also, less frequent collection of the information outlined in this submission will result in the Corporation not having the information needed for internal program management and improvement purposes on a timely basis.
A7 Special Circumstances
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling paperwork Burden on the Public—General Information Collection Guidelines). There are no special circumstances that require deviation from these guidelines.
A8 Consultations Outside the Agency
The surveys have some similarities to the ones previously approved by OMB. The Urban Institute, an eight-member external Technical Working Group established by the Urban Institute and Princeton Survey Research Associates worked with the Corporation and its grantees to develop and test the indicators and related survey instruments that are summarized in this submission.
No payments are being made to respondents, who voluntarily agree to participate in this data collection.
Participation in the surveys associated with this submission is voluntary, with no adverse consequences to respondents for non- or partial participation.
Respondents will be informed that their responses are considered confidential—i.e., that there will be no names, identifiers, or other obvious means to connect individual respondents to their responses on the data files that will be forwarded to the Corporation at the conclusion of this project, unless required by law. Furthermore, reports prepared with these data will only involve aggregations that do not identify any individual respondents with their responses.
We also required that an outside group that collects this information, such as the Urban Institute and its subcontractors, has formal policies in place to ensure that data security is maintained when confidential data are collected or used for research purposes by that external group’s staff. All staff of the contractor and subcontractor, who will have access to the data being collected, will sign confidentiality pledges agreeing to maintain the confidential nature of the data. They will also destroy any linkage files that can connect individual respondents to their responses when these are no longer needed for quality control purposes.
The questions being asked on the surveys associated with this submission are not considered to be sensitive. Sensitive questions are defined as those whose answers, if made public, could cause physical, mental, emotional, economic, or other harm to an individual.
A12 Estimate of Annualized Burden Hours
Exhibit A12 provides information on the estimated time to complete the data collection and the total burden for the data collection effort—separately for each survey module and for the total survey effort.3
Exhibit A12: Estimate of Annualized Burden Hours
SURVEY MODULE |
A |
B |
C |
D |
E |
F |
G |
|
|
(A x B) |
|
(C x D) |
(B x D) |
(E / 60) |
|
Number of respon-dents |
Burden per Respon-dent (Minutes) |
Total Annual Burden (Minutes) |
Num-ber of Re-sponses |
Total Respon-dent Burden (Minutes) |
Total Burden per Respon-dent (Minutes) |
Total Project Burden (Hours) |
|
Current State and National Direct Members |
65,000 |
10 |
650,000 |
1 |
650,000 |
10 |
10,833 |
Current VISTA Members * |
(8600)* |
0 |
0 |
1 |
0 |
0 |
0 |
Current NCCC Members * |
(1400)* |
0 |
0 |
1 |
0 |
0 |
0 |
Former AmeriCorps Members |
2000 |
10 |
20,000 |
1 |
20,000 |
10 |
333 |
Organizations |
2000 |
10 |
20,000 |
1 |
5,850 |
10 |
333 |
TOTAL |
79,000** |
|
690,000 |
|
690,000 |
|
11,499 |
The current member numbers include those members who dropout of service.
*Current AmeriCorps members affiliated with the VISTA and NCCC programs are serving in federally operated programs and, as such, are deemed not to be members of the "public" for information collection purposes. Since they are exempt from burden calculation, the time associated with this aspect of the data collection is entered as zero (0) hours
** Does not include VISTA and NCCC current members.
A13 Estimated Recordkeeping and Reporting Cost Burden on Respondents
There are no record keeping or preparatory requirements in this data collection. The cost burden to respondents, therefore, is essentially the time required to respond to the survey questions.
Using the Independent Sector average cost per volunteer hour ($18.77), the estimated cost burden for all surveys is$215,836.22.
A14 Estimated Cost to the Federal Government
The total contracted cost to the Federal government for this effort in FY 2008 is $150,000 for indicator development, survey design, instrument development, data processing, data analysis, and report preparation. Other costs are a part of ordinary administrative costs.
A15 Reasons for Changes in Burden
This submission to OMB is a renewed request for approval with some changes in content and burden being requested.
A16 Tabulation Plans, Statistical Analysis, and Study Schedule
The primary external reporting of performance measures by the Corporation (as with most federal agencies) involves univariate analysis, with data for each outcome indicator aggregated across all respondents—separately for AmeriCorps State and National, NCCC, and VISTA. This will be the first level of analysis that will be used. The analysis will go further, however, as follows:
A series of bivariate analyses will be done to relate data for individual outcome indicators to specific service organization characteristics. This disaggregation into subgroups (such as whether the service organizations are large or small or are faith-based or not) will provide the Corporation and the AmeriCorps programs with considerably more detail, and thus considerably more useful information on which to explore where improvements may be needed
A series of bivariate analyses will be done to relate the data for individual outcome indicators to specific AmeriCorps member and former characteristics. This disaggregation into subgroups (such as for various member demographic characteristics and types of service provided) will provide the Corporation and the AmeriCorps programs with considerably more detail, and thus considerably more useful information on which to explore where improvements may be needed
A content analysis of the answers to open-ended questions will be done to identify patterns of responses that suggest particular service problems or provide improvement suggestions and reinforce positive management and operation strategies. (Each questionnaire contains a few questions that ask respondents why they gave poor ratings to particular service characteristics and for suggestions for improving the service.)
For a few questions in the member surveys for which the same wording is being used as that used in recent national surveys, results for AmeriCorps member respondents will be compared to those other surveys (consisting of data for the general population or relevant sub-populations) as a benchmark.
After examining the bivariate analyses, it is expected that a small number of multivariate analyses will be done that relate a particular outcome to two or more organization, member, and former member characteristics—if it appears that this will provide valuable insights with respect to the AmeriCorps program—as to specific conditions which correspond to successful outcomes.
Comparisons will be made of the outcome data to targets selected by AmeriCorps officials prior to tabulation of the data.
Comparisons of the latest data collected to previous time periods will be available. Year 2004 data will provide a baseline from which to compare future years’ outcomes.
A17 Expiration Date Display Exemption
Any reproduction of the data collection instrument, including an online version, will prominently display the OMB approval number and expiration date. In addition, cover letters sent to prospective respondents who will be interviewed by telephone will include the OMB approval number and expiration date.
A18 Exceptions to Certification
This submission, describing data collection, requests no exceptions to the Certificate for paperwork Reduction Act (5 CFR 1320.9).
Part B
STATISTICAL METHODS OF DATA COLLECTION
B1 Potential Respondent Universe
The potential respondent universe for the AmeriCorps outcome indicator surveys covered by this submission is divided into three categories: members, former members, and organizations. Each, in turn, is subdivided into multiple clusters.
Members. AmeriCorps members are separated into the following categories:
Current members, consisting of all full- and half-time AmeriCorps members whose service term is scheduled to expire within three months of the date of the survey. Requiring as a criterion for inclusion that current AmeriCorps members have completed the bulk of their service prior to the survey is intended to ensure that a sufficient amount of service time has occurred to be able to measure outcomes. Current members are separated by AmeriCorps program into State and National current members, VISTA current members, and NCCC current members.
Former members, consisting of all full- and half-time AmeriCorps members whose service term had previously expired and to ensure that a sufficient amount of time has elapsed to be able to measure sustained program outcomes while also ensuring that members’ recollections of their service experiences are still reasonably fresh. Former members are separated by AmeriCorps program into State and National former members, VISTA former members, and NCCC former members. Member swho dropped out of service are further separated for each program.
Organizations. The potential universe consists of officials representing all service delivery organizations (including non-profit, public, and educational institutions) that supervise AmeriCorps members or are the recipients of services provided by AmeriCorps members. Organizations are separated into the following categories:
Those supervising State and National members. As of the date of the survey, all such organizations must have been supervising State and National members for at least 12 months prior to the survey. Requiring as a criterion for inclusion that organizations have been supervising members for at least 12 months is intended to ensure a sufficient amount of service time to be able to measure program outcomes.
Those supervising VISTA members. As of the date of the survey, all such organizations must be supervising VISTA members who have been placed for at least 12 months prior to the survey. Requiring as a criterion for inclusion that members have been in place for at least 12 months is intended to ensure a sufficient amount of service time to be able to measure program outcomes.
Those receiving services from NCCC member teams. As of the date of the survey, all such organizations must have been provided services from NCCC members (teams)—consisting of at least one project completed at least three months prior to the survey. Requiring as a criterion for inclusion that organizations have been assisted for at least one project that had been completed at least three months prior to the survey is intended to ensure that there are program outcomes to measure while also that recollections of the organizations’ service experiences are still reasonably fresh.
B2 Sampling Method and Respondent Universe
Sampling methods will vary depending on the group to be surveyed. Surveys of current members and dropouts will be a universe. Former members and organizations will involve equal probability random samples drawn from AmeriCorps administrative records that meet the respective universe criteria indicated above. The various methods are described below.
Members. Two types of member samples will be drawn separately for each of the three AmeriCorps programs (State and National, VISTA, and NCCC).
B2.2 Procedures to Deal With Non-Response
A number of appropriate actions will be taken to ensure a high rate of response to the surveys covered in this submission.
For interviews to be conducted, an advance letter (on CNCS letterhead) describing the survey and encouraging participation will be sent to all prospective respondents. At least 20 calls will be made to reach prospective respondents. Calls will be staggered over different times of the day and days of the week to maximize the chances of making a contact. To further maximize response rates, specially trained and experienced refusal conversion interviewers will make additional calls to all refusal respondents; skilled interviewers can often convince reluctant potential respondents to participate in a survey, particularly a short survey such as those outlined in this submission. A toll free 800 number will be established that prospective respondents can call if they prefer to initiate the interview rather than schedule an appointment to complete the interview. This will give respondents total flexibility in determining when it is best for them to complete the survey.
For online interviews (with telephone back up in instances where potential respondents do not respond), an advance letter on CNCS letterhead will be sent to each potential respondent. It will include information on the web site location and provide a unique username and password. The site will be designed so that potential respondents can answer the questionnaire in one sitting or can return to the site as often as necessary to complete it; it is not anticipated, however, that the short surveys will require multiple sessions in most instances. Prospective respondents will also be given a telephone number and e-mail address should they have any questions or require technical assistance. Over an eight to 10 week period, two mailings of the cover letter and two postcards will be sent to organizations to encourage their participation in the survey. Potential respondents who do not complete a survey online will be contacted by phone and asked to complete a telephone interview. Up to 10 calls will be made to reach each potential respondent. Since respondents will consist of organization officials, executive-level interviews will conduct the interviews. Although most such calls will occur during the day, respondents' schedules will be accommodated should they wish to be interviewed in the evening or on the weekend. A toll free 800 number will be established that prospective respondents can call if they prefer to initiate the interview rather than schedule an appointment to complete it. This will give respondents total flexibility in determining when it is best for them to complete the survey.
For on-line surveys of current members, automated communications about the survey and other information will be sent regularly. Upon exit from the program, these members will be directed to the on-line survey.
Despite best efforts to reduce non-response, like most surveys there will likely be some non-response issues that need to be evaluated. This necessitates a careful look at patterns of non-response at each stage of data collection—not only reviewing the characteristics of members, former members, organizations, and end-beneficiaries who are not participating but also reviewing instances where only partial information has been provided. If the evaluation suggests that much of the non-response is non-ignorable (not occurring at random), then a non-response weighting adjustment will be considered to facilitate the analysis. If this is done, it will be documented and clearly reported in any presentation of the survey results.
B3 Pre-Testing of Procedures
Pre-testing will occur for the portal and organization survey questions. Other survey questions have been pre-tested and implemented through the same means that the survey will be administered. Pre-submission pre-testing of the instruments (using fewer than nine respondents per survey) has been done to consider wording, question order, and questionnaire length issues. More formal pre-testing (including cognitive interviews, testing of procedures for gathering information to create sample frames for end-beneficiary surveys, and testing of online and CATI systems) will occur subsequent to this Paperwork Reduction Act submission. Should that pre-testing experience result in the need to change procedures or item wording, proposed changes will be submitted to OMB for consideration as an amendment to this package.
B4 Persons Responsible for Statistical Aspects of the Design
The persons most directly responsible for statistical aspects of the design are:
Corporation for National and Community Service
Office of Research and Policy Development
Dr. Martin Abravanel
Senior Research Associate
The Urban Institute
(202) 261-5834
Mr. Harry Hatry
Principal Research Associate
The Urban Institute
(202) 261- 521
1 Current VISTA and NCCC members are serving in federally operated programs and, as such, are deemed not to be “members of the public" for information-collection purposes.
2 AmeriCorps members are individuals enrolled in an approved national service position and program who are or can become eligible for certain national service benefits, including an education award.
3 Data collection includes VISTA and NCCC members who are serving in federally operated programs. As such, they are deemed to be “members of the public" for information-collection purposes.
| File Type | application/msword |
| File Title | Part A |
| Author | MAbravan |
| Last Modified By | lshelton |
| File Modified | 2007-11-30 |
| File Created | 2007-11-30 |