Part B

Part B

STATISTICAL METHODS OF DATA COLLECTION

B1 Potential Respondent Universe

The potential respondent universe for the AmeriCorps outcome indicator surveys covered by this submission is divided into three categories: members, former members, and organizations. Each, in turn, is subdivided into multiple clusters.

1. Members. AmeriCorps members are separated into the following categories:

• Current members, consisting of all full- and half-time AmeriCorps members whose service term is scheduled to expire within three months of the date of the survey. Requiring as a criterion for inclusion that current AmeriCorps members have completed the bulk of their service prior to the survey is intended to ensure that a sufficient amount of service time has occurred to be able to measure outcomes. Current members are separated by AmeriCorps program into State and National current members, VISTA current members, and NCCC current members.

• Former members, consisting of all full- and half-time AmeriCorps members whose service term had previously expired and to ensure that a sufficient amount of time has elapsed to be able to measure sustained program outcomes while also ensuring that members’ recollections of their service experiences are still reasonably fresh. Former members are separated by AmeriCorps program into State and National former members, VISTA former members, and NCCC former members. Member swho dropped out of service are further separated for each program.

2. Organizations. The potential universe consists of officials representing all service delivery organizations (including non-profit, public, and educational institutions) that supervise AmeriCorps members or are the recipients of services provided by AmeriCorps members. Organizations are separated into the following categories:

• Those supervising State and National members. As of the date of the survey, all such organizations must have been supervising State and National members for at least 12 months prior to the survey. Requiring as a criterion for inclusion that organizations have been supervising members for at least 12 months is intended to ensure a sufficient amount of service time to be able to measure program outcomes.

• Those supervising VISTA members. As of the date of the survey, all such organizations must be supervising VISTA members who have been placed for at least 12 months prior to the survey. Requiring as a criterion for inclusion that members have been in place for at least 12 months is intended to ensure a sufficient amount of service time to be able to measure program outcomes.

• Those receiving services from NCCC member teams. As of the date of the survey, all such organizations must have been provided services from NCCC members (teams)—consisting of at least one project completed at least three months prior to the survey. Requiring as a criterion for inclusion that organizations have been assisted for at least one project that had been completed at least three months prior to the survey is intended to ensure that there are program outcomes to measure while also that recollections of the organizations’ service experiences are still reasonably fresh.

B2 Sampling Method and Respondent Universe

Sampling methods will vary depending on the group to be surveyed. Surveys of current members and dropouts will be a universe. Former members and organizations will involve equal probability random samples drawn from AmeriCorps administrative records that meet the respective universe criteria indicated above. The various methods are described below.

Members. Two types of member samples will be drawn separately for each of the three AmeriCorps programs (State and National, VISTA, and NCCC).

B2.2 Procedures to Deal With Non-Response

A number of appropriate actions will be taken to ensure a high rate of response to the surveys covered in this submission.

 

For interviews to be conducted, an advance letter (on CNCS letterhead) describing the survey and encouraging participation will be sent to all prospective respondents. At least 20 calls will be made to reach prospective respondents. Calls will be staggered over different times of the day and days of the week to maximize the chances of making a contact. To further maximize response rates, specially trained and experienced refusal conversion interviewers will make additional calls to all refusal respondents; skilled interviewers can often convince reluctant potential respondents to participate in a survey, particularly a short survey such as those outlined in this submission.  A toll free 800 number will be established that prospective respondents can call if they prefer to initiate the interview rather than schedule an appointment to complete the interview. This will give respondents total flexibility in determining when it is best for them to complete the survey.

 

For online interviews (with telephone back up in instances where potential respondents do not respond), an advance letter on CNCS letterhead will be sent to each potential respondent. It will include information on the web site location and provide a unique username and password. The site will be designed so that potential respondents can answer the questionnaire in one sitting or can return to the site as often as necessary to complete it; it is not anticipated, however, that the short surveys will require multiple sessions in most instances. Prospective respondents will also be given a telephone number and e-mail address should they have any questions or require technical assistance. Over an eight to 10 week period, two mailings of the cover letter and two postcards will be sent to organizations to encourage their participation in the survey. Potential respondents who do not complete a survey online will be contacted by phone and asked to complete a telephone interview. Up to 10 calls will be made to reach each potential respondent. Since respondents will consist of organization officials, executive-level interviews will conduct the interviews. Although most such calls will occur during the day, respondents' schedules will be accommodated should they wish to be interviewed in the evening or on the weekend. A toll free 800 number will be established that prospective respondents can call if they prefer to initiate the interview rather than schedule an appointment to complete it. This will give respondents total flexibility in determining when it is best for them to complete the survey.

For on-line surveys of current members, automated communications about the survey and other information will be sent regularly. Upon exit from the program, these members will be directed to the on-line survey.

Despite best efforts to reduce non-response, like most surveys there will likely be some non-response issues that need to be evaluated. This necessitates a careful look at patterns of non-response at each stage of data collection—not only reviewing the characteristics of members, former members, organizations, and end-beneficiaries who are not participating but also reviewing instances where only partial information has been provided. If the evaluation suggests that much of the non-response is non-ignorable (not occurring at random), then a non-response weighting adjustment will be considered to facilitate the analysis. If this is done, it will be documented and clearly reported in any presentation of the survey results.

B3 Pre-Testing of Procedures

Pre-testing will occur for the portal and organization survey questions. Other survey questions have been pre-tested and implemented through the same means that the survey will be administered. Pre-submission pre-testing of the instruments (using fewer than nine respondents per survey) has been done to consider wording, question order, and questionnaire length issues. More formal pre-testing (including cognitive interviews, testing of procedures for gathering information to create sample frames for end-beneficiary surveys, and testing of online and CATI systems) will occur subsequent to this Paperwork Reduction Act submission. Should that pre-testing experience result in the need to change procedures or item wording, proposed changes will be submitted to OMB for consideration as an amendment to this package.

B4 Persons Responsible for Statistical Aspects of the Design

The persons most directly responsible for statistical aspects of the design are:

Corporation for National and Community Service

Office of Research and Policy Development

Dr. Martin Abravanel

Senior Research Associate

The Urban Institute

(202) 261-5834

Mr. Harry Hatry

Principal Research Associate

The Urban Institute

(202) 261- 521

4