VISTA OMB Clearance Application Submission Draft Part B

Surveys Collecting Feedback on VISTA Member and Supervisor Training:

Pilot and Ongoing Annual Data Collection

Part B: Collections of Information Employing Statistical Methods

1. Potential Respondents and Sampling

The universe for this data collection is VISTA Alumni, Early Terminators and Supervisors.

The surveys will be administered to three groups of respondents that are subject to this clearance request:

• Private sector and state, local and Tribal government Supervisors

• Individuals who are VISTA Alumni having completed a full term of service with VISTA

• Individuals who are Early Terminators that did not complete their VISTA term of service

Supervisors will be surveyed in both phases of the study and the ongoing annual survey. VISTA alumni and early terminators will be surveyed only during Phase 1 (pilot).

Phase 1:

The samples for Phase 1 pilot study consist of three separate simple random samples of Alumni, Early Terminators and Supervisors drawn using SAS statistical software (PROC SURVEYSELECT).

• VISTA Alumni who served in 2010 and 2011 (N = 385 individuals);

• Early Terminators who served in 2010 and 2011 (N = 385 individuals); and

• Supervisors who served in 2010 and 2011 (N = 250).

The analysis of items and scales will then inform questionnaire revisions. The sample sizes for Phase 1 are small and designed to provide sufficient data to conduct the factor analysis of scales and represent a ratio of 31:1 sample members to scale items for the largest scale that is consistent with current practice in terms of sampling. A sample size of 385 means that the point estimates from the binary survey results will all have 95% confidence limits of plus or minus 5%. The supervisor survey sample size of 250 means that the binary point estimates will have a 95% confidence interval of plus or minus 6%.

Phase 2:

For Phase 2, the survey will be administered to the entire universe of Supervisors (a census). Data obtained from full-scale implementation of the on-line survey will provide another opportunity to generate statistical estimates of reliability for scaled items and constructs. The data will also serve as core information for JBS’ final project report to CNCS.

Ongoing Annual Survey:

All Supervisors will be surveyed annually using the validated supervisor instrument.

2. Procedures for Collection of Information

The study is divided into two phases. The Phase 1 pilot study samples consists of three separate simple random samples of Alumni, Early Terminators and Supervisors drawn using SAS statistical software’ (PROC SURVEY SELECT). For the Phase 2, the survey will be administered to the entire universe of Supervisors.

The dataset of completed surveys from each group from Phase 1 will serve as the basis for statistical estimation of reliability for survey items and constructs. Scaled items will be assessed through use of the Cronbach's alpha, requiring a score of 0.8 or better. Each sample will be analyzed individually as final items will need to work in all three groups (Alumni, Early Terminators and Supervisors). Data obtained from Phase 2 implementation of the on-line survey will provide another opportunity to generate statistical estimates of reliability for scaled items and constructs. The data will also serve as core information for JBS’ final project report to the CNCS.

3. Methods to Maximize Response Rates

For Phase 1 oversampling will be used to ensure satisfactory response rates. We assume an 80 percent response rate for each group. For example, 315 Supervisors will receive information about the survey to assure that there is an adequate pool of survey respondents (250). A satisfactory response rate will be achieved by preparing respondents ahead of time with a description of the survey and its importance, and by sending reminder notes.

For Phase 2, which will include the entire universe of Supervisors, we will send multiple follow-up e-mail encouraging participation in the survey. If needed, we will contact participants by telephone to encourage their participation.

4. Tests of Procedures or Methods

The Phase 1 pilot is the agency’s proposed test to refine the collection of information to minimize burden and improve utility. Phase I will collect information from a sample of VISTA Alumni, Early Terminators and Supervisors to calibrate and validate a competency model for VISTA members’ service. The goals are to examine how the competency related items and scales work for these key populations (Supervisors, Alumni and Early Terminators) in order to test the survey scales and items, and make questionnaire revisions. In addition, statistical analysis will be performed to determine whether the feedback from supervisors provides useful data for improving the training process.

5. Contact Information

• The following individuals were consulted on statistical aspects of the design:

Susan Gabbard, PhD

Annie Georges, PhD

JBS International, Inc.

Aguirre Division

555 Airport Boulevard, Suite 400

Burlingame, CA 94010

T: 650.373.4949 (direct)

T: 650.373.4900 (main)

F: 650.348.0260 (fax)

E: [email protected]

• Name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency

JBS International, Inc.

Aguirre Division

555 Airport Boulevard, Suite 400

Burlingame, CA 94010

Page 3 of 3