VISTA_OMB_Clearance_Submission_-_Part_A[1]

Surveys Collecting Feedback on VISTA Member and Supervisor Training:

Pilot and Ongoing Annual Data Collection

Part A: Justification

1. Necessity of the Data Collection

Corporation for National and Community Service (CNCS) seeks approval to conduct a study to improve survey instruments that collect feedback on VISTA program training. The survey instruments collect the opinions of VISTA program participants about the training they received and their usefulness and relevance to their program experience. CNCS expects to use the improved instruments in future data collections that will provide ongoing feedback from program participants and supervisors to improve training content, timing and formats and thus increase the trainings’ ability to improve VISTA members experience and performance during their service year. The goal is that training be optimized and cost-effective.

2. Purpose of the Data Collection

The information is to be used for three purposes. First, the information from an initial round of data collection from a small sample will be used to calibrate and validate a competency model for VISTA members, and to revise the supervisor instrument. Second, the revised instruments will be validated using a universe of respondents. Third, the validated instruments will be used for an ongoing annual survey after the completion of the pilot study.

• Phase 1: During Phase 1 this study will collect information from a sample of VISTA Alumni, Early Terminators and Supervisors to calibrate and validate a competency model for VISTA members’ service. The goals of this phase are to provide data to finalize the competency model, examine how the competency related items work for these key populations (Supervisors, Alumni and Early Terminators), test the survey scales and items, and make questionnaire revisions. These revisions are anticipated to include shortening competency related scales to include only scales that will require regular input from supervisors, rather than all elements of the competency model. In addition, statistical analysis will be performed to determine whether the feedback from supervisors provides useful data for improving the training process. The revised instruments will be provided to OMB for approval before proceeding to Phase 2.

• Phase 2: The revised supervisor instrument will be administered to all current supervisors who did not receive or complete the survey during Phase 1. The goal is to complete the first annual data collection from VISTA supervisors and assess the performance of the revised instruments.

• Annual Survey: The final instruments will be used for an annual survey of VISTA supervisors upon the completion of Phase 2 of the pilot study. The purpose of this data collection is to allow for annual supervisor input into the VISTA training process and ensure continuing training feedback and improvement.

AmeriCorps VISTA members spend one year in full-time service to address the needs of low-income communities. All projects focus on building permanent infrastructure in organizations to help them more effectively bring individuals and communities out of poverty. VISTA service focuses on building permanent infrastructure and so requires skills in areas such as grant writing and volunteer mobilization. Organizations apply for and receive grant funds from CNCS to recruit and supervise VISTA members for specific service projects they identify. Within the grantee organization, members are assigned a supervisor who is generally a staff member of their sponsoring organizations.

Members receive training relevant to their service experience via pre-service orientation, on site orientation and training and the online VISTA Campus. In addition, at times during the service year, supervisors may provide additional organization and project specific training to VISTA members. Supervisors participate in a supervisor’s orientation that includes training.

To ensure that feedback on the training is useful, the survey instruments were developed using U.S. Department of Labor’s Competency Model. The Competency Model uses a conceptual framework that identifies competencies in four areas: foundation competencies needed for a work role (e.g. showing up or completing work on time); academic competencies; competencies for a specific occupation; and competencies that are required for that occupation in a specific industry. The VISTA Competency Model developed identifies “core competencies” that are needed for the full range of specific VISTA assignments while, at the same time, representing reasonable performance expectations (benchmarks) for VISTA members. The model helps define what skills VISTA members need to develop through training and provides a framework for obtaining feedback and identifying areas for improving training.

As a result, the items in the instruments focus on competencies, and not training content or topics. The items in the surveys solicit VISTA Supervisor, Alumni and Early Terminators opinions on:

• The degree to which they/the members they supervise have acquired and use the core competencies.

• The training models that they find most suitable for the various competencies.

• The training models that can most effectively support VISTA members in finding satisfaction, meaning, and opportunity in their service experience.

• The additional training that can provide ongoing development and support.

• The advantages or disadvantages of conducting training in-person versus a de-centralized, self-directed distance-learning format or a combination of formats.

3. Use of Electronic Media

The survey instruments will be administered on-line to all respondents. Informing respondents about the surveys will also be done electronically. Respondents will submit their responses electronically. CNCS maintains lists of all members including members who have completed a full term of service with VISTA Alumni and members that did not complete their VISTA term of service (Early Terminators), and of supervisors as part of its grant monitoring process. The lists include e-mail addresses for all respondents. Respondents will receive a link via e-mail that will take them directly to the on-line survey. Using electronic media to administer the surveys minimizes both respondent burden and cost to the government. Respondents can respond to the survey in any location and at their convenience.

4. Identification of Duplication

The instruments were developed using the Department of Labor’s Competency Model. The focus is to understand the effectiveness of the training program to help members develop core competencies. There are no other organizations with a program similar to CNCS’s VISTA program or that provide the specific training needed by VISTA members and supervisors. Also, this is CNCS’s first attempt to measure core competencies to inform CNCS’s training content to make the VISTA program more cost effective. Therefore, the information to be collected does not exist with another organization or with CNCS.

5. Impacts on Small Businesses

The proposed data collection will require input from VISTA site supervisors who are often staff members of small, non-profit entities (see level of burden estimates below), but will not impact small businesses. The survey is being offered on-line to minimize response burden, and the questions are designed to ask for information that, given the confidentiality of the survey, only the supervisor can provide directly (not data that is available from records or other sources).

6. Consequence to Federal Program or Policy

VISTA members spend a year of service helping non-profit and government agencies working with low-income communities. It is important for members to receive training to improve and enhance the core competencies needed for their specific service projects. The training is intended to assist members to effectively serve the low-income communities in which they will spend their service term. The survey will provide information to CNCS that will be informative in improving training content for members and thus likely improve members’ experience and performance.

7. Special Circumstances

• Alumni and Early Terminators will report information once during the study.

• Respondents will not be required to prepare any written response.

• Respondents will submit their responses once electronically.

• Respondents will not be required to retain any records.

• The information to be collected is designed to produce valid and reliable results that can be generalized to the universe.

• The study will use statistical data classification that has been reviewed and approved by OMB.

• The study will use a pledge of confidentiality that is supported by authority established in statute or regulation.

• The study will not require respondents to submit proprietary trade secrets or other confidential information.

8. Publication in the Federal Register and Consultation with Other Agency

The 60 Day Notice was published in the Federal Register on April 17, 2012. CNCS received one comment which stated that the information being sought should already be available to CNCS staff without the requested information collection. However, such information in the detail required to make informed decisions on the future structure and content of VISTA member training is not available to CNCS staff without the requested information collection.

CNCS consulted with a selected group of Stakeholders Advisory Group (SAG) which represents VISTA Supervisors, members and alumni at host agencies.  SAG members represented various program models and sizes comparable to other programs under the CNCS umbrella. CNCS and its contractor JBS had extensive and ongoing discussions with the SAG members during the instrument development stage and received feedback on all instruments. SAG members were first asked to participate in pilot surveys by telephone for their respective roles. During the course of the interview, members were also asked to provide feedback on the content matter that would assist with future editions of the instrument.  A second round of instruments was developed based on members’ feedback. A subset of the SAG group was invited to take part in a cognitive interview with the new instruments. During this process we probed participating SAG members on designated questions to minimize the burden on participants. This last step ensured that CNCS had duly incorporated the Stakeholders Advisory Group feedback and streamlined the content of the instrument to better reflect the work done by CNCS programs.

9. Gifts or Payments

Respondents will not receive any incentive for participating in the study.

10. Assurance of Confidentiality

The beginning of each instrument includes a statement of assurance of confidentiality. The statement indicates that participation in this survey is voluntary. It also states that individual responses will be seen only by the JBS research team and will not be shared with staff from CNCS, AmeriCorps VISTA, or anyone else. The statement also notes that the information will be reported to CNCS as averages and totals; and that input is entirely confidential.

The assurance of confidentiality reads as follows:

This survey is being administrated by JBS International, an independent research firm contracted by the Corporation for National and Community Service (CNCS) to collect information in order to better understand the VISTA training and service experience. Your participation in this survey is voluntary. Individual responses will be seen only by the JBS research team and will not be shared with staff from CNCS, AmeriCorps VISTA, or anyone else. All information will be reported to CNCS as averages and totals; your input is entirely confidential.

11. Questions of Sensitive Nature

The survey instruments do not include any questions of a sensitive nature.

12. Burden Hours

The instruments will be administered to three groups of respondents who are subject to this clearance. The three groups are:

• Private sector and state, local and Tribal government supervisors of current VISTA members;

• Individuals who are VISTA Alumni having completed a full term of service with VISTA;

• Individuals who are Early Terminators that did not complete their VISTA term of service.

The burden hours for the three groups subject to this clearance are described in the table below.

Category of Respondent	No. of Respondents	Participation Time (Minutes) per Respondent	Burden Hours per Respondent	Burden Hours All Respondents
Phase 1
Private Sector and State, Local or Tribal Government: Supervisors*	250	30	0.50	125
Individuals: VISTA Alumni	385	30	0.50	192.5
Total	635	30	0.50	317.5

* VISTA supervisors may be staff from the private sector, non-profits or state, local or tribal governments. Since new projects begin throughout the year, it is not possible to define exactly how many belong to each group.

We conducted an internal test of each survey instrument with five JBS staff. We asked JBS staff to record how long it took to complete the survey and what problems, if any, they encountered.

Testers reported that the instructions were easy to follow and the questions were clear and easy to answer. Testers did identify instances of unclear wording, redundant questions, and problems with response options. They also identified some formatting issues and typographical errors. The instruments were revised to address the problems testers identified.

The time to complete each instrument varied and depended as much on who completed an instrument as on instrument length. Based on the internal testing we eliminated and revised items to reduce participation time. We retested the instruments in an on-line format with additional staff members. And based on the revised, on-line tests we estimated 25 to 30 minutes to complete an instrument.

The results of the second round of internal testing were:

Instrument	Minutes to Complete Instrument	Number of Screens in Instrument	Number of Items in Instrument
Member Alumni	30	30	32
Member Early Terminator	25	26	26
Supervisor	30	24	37

13. Explain any Program Changes

This is an application for new collection. There are no program changes.

14. Dissemination of Information

The information collected will be analyzed in the following manner:

• Comparison of findings from alumni, early terminators and supervisors on the same issues;

• Statistical analysis of the strength of the scales constructed to measure the basic concepts, using appropriate statistical analyses; and

• Analysis of the findings of the surveys to assess the strength of the different factors in the conceptualization for predicting relative success as a VISTA member.

The results of the analysis will inform a final set of instruments that will be used in future years for data collections by CNCS to collect feedback on its training. Analysis of this information will be used to improve VISTA training for both members and supervisors as well as allocate training resources across the various training formats and fine tune training delivery times (e.g., orientation versus mid-term) and curriculum.

Timeline

Activity/Task	Date
Data collection (Pilot)	9/17/2012
Analyze, revise and finalize instruments based on data from pilot	10/15/2012
Data collection universe of supervisors	11/1/2012
Analysis of Phase 2 Data	12/1/2012
Final Report	2/28/2013

15. Display of Expiration Date for OMB Approval

The expiration data for OMB approval of the information collection will be displayed on the first page of the online survey.

16. Explanation of Exceptions in Item 19, Certification for Paperwork Reduction Act Submissions

There are no exceptions claimed in Item 19, Certification for Paperwork Reduction Act Submissions of OMB Form 83-I.