TTA OMB Section B

OMB Clearance

Assessment of TTA

Package – Section B

August 28, 2013

Prepared by

Chris Spera

Director of Research and Evaluation, CNCS

Barry Goodstadt, PhD.

Research and Evaluation, CNCS

Marlene Zakai

Director of Strategic Initiatives, CNCS

Contents

B. Collection of Information Employing Statistical Methods 2

B.1. Respondent Universe 2

B.2. Procedures for the Collection of Information/Limitations of the Study 2

B.3. Methods To Maximize Response Rates and Deal With Issues of Nonresponse 3

B.4. Tests of Procedures or Methods 3

B.5. Names and Telephone Numbers of Individuals Consulted 3

B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe

The respondent universe is all CNCS grantee training participants, for a total of 10,000 respondents. This comprises participants in Regional Grantee Conferences, the AmeriCorps Grantee Symposium, the Senior Corps Virtual Conference.

A key part of this assessment will be to pinpoint which elements of the program are more effective in teaching and which parts of the program less effective.   With this in mind the measurement schema includes a set of pre- and post-training measures  which will be used to detect learning.  One of the challenges of this situation is that measurement needs to  place on-site at the training location where information is being presented.   To provide for more accurate measures of learning effects,  pre- and post-tests should be administered to the same individuals thereby providing the most direct measure of learning (or change) from before training until after training.  This is effectively  a repeated measures type of design where each individual serves as their own control (cf.,Winer, 1962).   

While it is conceivably possible to take independent samples of pre and post training measures from different individuals and create  total difference score between the pre-measures and the post measures to assess learning,  such an approach is not as accurate as a repeated measures approach. Given the onsite nature of data collection in this project, it is exceedingly difficult to develop a unbiased sampling procedure.  Thus, in the training environment  it will be difficult to select a random sample of participants who will complete both pre- and post-tests.  In a large room where training is conducted randomized selection of participants may not be logistically possible since people attending training do not sit in random order (people who know each other sit together and may enter the room at the same time).  Any sampling schema would have to assure that those selected to take the pretest would also complete the posttest. 

Administering pre- and posttests to all trainees gets around this problem. Even if we were to find a way to randomly select individuals for testing, the mere selection of such individuals could create a "demand  characteristic" which would bias the results within that sub-population (e.g ,Rosenthal & Rosnow, 1969.  Here again, giving pre- and posttests to all participants controls for such effects. 

B.2. Procedures for the Collection of Information/Limitations of the Study

For in-person training events such as the Regional Grantee Conferences and the AmeriCorps Symposium, the data collection will be administered in a pre and post test scantron questionnaire. This is the most cost effective means for the collection as it ensures participation by all trainees. The Senior Corps Virtual Conference in 2014 will provide an opportunity to pilot the collection in an on-line internet based format. Since all participants at the Senior Corps Virtual Conference must have internet access to participate in the conference, we believe all participants will have access and this will be a cost effective and timely way to collect the data since the posttest will occur immediately after training has been completed.

B.2.1. Statistical Methodology for Stratification and Sample Selection

Not applicable.

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not applicable.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not applicable.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not applicable.

B.3. Methods To Maximize Response Rates and Deal With Issues of Nonresponse

We are planning to collect data from training participants in the location and at the time at which they complete the training activity.  This method is designed to maximize response since the individuals being trained will be at the location where they have completed training. We anticipate that nonresponse will be quite limited.

We expect a response rate of 85% or better.

B.4. Tests of Procedures or Methods

This effort is designed to evaluate the effectiveness of the training program and will yield rapid results to ensure that the training program is accomplishing its goals. All questions have been drafted and have undergone two internal reviews: (1) a review by the Research and Evaluation Office and (2) a review by staff subject matter experts (AmeriCorps, Senior Corps, Grants Management and Federal Financial Management Offices).

B.5. Names and Telephone Numbers of Individuals Consulted

Below is the contact information for the individuals responsible for this effort.

Name Number

Chris Spera, Director of

Research and Evaluation 202-606-3910

Barry Goodstadt

Research and Evaluation 202-606-3238

Marlene Zakai,

Director of Strategic Initiatives 202-606-6692