TechTrainEvalSS.Part B.AW.2012

Supporting Statement B

Technical Training Program Course Effectiveness Evaluations

OMB Control Number 1029-0110

Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses, or employ statistical methods?” is checked "Yes," the following documentation should be included in Supporting Statement B to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

During 2012, OSM sent 646 surveys to all students and their 566 Supervisors who participated in OSM’s training classes requesting their input into refinement of each training course taken. The following table indicates the number of surveys sent and the response rate.

	Students	Supervisors	Total
Surveys sent	646	566	1,212
Surveys Returned	200	110	310
Response Rate	31%	19%	26%

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Each participant in a training class and his/her supervisor are e-mailed a request approximately three months after they have completed the training course requesting that they complete the evaluation survey and return it to OSM. Since all students are sent the survey, no specialized sampling methodology is employed. This is a qualitative survey not intended to develop statistics.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

OSM sends e-mail reminders with a “hot link” to the evaluation form in order to help bolster the response rate. Also, OSM’s training program is enthusiastically supported by respondents, the State regulatory authorities, who have a natural incentive to respond, providing course feedback. Lately, OSM has received a 26% response rate. This is a qualitative survey and such a moderate response rate is adequate for our evaluation purposes. However, we are exploring ways to increase our response rate. We are considering expanding our time frame for returning surveys for one week longer.

We are looking for response rates between 45%-50%. We are interested in hearing from course participants and their supervisors who want to make things better and courses more useful. We are looking for feedback to improve course content and presentation, rather than to create numerical statistics on perceived course quality. Such statistics would not be meaningful, because course content would be changing significantly over time.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

OSM did not conduct tests prior to the initial use of the surveys. However, several individuals with knowledge of survey techniques provided guidance in developing the surveys to ensure their utility and response rate.

5. Provide the names and telephone numbers of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The individual who created the surveys and who has responsibility for the data collection and analyzes is:

Ann M. Walker, Chief, OSM, National Technical Training

(202) 208-2824

3