Supporting Statement part B_2015

SUPPORTING STATEMENT

FOR PAPERWORK REDUCTION ACT SUBMISSION

B. Collection of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on Form 83-I is checked “Yes,” the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe the potential respondent universe (including a numerical estimate) and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, state and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

IES currently has supported over 1000 training participants through its fellowship programs and methods/researcher training programs, and IES introduced a new training program for undergraduates in FY 2016. Because IES has provided hundreds of pre-/postdoctoral fellows and other researchers and intends to continue to do so, IES wishes to survey participants in each type of training program. In developing this new plan, IES is incorporating what it has learned from previous surveys. For example, in recent years, participation has dropped to a response rate of approximately 60%. One possible reason for the decline is that IES conducted annual, population surveys. Thus, participants may have been fatigued. To account for this and to include new feedback from participants in our other training programs, IES proposes a few changes to its customer-satisfaction survey:

• Survey fellows (predoctoral and postdoctoral) at select time points rather than continually and in perpetuity.

• Use more targeted survey questions that align with the participants’ particular role (e.g., beginning IES postdoctoral fellow, completing IES predoctoral fellow, researcher in Methods Training program).

• Discontinue population surveys of all fellows.

• Begin surveying non-fellow training participants.

As noted above, IES has provided training through various grants awarded by the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER). These awards began in FY 2004 and have supported predoctoral and postdoctoral fellowships as well as trained current, active researchers.

IES previously surveyed all the fellowship program participants but none of the methods training programs, and IES used to survey the full population of fellowship participants annually, using the same survey items with minimal skip rules. We wish to change this moving ahead.

Our first major change will be that, each year, we will survey all of the incoming predoctoral and postdoctoral fellows using targeted surveys that include questions relevant to their particular positions (e.g., why they decided to apply for an IES fellowship). Thus, the surveyed participants will receive fewer questions. On average, the predoctoral programs hire 45 new fellows per year per program, and the postdoctoral programs hire one to two new fellow per year per program. We predict sending out approximately 78 surveys per year depending on recruitment efforts and the number of new programs that begin during Year 2 and 3.

Table 1: Estimate of Participants for Entrance Survey

	Predoctoral Fellows	Postdoctoral Fellows	Total
Year 1	45	30	75
Year 2	45	35	80
Year 3	45	35	80
Total	135	90	235

Each year, we will also survey all of the exiting predoctoral and postdoctoral fellows using targeted surveys that include questions relevant to their particular positions (e.g., if they have found employment). On average, the predoctoral programs complete 20 to 30 fellows per year per program, and the postdoctoral programs complete one to two fellow per year per program. We predict sending out approximately 145 surveys per year depending on recruitment efforts and the number of new programs that begin during Year 2 and 3.

Table 2: Estimate of Participants for Exit Survey

	Predoctoral Fellows	Postdoctoral Fellows	Total
Year 1	170	35	205
Year 2	100	30	130
Year 3	70	30	100
Total	340	95	435

Every other year, we will survey all active predoctoral fellows. Because this package covers 3 years, we believe that we will conduct this survey only once prior to resubmitting for continued approval. The numbers below reflect our estimate of the number of active predoctoral fellows during the possible survey period.

Table 3: Estimate of Participants for Fellows Satisfaction Survey

	Predoctoral Fellows
Year 2 or 3	170

Every year, we will also survey fellows who completed their programs 2 to 3 years previous to the time of the survey.

Table 4: Estimate of Participants for Follow-up Survey

	Predoctoral Fellows	Postdoctoral Fellows	Total
Year 1	100	40	140
Year 2	110	40	140
Year 3	170	35	205
Total	380	115	495

Each year, we will also conduct follow-up surveys with participants who had completed a non-fellowship IES training program, e.g. Methods Training and Pathways Training. These participants will receive only one survey for their participation in a given training.

Table 5: Estimate of Participants for Non-Fellowship IES Training Satisfaction Survey

	Training Participants
Year 1	135
Year 2	135
Year 3	135

2. Describe the procedures for the collection of information, including:

• Statistical methodology for stratification and sample selection.

• Estimation procedure.

• Degree of accuracy needed for the purpose described in the justification.

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

No sampling is done. The survey will be emailed to all potential respondents who meet the requirements for a particular survey (e.g., entering fellowship program, exiting a fellowship program) and will be collected once a year. IES is using the census approach because the populations of fellows and non-fellow trainees are small enough to make a census survey manageable and to make a sample not reliable for subgroup findings.

Table 7: Summary Table of Surveys, Participants, and Frequency

Name of Survey	Type of Participant	Frequency	Approximate number of Participants
Fellows Entry	All incoming predoctoral and postdoctoral fellows	Annual	78
Fellows Exit	All predoctoral and postdoctoral fellows who completed that year	Annual	145
Fellows Satisfaction	All currently active predoctoral fellows	Bi-annually	170
Fellows Follow-up	All predoctoral and postdoctoral fellows who completed 2 to 3 years prior to the survey	Bi-annually	161
IES Training Satisfaction	All training non-fellowship training participants	Annual	135

As Table 7 indicates, predoctoral and postdoctoral fellows will receive multiple surveys. To help reduce survey fatigue and burden, however, IES will survey fellowship training participants only at important milestones as noted in the table below. Furthermore, two of the surveys will happen every-other year, and no fellow will complete more than one survey per year. Most fellowship participants will receive only three surveys over the course of their training activities and beyond, though a subset of the predoctoral fellows may receive a fourth survey (i.e., the “Active” survey) if they are active fellows during the time of the survey.

Table 8: Summary Table of Frequency of Participation

	Entrance	During training	Exit	Follow-up
Predoctoral Fellow	X	X	X	X (only once, 2 or 3 years after completing)
Postdoctoral Fellow	X		X	X (only once, 2 or 3 years after completing)
Other Trainees				X (only once, during the calendar year after completing)

3. Describe methods to maximize response and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Response will be maximized through multiple avenues of contact:

1. emails from IES,

2. direct contact from the on-site director of the fellowship or training program, and if needed

3. phone contact from IES.

IES sends a link to the online survey by email, and responses are collected online to help reduce burden and increase response rates.

In the case of non-response, IES will send a follow-up email to the participant 3 business days after the deadline of the survey and again 2 weeks after that. If there is no response within 2 weeks, IES will verify the email address and will send a third email using any corrected email addresses. In addition, each training program has a director based at a university or training organization. IES will work with these partners to ensure that IES has the most accurate contact information and that fellows and training participants are aware of the survey. Also, if there is high degree of non-responsiveness, IES will ask the training directors to send a general reminder to all fellows or participants to complete the survey. The director will not be informed who has or has not responded. In addition, the director will not have access to individual fellows’ or participants’ responses to the survey.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The questions on the survey are drawn from two fellow satisfaction surveys that had been given in 2008 and that OMB determined needed to be converted into a survey. These satisfaction surveys were done with IES fellows, similar to the population IES is targeting here. There will be no testing of the survey to be used in future years.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other persons who will actually collect and/or analyze the information for the agency.

As the entire population is being surveyed, results are to be simple tabulations, and the people analyzing the data have appropriate backgrounds, no additional consultation will occur.

Staff who will send out and analyze data:

Dr. Corrine Alfeld 202-245-8203

Dr. Phill Gagne 202-245-7139

Dr. Meredith Larson 202-245-7037

Dr. Katina Stapleton 202-245-6566

Dr. Katie Taylor 202-245-6716

Wendy Wai 202- 245-7564

5