Supporting Statement

OMB Control No. 0693-0033

NIST Generic Clearance for Program Evaluation Data Collections

Request for Clearance to Conduct a Survey for

The National Institute of Standards and Technology (NIST)

National Research Council (NRC) NIST Award Decliners

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

The National Institute of Standards and Technology (NIST) proposes to conduct a telephone interview to assess the NIST National Research Council (NRC) Postdoctoral Research Associateship Program (RAP) through the company Westat, who has been contracted to carry out the assessment of the Program. This two-year, multi-method study seeks to examine the postdoctoral experiences and career trajectories of NRC NIST postdocs. 

Telephone interviews will be conducted to survey NRC NIST award decliners.

This group was selected because the input from these respondents will help us answer questions about their postdoctoral experiences and career trajectories of NRC NIST postdocs.

The assessment of the NRC NIST Postdoctoral Research Associateship Program has two major objectives:

The first objective is to conduct a program evaluation that assesses the organizational benefits, if any that accrue to NIST as a result of RAP. This objective has three major goals to determine: (1a) if RAP is an effective recruitment tool that converts talented postdocs into permanent employees; (1b) if, and how, RAP advances NIST programs and its mission; and (1c) if the outreach efforts of NIST are effective in attracting high quality candidates to apply for and accept RAP awards.

The second objective is to examine the perceived and measurable program benefits for the NRC NIST postdocs. The two major goals of this objective are to determine: (2a) the usefulness of participation for advancing the careers and productivity of the NRC NIST postdocs; and (2b) participants’ perceptions of the effects of program participation. As shown in Chart 1, the interviews will explore goal 2b.

The primary purpose of this survey data collection is to gain a better understanding of how the program is being implemented and what its impact is on the sponsoring institution and participating individuals. NIST will use the data to assess the extent and effects of programmatic activities and associated outcomes of these activities. All information collected will be used to provide analytical and policy support to NIST’s International and Academic Affairs Office (IAAO), helping NIST make decisions about future program initiatives to improve postdoctoral training.

Chart 1.—Goals of Study and Links to Interview with NRC NIST Decliners

Goal/research question	Indicator
1a. Is RAP beneficial to NIST as a recruitment tool?	Experiences of NRC NIST Decliners
1b. Beneficial to NIST programs and mission?	Experiences of current NRC NIST Decliners
1c. Effectiveness of NIST outreach efforts to attract high quality candidates?	Quality of NRC NIST decliners versus NRC NIST applicants
2a. What are the career trajectories of NRC NIST postdocs and how do they compare to non-RAP awardees?	Reports of career activities of NRC NIST decliners versus- NRC NIST applicants
2b. What is the attractiveness of the NIST RAP program?	Subjective list of benefits and concerns (e.g., relationship with advisor, career opportunities)

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The brief telephone interview was designed to collect data from the NRC NIST decliners and developed in consultation with the NIST IAAO leadership and the evaluation team.  The instrument includes two questions: 1) why awardees declined their award, and 2) what they did instead.  During development, a third qualitative question was developed in response to suggestions for improvement. This question is designed to obtain a retrospective reflection on respondents’ decision to decline the award offer.  Suggestions to collect additional data were rejected in favor of the most streamlined design.  The telephone script includes standard informed consent language explaining the purpose of the study and assurances of confidentiality and the instrument was pre-tested among project staff.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

Updating Respondent Contact Information

We will need contact information for respondents in order to administer the surveys and telephone interviews. This information will include e-mail addresses, telephone numbers (work or home), and work or home addresses.

Chart 2 below shows the number of respondents for whom there is contact information. As seen in the chart, all 65 respondents will need updated contact information.

Chart 2.—Selected Sample With Contact Information

Population size	Select sample size	Select sample with contact information	Select sample need contact information
65	65	0	65

Westat will utilize hourly staff to trace contact information. This staff will include individuals with tracing experience, who will be trained in a multitude of tracing techniques by one of the Westat team members working on the NIST project. Based on the experiment with small samples, it is expected that accurate contact information will be obtained for approximately 60 percent of the award decliners in the list.

Sample Selection and Expected Response Rate for Award Decliners

Chart 3 shows information on selected, field, and final sample sizes for award decliners. Westat assumes that the response rate for those with good contact information will be approximately 80 percent. The final target sample size is 31 respondents. All of the NRC NIST award decliners will be surveyed and 31 respondents are expected to complete the survey.

Chart 3.—Various Sample Sizes and Expected Response Rate

Population size (N)	Selected sample sizeA	Correct contact information rate (%)	Field sample sizeB	Response rate (%)	Final sample size (n)
65	65	60	39	80	31

The information collected in this study represents the minimum effort required to assess the benefits of the NRC NIST Postdoctoral Research Associateship Program. Chart 4 shows the estimated burden hours to complete the interview. It is estimated that the total annual burden will be 3 hours.

Chart 4.—Estimated Annual Burden Hours for Interview with NRC NIST Decliners

Number of respondents	Burden per respondent (in minutes)	Total annual burden1 (in hours)
31	5	3

Survey Administration

The data collection will be accomplished by means of telephone interviewing. A short four-question telephone interview is scheduled to be administered in summer 2011 among approximately 31 NRC NIST decliners.

To improve response rates, experienced telephone interviewers, trained in nonresponse conversion, will conduct up to four follow-up calls.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The present study will use a mixed methods approach to analyze the survey data collected. Close-ended items on the survey will be analyzed quantitatively. Data will first be analyzed descriptively and then used in multivariate statistical models to examine study questions.

Qualitative techniques will be employed to analyze open-ended survey items. Content analysis will be used to code data thematically, and pattern codes will be generated for the final descriptive analysis. Evaluators will first examine the data individually and then convene with each other to compare and contrast observed patterns. The frequencies of agreed-upon patterns will be included in the final report.

The final report will be delivered to the IAAO at NIST upon the conclusion of the project (approximately 24 months from launch). This report will include a summary of all the data collected and analyzed during the assessment, as well as conclusions that address each of the research questions and project components.

4