Section B

Section B



B.1. Respondent Universe and Sampling Methods

The target population for this study includes two groups of individuals: 1) potential NIH job applicants, and 2) senior scientists and administrators who voluntarily left NIH since February 3, 2005. Respondent universe and sampling methods are described for each below.

 

1) Two groups of individuals were identified as forming a readily available pool of potential NIH employees: members of grant application review committees and panels who have been active in the past 24 months, and post-doctorate fellows applying for Extramural Post Doctoral Individual National Research Service Awards and Intramural Post Doctoral Individual National Research Service Awards.

NIH is interested in reaching potential intramural scientists and high-level administrators who might be affected by the regulations, and these two groups will produce appropriate respondents. Intramural scientists who come to NIH are most likely to be young researchers beginning their careers, such as post-doctoral award applicants. Active grant application committees and panels will include scientists interested in directing the course of research and guiding NIH policy, and these are the individuals most likely to be interested in becoming NIH administrators.

 

These two populations have been identified using the NIH Information for Management, Planning, Analysis, and Coordination (IMPAC) II database, a central administrative database of individual grant applicants, awardees, trainees, and review committee members. This database provides names and contact information is updated daily. ORC Macro, the data collection contractors, has identified more than 24,000 individuals who fall into these two groups of potential respondents. The target is 400 completed interviews, and the sample will be randomly drawn from the respondent universe.

 

2) The respondent universe of former NIH employees is maintained by NIH. Upon OMB approval, NIH will identify all senior scientists and administrators who have voluntarily left NIH since February 3, 2005 and provide ORC Macro with their contact information. This group is not expected to exceed 100 individuals, and we will conduct a census of those individuals. Therefore, there is no additional sampling or sampling error (estimates of precision) associated with this population of respondents.

 

Survey results will be analyzed and conclusions drawn for each type of group of individuals separately. The following table provides the sample size and the estimates of precision level:

 

Table B1-1. Sample Sizes and Estimates of Precision Levels

	Potential Applicants	Employees that have left NIH since February 3, 2005
Pop size (Nh)	24,277	100
Sample Size (nh)	400	100
+/- at 95% Confidence	4.86%	N/A

B.2. Information Collection Procedures/Limitations of the Study

ORC Macro will implement the telephone interviews and the mail survey according to a protocol designed to maximize response rates, yield accurate and reliable results, minimize response bias, and provide respondents with options for providing the information.

 

Potential NIH job applicants will be contacted by telephone. When contact is made with a target respondent, the respondent will be asked for his/her participation, provided with any requested background or verification of the survey’s purpose, and will be offered several options for completing the survey. If the respondent asks to complete the survey on the phone at that time, the interviewer will proceed with the survey. The respondent can also schedule an appointment to complete the survey on the phone at a later date, either at a specifically requested day and time or on a non-specific day and time in the future. If the person would like to complete the survey using another method, the interviewer will offer to fax, mail, or email the respondent a copy of the survey to complete and return. To minimize interviewer bias, an Interviewer Training Manual that details all protocols to be followed by interviewers during the administration of the survey will be developed prior to the telephone fielding. Prior to the beginning of telephone interviews all interviewing staff, quality assurance staff, and supervisors will be required to attend a training session and will receive both background information on the project and the training manual. The training manual will include all information needed to conduct the interview, including a detailed explanation of all survey items, skip patterns, and probes. Following training and a practice period, interviewers will be monitored during the interview process. Supervisors and quality control managers will have the ability to listen to calls remotely. If, at any time, an interviewer is performing poorly or is not following protocol, the interviewer will be either retrained or removed altogether.

 

Former NIH employees will be contacted by mail and asked to complete a survey. ORC Macro will use a methodology that will include:

 

·         A pre-notification letter

·         A survey package that includes a cover letter, a mail survey, and a postage-paid business reply envelope.

·         A second survey package to non-respondents only.

 

We estimate the total fielding period to be 8 weeks.

 

The cover letter will include a telephone number if the participant has any questions about the survey. If the contact raises questions outside ORC Macro’s purview, ORC Macro will forward them to a designated NIH staff member in accordance with applicable confidentiality protocols.

 

We anticipate the survey population will be highly cooperative and the response rate for this study will exceed 80 percent. This response rate is consistent will many other surveys ORC Macro has conducted with employees.

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

In addition to keeping the surveys brief, providing respondents with different options for completing the surveys, and allowing them to take the survey at their convenience, the survey data collection process has been designed to maximize response rates and minimize respondent burden.

 

The use of CATI technology for the potential job applicant survey allows for telephone data collection that will follow the rigorous calling protocol outlined below.

 

         Number of Call Attempts. Interviewers will make a maximum of 7 attempts to a telephone number to try to contact the selected respondent and complete an interview. These attempts will be rotated over calling periods or until a completed interview, request for a call back or a resolved status is achieved. The attempts will be rotated through weekday day, weekday evening, Saturday daytime, and Sunday afternoon/evening shifts to maximize coverage of the residential population. No number in the sample file will be attempted more than once per shift, except in the cases of busy lines that will be attempted 30 minutes later. Five busy signals in one call shift are considered one attempt on a record.

 

         Callbacks. The CATI system automatically handles callbacks for “no answer,” “busy,” and “answering machine” outcomes. The CATI system also has the ability to accurately and efficiently manage large numbers of scheduled, definite appointments and optimizes queuing for definite callbacks by continuously comparing station sample activity and the index of definite callback records. When a definite appointment arrives, the system locates the next available station and delivers the record as the next call. Specific appointments can be scheduled using the CATI sample management and can be made for anytime the call center is in operation -- daytime, evening, or weekend. The effective handling of callbacks to respondents is always crucial to the success of any telephone survey project and increases the response rate and the coverage of the population.

 

         Call History. The CATI system automatically stores the outcome or disposition of each attempt in the sample management database. This provides a complete call history for each record in the sample. The call history is displayed on the interviewer’s screen during each new attempt.

 

         Privacy Manager Protocol. While privacy managers are relatively uncommon, ORC Macro interviewers will be trained to leave messages on privacy managers, informing respondents of the study. The message states that ORC Macro interviewers are calling on behalf of the client, and their cooperation would be appreciated. ORC Macro has learned that this protocol can improve response rates and more quickly resolve dispositions.

 

         Refusal Aversion and Conversion. Research indicates that most refusals are situational, i.e., the respondent was reached at an inconvenient time. Interviewers will make every attempt to complete the survey at that time or schedule a callback. In an effort not to jeopardize NIH’s relationships with this specific audience, ORC Macro will not be re-contacting any hard refusals.

 

The mail survey of former NIH employees also minimizes the burden on respondents and makes the process for submitting data as simple as possible. Respondents will be notified by mail prior to receiving the survey, the brief survey contains only essential questions and respondents will be provided with a postage-paid business reply envelope. Only those respondents who do not reply to the survey will be sent a second survey package. Respondents will be able to complete the survey at a time and location they find convenient.

B.4. Tests of Procedures or Methods

After obtaining OMB Clearance, a pre-test of the survey instruments will be conducted with up to 25 actual respondents for the telephone survey. During this pre-test, responses will be monitored closely by project staff and quality assurance supervisors. If it is determined that no changes are necessary based on the pretest, these survey responses are maintained as valid responses, but if changes are necessary, modifications to the survey will be made until it is correct. The pilot test will focus on the administration of the surveys, and any modifications that may be made will be to the organization of the survey and not to the content of the questions or the type of data being collected.

B.5. Names and Telephone Numbers of Individuals Consulted

Staff of the data collection contractor, ORC Macro, were consulted on the design of the survey. Larry Luskin (301-572-0334) of ORC Macro was also consulted on statistical aspects of the project. He is a member of the American Association for Public Opinion Research.