Supporting Statement B for
GENERIC CLEARANCE FOR SURVEYS
OF CUSTOMERS AND PARTNERS OF
THE OFFICE OF EXTRAMURAL RESEARCH
OF THE
NATIONAL INSTITUTES OF HEALTH
Date: 11/15/2010
Name: Sally Rockey, Ph.D.
Deputy Director for Extramural Research, NIH
Address: 1 Center Drive, Rm 144, Bethesda MD 20892
Telephone: 301-496-1096
Fax: 301-402-3469
Email: [email protected]
Table of Contents
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 1
B.1 RESPONDENT UNIVERSE AND SAMPLING METHODS 1
B.2 PROCEDURES FOR THE COLLECTION OF INFORMATION 3
B.3 METHODS TO MAXIMIZE RESPONSE RATES AND DEAL WITH NON-RESPONSE 4
B.4 TEST OF PROCEDURES OR METHODS TO BE UNDERTAKEN 4
B.5 INDIVIDUALS CONSULTED ON STATISTICAL ASPECTS AND INDIVIDUALS COLLECTING AND/OR ANALYZING DATA 4
The respondent universe will be separately identified for each initiative whose customers and/or partners are to be surveyed. The methods chosen for use will depend on the nature of the policies, programs, operations and services being examined, as well as their associated stakeholder communities. Recommended methodologies and sample sizes will be based on a review of the relevant literature and consultation with experts in customer satisfaction surveys. The customer satisfaction surveys will use methods and approaches tailored to the analytical need, but are anticipated to be quantitative surveys (e.g. self-administered surveys with measurable results), qualitative surveys (e.g. focus groups), or a combination of qualitative and quantitative surveys (e.g. self-administered surveys that include both measurable outputs and open-ended narrative sections). Quantitative and qualitative surveys will be designed to minimize burden on respondents while obtaining essential information. The expectation is that information collection instruments will require no more than 15 minutes response time for quantitative surveys and 1 hour for qualitative surveys.
Quantitative survey methods target particular audiences with statistical sampling procedures employed to identify potential survey respondents. Email, mail, telephone, and Internet surveys will seek a purposive or convenience sample, depending on the statistical objectives of the customer satisfaction survey. In qualitative studies, quota sampling is often used to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not representative of the population. In qualitative studies, all respondents are initially contacted by telephone, mail or email.
In virtually all instances, there will be existing lists of respondents readily available for sampling (e.g., extramural scientists, applicants and awardees, scientific administrators and institutional officials, members of professional scientific organizations, etc). Appropriate probability sampling techniques will be used to select samples. Respondents are expected to be a mix of adult scientists connected to the research community, adult science trainees and the general public. The expected response rate for these surveys is 80 percent due to the strong interest of the scientific community and the public in health and health research.
ESTIMATED ANNUAL HOURS OF BURDEN
Quantitative survey, e.g. email/mail survey measuring customer satisfaction (see Table B1-1.a., below):
9,820 respondents = total number of respondents per survey
15 minutes = average time per response to each email/mail surveys
1 response = annual frequency of response per survey
2,455 hours = total hours of burden per quantitative survey
24,550 hours = total hours of burden per year based on 10 quantitative surveys
Qualitative survey, e.g. focus group on customer satisfaction (see Table B1-1.b., below):
30 respondents = total number of focus group members per survey
1 hour = average time per focus group
1 response = annual frequency of response per survey
30 hours = total hours of burden per qualitative survey
120 hours = total hours of burden per year based on 4 qualitative surveys
Based on an estimated 10 quantitative and 4 qualitative surveys per year:
24,670 hours = total annual hours of burden in each of the 3 years
74,010 hours = total combined hours of burden to respondents for 2011, 2012, and 2013
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Response (in hours) |
Annual Hours of Burden Per Survey |
Science professionals – applicants, reviewers, Institutional Officials |
3,820 |
1 |
0.25 |
955 |
Adult science trainees |
2,000 |
1 |
0.25 |
500 |
General public |
4,000 |
1 |
0.25 |
1,000 |
Totals |
9,820 |
1 |
0.25 |
2,455 |
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Response (in hours) |
Annual Hours of Burden Per Survey |
Science professionals – applicants, reviewers, Institutional Officials |
12 |
1 |
1 |
12 |
Adult science trainees |
6 |
1 |
1 |
6 |
General public |
12 |
1 |
1 |
12 |
Totals |
30 |
1 |
1 |
30 |
A list of potential respondents and their contact information will be developed for each topic being evaluated. The NIH eRA Commons is an online interface where grant applicants, grantees and federal staff can access and share certain administrative information relating to research grants. It
includes an extensive roster of current and former applicants, reviewers, and Institutional Officials, as well as adult science trainees. Surveys of the general public will rely on professional survey organizations to provide a representative sample to gauge public opinion. All NIH-developed rosters of respondents will be checked for duplications; all contact information will be checked and verified. When the rosters are finalized, OER will contact each intended respondent (e.g. email a letter) and explain the purpose and importance of the survey, and ask each one to participate. The initial contact will also inform each intended respondent of the process for participating in the survey (e.g. a web address for a survey, mailing instructions for a written survey, or focus group participation instructions).
Self-Administered Surveys. Self-administered surveys can be used to combine quantitative evaluations of experiences with qualitative, narrative input on customer satisfaction. Surveys can be distributed in person, emailed, mailed, or accessed on-line. Such surveys will be appropriate in a variety of instances, but especially those cases where customer satisfaction needs to be measured quantitatively while also providing the opportunity to collect unanticipated feedback on customer experiences. When a self-administered survey will be accessed on-line, the the invitation letter will contain the unique access code and password needed to gain entry to the survey. No access to any other survey instrument except for the one assigned to the respondent will be allowed. The survey will be submitted electronically by clicking a submission button. With submission, the password will be automatically deactivated. No re-access to the site will be allowed for security reasons.
Focus Groups. Focus groups can be used to obtain insights into shared experiences related to NIH policies, programs and procedures. Focus groups are usually composed of 8-10 people who have characteristics similar to the target audience, or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. Focus groups are valuable in exploring shared experiences and stakeholder consensus.
Professionally recognized procedures will be followed in each information collection activity to ensure high quality data. All data collection will be conducted in a manner that is consistent with the following principles:
o Appropriate sample sizes will be determined for each activity to assure that burden is minimized while reliable estimates are produced.
o Participation will be fully voluntary, and non-participation will have no impact on eligibility for or receipt of future services. If necessary, steps will be taken to ensure unbiased completion of questionnaires by use of third-party distribution and receipt by a party not directly involved in provision of the service being assessed.
o Information to be collected will be limited to that needed to assess customer satisfaction. Repeated implementation of quantitative surveys will be at an interval appropriate to measure the impact of changes and to monitor ongoing levels of satisfaction.
o Efforts will be made to obtain the highest possible response rates, given the voluntary nature of the data collection efforts. To the extent feasible, efforts will be made to assess non-response bias.
o Appropriate monitoring and recording of interactions with respondents will take place, and data from self-administered surveys will be double-checked.
o All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.
Consistent with sound survey methodology, the design of each survey will include approaches to maximize response rates, while retaining the voluntary nature of the effort. For Web based surveys, for example, this is expected to include a second emailing of the invitation to participate, a reminder email, and possibly some telephone follow-up, if phone numbers are available. As appropriate, some qualitative surveys (e.g. focus group sessions) may use teleconferencing to limit burden. On other occasions, regional group meetings sponsored by NIH/OER will have ready access to respondents.
Before each customer satisfaction survey is implemented, pre-testing will be carried out at a level and in a manner consistent with the specific survey. All mail and Web based surveys are expected to include pre-testing with a small number of potential respondents with telephone debriefing of pretest respondents as needed to clarify responses. Lessons from the pre-test will be identified, and changes as necessary will be incorporated into the instrument and method. All pilot tests will involve no more than nine individuals unless OMB clearance is sought for more than nine.
Each survey team will obtain input from statisticians in the development, design, conduct and analysis of customer satisfaction surveys. This statistical expertise will be available from agency evaluators or from contractors.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement B for |
| Author | Andrea Kopstein |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |