CIRCUMSTANCES MAKING COLLECTION OF INFORMATION NECESSARY
On September 11, 1993, President Clinton issued Executive Order 12862, “Setting Customer Service Standards,” which clearly defined his vision that the Federal agencies will put the public first. To accomplish this, President Clinton called for a “revolution within the Federal government to change the way it does business.” He expected this process to require continual reform of government practices and operations to the end that, “when dealing with the Federal agencies, all people receive service that matches or exceeds the best service available in the private sector.”
Section 1(b) of this E.O. requires agencies to “survey customers to determine the kind and quality of services they want and their level of satisfaction with existing services” and Section 1(a) requires agencies to “survey front- line employees on barriers to, and ideas for, matching the best in business.” These Presidential requirements established an ongoing need for the National Science Foundation (NSF) to engage in an interactive process of collecting information and using it to improve program services and processes.
Consistent with E.O. 12862, the purpose of the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research and Reviewers is to explore whether the National Science Foundation effectively supports unsolicited mid-scale research. The Task Force is interested in gathering and analyzing input from the NSF-funded customer community (university researchers and administrators) to determine sources of satisfaction. The survey will collect information that will be used to identify the obstacles to NSF’s customers regarding developing, submitting, reviewing, and performing mid-scale research, and the changes, if any, that NSF could consider to improve satisfaction.
HOW, BY WHOM, AND PURPOSE FOR WHICH INFORMATION IS TO BE USED
The National Science Board (NSB) Task Force on Unsolicited Mid-Scale Research is charged with exploring whether the National Science Foundation effectively supports unsolicited mid-scale research. The Task Force is interested in gathering and analyzing input from the NSF-funded customer community (university researchers and administrators) to determine sources of satisfaction. The following are our objectives for this survey.
To identify best practices that would increase customer satisfaction with NSF’s mid-scale research funding mechanisms
To measure customer satisfaction with their ability to find mid-scale research opportunities at NSF that meet their research needs and promote optimal scientific progress
To measure customer satisfaction regarding NSF’s support of mid-scale research infrastructure
To measure customer satisfaction with the balance between prescription and flexibility for NSF funding opportunities
To measure customer satisfaction with NSF’s communication of current mid-scale research funding opportunities
Responses from the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research will be used by the NSB Task Force in conjunction with several other information-gathering strategies to assess what changes, if any, are needed in NSF’s current funding mechanisms and communication with the mid-scale research customer community.
USE OF AUTOMATION
There are no legal or technical obstacles to the use of technology in these information collection activities. The NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research will be administered via the Internet, which will allow for a more convenient and less costly survey administration than a paper survey.
EFFORTS TO IDENTIFY DUPLICATION
The information to be collected by the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research does not duplicate any other information collection. After consulting with various senior NSF officials, members of the NSB Task Force on Unsolicited Mid-Scale Research have concluded that the information sought in this survey, which is critical to the Task Force’s efforts, is not already available.
SMALL BUSINESS CONSIDERATIONS
Not applicable.
CONSEQUENCES OF LESS FREQUENT COLLECTION
Not applicable.
SPECIAL CIRCUMSTANCES FOR COLLECTION
Not applicable.
FEDERAL REGISTER NOTICE
The agency’s notice, as required by 5 CFR 1320.8(d), was published in the Federal Register on January 12, 2011, at 76 FR 2151 and no substantive comments were received.
OUTSIDE CONSULTATION
NSF staff (National Science Board Office; NSBO) is responsible for the design, administration, and analysis of the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research. In the course of this work, the Science and Technology Policy Institute (STPI; contractor) will assist in survey data collection and analysis.
GIFTS OR REMUNERATION
Not applicable.
CONFIDENTIALITY PROVIDED TO RESPONDENTS
The NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research will be a confidential survey. The instructions to the survey will clearly assure participants that individual responses will not be linked to the respondents’ identities. From the survey: “Your identity is not linked to your responses. The National Science Board Office, as the survey administrator, will maintain the confidentiality of your responses.” Any survey data provided to anyone outside of the NSB Office will be purged of information that could be used to identify individual responses.
QUESTIONS OF A SENSITIVE NATURE
No questions of a sensitive nature will be asked.
ESTIMATE OF BURDEN
Each respondent will submit only one survey response. There are two general types of respondents: 1) Researchers, and 2) University Administrators. The “researcher” group is comprised of NSF-funded research faculty at research universities. The “university administrator” group is comprised of vice presidents for research (or an equivalent position) at universities receiving NSF-funding. It is anticipated that the average response time for researchers will be twenty (20) minutes; the average response time for administrators will be ten (10) minutes. This estimate is based on the survey length, feedback from NSF staff and external NSF-funded researchers who reviewed and tested the survey, and time required to complete similar surveys in the past. NSBO estimates the number of research responses to the survey to be 158 (88 universities x 3 researchers each x 60% response rate; see table below and section B.1 for more detailed information). The total hours of burden for the researcher group is estimated to be 53 hours (158 researchers x 20 minutes / 60 minutes). NSBO estimates the number of university administrator responses to the survey to be 75 for an annual burden of 12.5 hours (75 administrators x 10 minutes / 60 minutes; see table below and section B.1 for more detailed information). The total burden for the survey is estimated to be 65.5 hours (53 hours for researchers + 12.5 hours for university administrators). The estimated aggregate for both groups (233 total) of 25 minutes per response comes to 97 burden hours.
COST TO RESPONDENTS
In April 2011, The Chronicle of Higher Education published a table of average faculty salaries by field and rank at doctoral institutions for the academic year of 2010-2011. The data were collected from the “Annual survey by the American Association of University Professors.” The salaries are rounded to the nearest $100 and adjusted to a nine-month work year.
Cost to Respondents
Average salary of faculty (researchers) as described above |
$92,468 |
Hourly salary based on 1,560 annual hours (40 hours per week for 39 weeks) |
$59.29 |
Estimate of survey burden (researchers) |
53 hours |
Cost to researcher respondents |
$3,142.37 |
Average salary of university administrators as described above |
$197,304 |
Hourly salary based on 1,560 annual hours (40 hours per week for 39 weeks) |
$126.48 |
Estimate of survey burden (university administrators) |
12.5 hours |
Cost to university administrator respondents |
$1,581 |
Total Cost to Respondents |
$4,723.37 |
CAPITAL/STARTUP COSTS
Not applicable.
TOTAL COST TO THE FEDERAL GOVERNMENT
The table below estimates the cost to the government associated NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research. Costs include contractor support and the participation of federal government employees. Federal employee hourly rate was calculated from OPM’s Table NO 2011-ES for salaries effective January 2011. The average of the SES for Agencies with a Certified SES Performance Appraisal System was used. The total cost is estimated at $47,200.
Cost to the Federal Government
Contractor support for survey data collection and analysis |
$40,000 |
Hourly salary of federal government employee (SES Level IV) |
$72 |
Hours, federal government employee review and oversight |
100 |
Cost of federal government employee review and oversight |
$7,200 |
Cost to the Federal Government |
$47,200 |
CHANGES IN BURDEN
There are no changes in burden. This proposed collection fits within the limits of our generic clearance.
18. PUBLICATION OF COLLECTION
A summary of the survey results will be available on the NSF Web site (www.nsf.gov) in spring 2011.
19. SEEKING APPROVAL TO NOT DISPLAY OMB EXPIRATION DATE
Not applicable.
20. EXCEPTION(S) TO THE CERTIFICATION STATEMENT (19) ON OMB 83-I
There are no exceptions.
STATISTICAL METHODS
B.1. Universe and Sampling Procedures
The NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research respondents fall into two general categories: university researchers and university administrators. The university researcher pool was identified by soliciting via email faculty nominees from the vice presidents/provosts of research at 125 NSF-funded universities. The Task Force queried the top 100 universities in terms of NSF funding for 2009 and 2010. Since the top 100 NSF-funded universities for 2009 and 2010 were nearly identical, the total number of top 100 NSF-funded universities for 2009 and 2010 was 110. To increase diversity in the respondent pool, the Task Force also solicited nominees from 15 NSF-funded Minority Serving Institutions (MSIs). Therefore, all together, faculty nominees were solicited from a total of 125 NSF-funded universities. The Task Force requested that the university administration of each of these 125 universities provide the names of “creative, articulate researchers who either have previously conducted mid-scale research at NSF, or would like to do so in the future.”
We anticipate a 70% response rate from these 125 universities (125 x 70% = 88 universities) and an average of three nominees per university, yielding a total university researcher respondent sample pool of 264 (i.e., 88 universities x 3 nominees). For any university that has submitted the names of more than five nominees, we will randomly pick only five to take the survey so that no single university is overrepresented in the sample pool. Otherwise, we plan to distribute the survey to all of the nominees received by the Task Force. Since the respondents were nominated by universities receiving substantial NSF funding, we anticipate at least a 60% response rate for the survey. Therefore, the total number of university researcher respondents is anticipated to be 158 (264 x 60%). Since the contact information for the nominees was provided by their respective universities, we expect this information to be current and correct.
The university administrator respondent pool includes 125 vice presidents/provosts (or their equivalent) at the top 100 NSF-funded universities in 2009 and 2010 (110 total) and the vice presidents/provosts (or their equivalent) from 15 NSF-funded Minority Serving Institutions (15 total). This represents a total of 125 individuals for our university administrator sample pool. Because these individuals are responsible for the research faculty at NSF-funded universities, they will be able to provide a broad perspective on customer satisfaction with NSF-funded mid-scale research at their institution. We anticipate a 60% rate for the survey for this second group of respondents based on the topic and interest the NSBO has received from the academic community regarding this topic. The total sample pool for our survey is 389 (264 researchers + 125 university administrators). The total estimated respondents for the survey is 233 [(264 researchers x 60%) + (125 university administrators x 60%)].
The survey’s administration will include efforts to minimize the primary sources and effects of non-response bias. For purposes of this survey, the primary sources of non-response bias include:
(1) surveys not being delivered to their intended parties;
(2) unwillingness to participate in the survey; and
(3) respondents abandoning the survey before completing all responses.
Each of these areas will be addressed below.
To minimize the potential for undelivered surveys, the survey team will utilize and correct, as possible, information on members of the target population. To minimize undelivered surveys, the survey team will closely monitor the delivery status of all surveys (the online administration of the survey automates such monitoring). For surveys that are not delivered, the survey team will make all reasonable efforts to identify and the correct information required for survey delivery (such as checking for typographical errors, verifying proper email syntax, or consulting academic institution Web sites), and will use this information to re-send the survey to the appropriate parties.
To minimize unwillingness to participate in the survey, the survey team will utilize strategic communications, a respondent-friendly online interface and format, and follow-up reminders (as described in greater detail below in section B.3. Methods to Maximize Response). These efforts will also address the potential tendency of respondents to complete only part of the survey.
As stated above, the target population for the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research consists of 389 individuals (264 researchers + 125 university administrators). With an expected survey response rate of 60 percent, the anticipated number of completed surveys is 233 [(264 researchers x 60%) + (125 university administrators x 60%)].
B.2. Survey Methodology
The NSF seeks to address the following research questions:
To identify best practices that would increase customer satisfaction with NSF’s mid-scale research funding mechanisms
To measure customer satisfaction with their ability to find mid-scale research opportunities at NSF that meet their research needs and promote optimal scientific progress
To measure customer satisfaction regarding NSF’s support of mid-scale research infrastructure
To measure customer satisfaction with the balance between prescription and flexibility for NSF funding opportunities
To measure customer satisfaction with NSF’s communication of current mid-scale research funding opportunities
The survey methodology was designed to address these goals through the following steps:
Identify the survey objectives. Confirm objectives for the survey, review technical and practical assumptions, and identify internal and external stakeholders to engage in the survey development process.
Design the survey. Create and test a survey that will address the five research questions asked by the NSB Task Force and NSF.
Administer the survey. Develop and implement a communications strategy that will maximize the survey response rate. Administer the survey between December 19, 2011 – January 6, 2012.
Analyze survey results. Conduct analyses to summarize the data and identify patterns, trends, and significant relationships.
Report survey results. Provide an executive briefing to the NSB and NSF senior managers on the high-level survey findings and their implications.
Prepare and deliver a final report. Incorporate changes and suggestions from the NSB and NSF senior managers into the survey findings and deliver the final report to both the NSB and NSF.
This submission to OMB includes the survey instrument developed in accordance with the process just described. The intent of the survey instrument is to address the goals established by the NSB listed above. To accomplish this, several iterations of the instrument were developed and vetted with NSB Members, NSF management and staff, expert contractors (STPI), and internal (NSF) and external (NSF-funded researchers) testers.
The NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research will be distributed via email to each individual in the target population. The email will include an explanation of the purpose of the survey, assure respondents their responses will not be attributable and their identities will remain confidential, and describe where respondents may obtain survey results once the analysis of results is complete.
B.3. Methods to Maximize Response
To maximize the response rate, the administration of the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research will include a letter from the NSB Task Force on Unsolicited Mid-Scale Research Co-Chairmen sent to the survey sample a week prior to the survey distribution. This letter (which will be sent by email) will provide credibility to the survey effort, will explain how the survey will benefit the community (by allowing them to provide feedback so that NSF can continue to improve the proposal/research experience), and will encourage recipients to complete the survey.
The online format and layout of the survey instrument are other factors that will help maximize the survey response rate. Through pilot testing and iterative feedback from contract and NSF staff and external NSF-funded researchers, the survey instrument was tested and refined regarding its clarity and usability. Feedback from reviewers indicated that the survey will likely require approximately 20 minutes and 10 minutes to complete for researchers and university administrators, respectively. This length of time is well within the parameters for avoiding “questionnaire fatigue” and maintaining respondent interest.
The subject of the survey itself will enhance its response rate. The NSB and NSF are aware of its customers’ interest in NSF-funded mid-scale research. The presumption of the NSB and NSF, based on anecdotal evidence, has been that survey participants are concerned about the this area of research and will welcome an opportunity to share information on their views and experiences on this important issue.
There will be at least one reminder sent to survey recipients who have not yet responded. These reminders will be emailed only to non-respondents. The reminders will take place a few days following a drop-off and leveling from the response spike that typically follows a survey distribution or reminder. These reminders will also help to maximize the survey response rate.
While there will be no tangible reward or incentive to motivate participation in the survey, survey recipients will be informed of how they will be given access to survey results (the survey results will be posted on the NSF Web site after completion and submission of the final analysis of survey response data). This assurance is intended to engender respondent trust and confidence; by participating in the survey, respondents will have a stake in the survey’s outcome and will have access to the final results. Providing this assurance will contribute to maximizing the survey response rate.
B.4. Testing of Procedures
The development of the NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research included several processes to test the survey instrument and its administration.
To test the survey questionnaire iterative rounds of feedback were obtained from NSF staff, contract staff (STPI), and four external researchers representative of our target population. Through this feedback, successive refinements to the survey questionnaire were made and its clarity, ease of completion and technical focus were enhanced.
After the initial survey distribution, it is expected that some email addresses will “bounce” and be returned because they are incorrect or no longer exist. Reasonable efforts, such as contacting the respondents academic institution (the nominating university), will be made to correct any email addresses that are found to be incorrect or changed.
B.5. Contacts for Statistical Aspects of Data Collection
Contact for statistical aspects of data collection is Matthew Wilson ([email protected]).
Attachments
NSB 2011 Survey of Customer Satisfaction for NSF-Funded Mid-Scale Research. The online version of the survey can be found at the following URL: https://www.surveymonkey.com/s/NSB_MS_Survey
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | JFELDMAN |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |