Supporting Material 1

NSF Response to Questions and Comments from OMB

NSF Scientific Collections Survey

1. Why is NSF proposing a staggered roll out?  Not explained and is an unusual survey method.  Main problem could be targeting responses within a certain point in time, but use of an “as of” date probably could address.

The language in the NSF Clearance Package referring to the staggered roll-out was left over from preliminary discussions. The survey methodology NSF intends to implement is explained below, in the response to question 2.

Why hasn’t NSF approached its colleagues in SRS or some other survey methodologists for some survey and questionnaire design assistance?  There are a few obvious things missing that could improve this survey.  For example, using the standard “Dillman” protocols involve a pre-notice mailing, a reminder mailing, etc would likely boost response rates.  Also, some of the questionnaire wording is still a bit rough (e.g., question B-1, what is the reference date – the current fiscal year?  As is, is unclear).

NSF consulted with colleagues in SRS and with Abt Associates (under subcontract to IDA Science and Technology Policy Institute) to determine the best means of achieving high response rates, and we will follow their advice on pre-notice e-mailings and follow-up. We have already reached out to the community of potential respondents through lectures at symposia at several national scientific meetings for over a year. The survey has been endorsed by the Natural Science Collections Alliance and by the American Institute of Biological Sciences, whose memberships will be alerted to the questionnaire online.

NSF worked with the Interagency Working Group on Scientific Collections (IWGSC) to ensure parallel wording of the NSF surveys and the IWG Federal Collections survey. We will clarify the reference date as 2007 calendar year and proof the wording of other questions for clarity.

2. Item 4 in part b discusses results of a focus group, but does not discuss procedures or methods for the actual survey administration.   We would like to see this described, covering things like the procedures alluded to in #1 and 2 above.

Procedures for the Collection of Information:

Initial Contact Procedures:

NSF will contact all institutions with an object-based scientific collection that have received National Science Foundation support since 1985. The institution list, extracted from an internal NSF award information database, was vetted by NSF staff to ensure all institutions fit the scope of the aforementioned requirement. NSF verified the contact information included with the institution list about the relevant Administrators and Collections Managers to be correct and current by checking institutional websites. The resulting list includes 147 institutions. By validating contact information in advance, we anticipate more efficient data collection and improved response rates.

The first point-of-contact with the institution will be made in mid-August with an e-mail letter to the appropriate Administrator of the Institution, with a carbon copy sent to the relevant Collection Manager(s). Concurrently, a second and separate e-mail letter will be sent to each Collection Manager, with a carbon copy to the Administrator.

There will be two rounds of follow-up letters to non-respondent Administrators and Collections Managers. The first e-mail follow-up will be sent one month after the initial contact, in mid-September. The second e-mail follow-up will be sent in early October.

Administrator/Collections Manager Induction:

The initial contact e-mails will serve as the formal introduction of the NSF survey to the respondents. Throughout the planning process, NSF has engaged in conversations with representatives of the scientific collections community about NSF survey. These conversations have occurred in relevant NSF panel meetings and at professional society meetings, allowing NSF staff to infuse the respondent community with preliminary information regarding the upcoming survey.

Data Collection:

Abt Associates, which will work with NSF under subcontract to IDA STPI, will create and maintain the website to host the NSF Scientific Collections Survey instrument. Each institution will be sent the hyperlink to access the NSF Scientific Collections Survey instruments in the contact letters. Survey respondents will be able to access the website in mid-August. Based on the focus group feedback, we anticipate the survey will take less than an hour for completion. Responses will be collected online by Abt Associates

Monitoring Data Collection and Quality Control:

Abt Associates is responsible for providing help to Administrators and Collections Managers encountering technical difficulty completing the survey. Answers to commonly asked questions will also appear on the survey instrument’s FAQ page. NSF contacts will be listed on the survey instrument for respondents to address any questions and/or comments about the content of the survey.

Abt Associates will continue to collect data until November and will provide the data to NSF for the report. NSF staff with appropriate scientific expertise will interpret the data. Abt Associates will analyze the processed data in November and December, and will provide NSF a summary draft of tables and figures at the end of December. NSF staff will prepare a summary report of the NSF Scientific Collections Survey and will present it to the IWGSC at the end of February. The IWGSC report will be available publicly in summer 2008.

 

3. What happens to the results, specifically?  Would be useful to see a brief description of the web-based system that is working now for Federal collections (since they will use it for this purpose, as well). 

To clarify – there is no “web-based” data system. OSTP hosted an on-line survey instrument for the IWGSC Survey, the results of which are summarized in an interactive ACCESS database for use by IWGSC members. Web shots of the IWGSC’s electronic survey instrument follow.

NSF will conduct its survey via the web as described above. Survey responses will be transmitted by Abt to NSF electronically in EXCEL file format.

Results of the NSF Scientific Collections Survey will be included as a separate section in the IWGSC Report to the National Science and Technology Council/Committee on Science. The report will also include the results from the IWGSC’s Survey of object-based scientific collections owned by Federal Agencies. The IDA Science and Technology Policy Institute will work with NSF, Abt Associates and the IWGSC to summarize data from the NSF collections survey in a way that is compatible with the IWGSC report.

The IWGSC report will be used by the IWGSC to inform NSTC about the state of federal collections and pertinent issues related to those collections. NSF will be able to use findings from the survey from its surveys in a variety of ways, such as to assess major issues in past investments, to refine future program solicitations, and to guide strategic planning.

4. Related, what types of results (statistically speaking) will be presented?  The description of statistical methods in Part B should include information about the analysis and products envisioned (e.g., plan to run simple descriptive statistics).

Data will be presented only in aggregate form in tables consistent with those used for the Federal survey. We will be conducting only simple description statistics, univariate and cross tabulations. Results will be presented using histograms, tables, and basic, descriptive charts. Results will be summarized using the same kind of table formats being developed by the IWGSC to facilitate comparisons across the data collections. In general, the tables array data for each question in the questionnaire using “rows” that represent responses by collection type or other appropriate descriptor.

As the survey targets the universe of in-scope NSF grantees, no estimation will be made beyond this targeted group. We will look at characteristics of non-respondents (e.g. institution size) to minimize bias and ensure that the respondents do not differ from non-respondents in any systematic way.

5.  NSF needs to include the PRA statement on the questionnaire or somewhere in the questionnaire packet.

We will add the following statement to the cover page of the survey:

Pursuant to 5 CFR 1320.5(b), an agency may not conduct or sponsor, and a person is not required to respond to an information collection unless it displays a valid OMB control number. The OMB control number for this collection is 3145-XXXX. Public reporting burden for this collection of information is estimated to average 40 minutes per response, including the time for reviewing instructions. Send comments regarding this burden estimate and any other aspect of this collection of information, including suggestions for reducing this burden, to: Reports Clearance Officer, Facilities and Operations Branch, Division of Administrative Services, National Science Foundation, Arlington, VA 22230.

Selected web shots of the survey instrument

Figure 1. Homepage of survey instrument with link to “FAQ” and “Question and Answer” pages highlighted.

Figure 2. Screenshot of first page of electronic survey.

5