supporting statement, statistical methods

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

SRS will collect data from the survey in two ways. First, the survey will be available to SRS web customers via a link from the SRS home page. Any SRS customer who comes through the SRS home page has the opportunity to complete the survey. Second, SRS will send an email invitation to a sample of 1,000 subscribers to its GovDelivery email notification service. SRS currently has approximately 13,500 GovDelivery subscribers. GovDelivery is a system that allows citizens to select the types of information they want to receive. While there is likely overlap between GovDelivery subscribers and those who come to the SRS home page, SRS will not try to unduplicate the two sets of data users.

Average response rates for self-selected surveys are exceedingly low. SRS receives approximately 2300¹ hits per day on its website. Assuming a ½ of 1% response rate over the 9 week period, SRS could expect about 700 (roughly 2300*7*9*.005) survey responses. It is possible that those answering the survey through the home page link could be either highly satisfied or dissatisfied with their experience. These data will provide some useful insights into what SRS might be doing well, where SRS might be lacking in its web presentation, who is visiting the site, and what types of products and formats they are looking for on their visit.

Average response rates for customer satisfaction surveys are generally between 10 to 15%. Assuming a 10% response rate, SRS could expect about 100 responses to the one-time email request to GovDelivery responders. Data from these responders will help SRS gain a better understanding of who receives notifications about our data and what types of products and formats they are interested in.

Data from the two different samples will not be combined.

All data will be used for internal purposes only. SRS is beginning a review of our dissemination activities. The data being collected under this activity are one aspect of that review and will be used to help focus our efforts on our web delivery portion.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Simple random sampling will be used to select the sample of GovDelivery subscribers. Initially, SRS will select two replicate samples of size 1000 each. The first replicate of 1000 will be fielded. Midway through the data collection, approximately 4 weeks, SRS will evaluate the response rate. SRS may field the second replicate if the response rate is exceeding low, below 5%.

Since the data will be used for internal purposes only as a guide for future evaluation efforts, SRS will be looking at the percent of respondents in each response category. Future efforts will likely focus on the top 2 or 3 response categories.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

In order to minimize burden, no non-response follow-up will be conducted. The data will not be used to generate any statistics outside of NSF and will be used for internal purposes only for additional evaluation efforts. The data will be used for exploratory purposes and will help focus further evaluation on the top 2 or 3 response categories and will not be used to compute averages. Any bias due to non-response will likely be from those that are either highly satisfied or dissatisfied with their experiences which will provide SRS with some useful insights into what SRS might be doing well or where SRS might be lacking in its web presentation.

A sample of 700 site users will yield better than 5% precision with 95% confidence for a percentage of p=.5 which is more than sufficient for exploratory purposes. A sample of 100 GovDelivery subscribers will not be as precise and will yield approximately 5% precision with 90% confidence for a percentage of p = .1. The two samples will not be combined. The sample of GovDelivery subscribers will be used to provide a rough profile of those receiving notices and to determine if additional resources should be allocated for further exploration of this set of users.

3. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The survey will be pretested with individuals in the division and will be modified as needed. SRS has recently hired several new staff who are ideal for testing the survey.

3. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Jeri Mulrow will provide oversight on the sample selection and computation of percentages.

Jeri Mulrow, Senior Mathematical Statistician, Division of Science Resources Statistics. 703 292 4784, [email protected].

1 During the week of 9/6/2010 to 9/12/2010, SRS had an average of 2,316 sessions per day. Sessions ranged from a low of 1,483 on Sat to a high of 2,903 on Weds.