Justification--Colorado CS Survey

U.S. Department of the Interior

Office of Policy Analysis

Guidelines for Accessing the Department of the Interior’s Generic Clearance for Customer Satisfaction Surveys

Revised February 2009

ATTACHMENT 1:

Instructions for Completing Supporting Statement for DOI Generic Clearance Submission, OMB Approval Number 1040-0001

36944. Survey Title/Date Submitted to the Office of Policy Analysis (PPA): Insert title for the proposed survey. Insert date that the expedited approval package will be submitted to PPA. Reminder: Please submit the package through your bureau/office Information Collection Clearance Officer.

36945. Bureau/Office: Insert the name of the bureau/office conducting the survey.

36946. Abstract: Summarize the proposed study with an abstract not to exceed 150 words.

36947. Bureau/Office Point of Contact Information: Complete the bureau/office contact information. PPA will communicate with the point of contact listed here throughout the entire approval process.

36948. Principal Investigator (PI) Conducting the Survey: Complete information about the PI who will be conducting the survey, if different than Point of Contact listed in #4. Otherwise note: Same as #4.

36949. Name of Program Office Conducting Survey: Provide the name of the bureau program, office, or organizational unit conducting the survey.

36950. Description of Customers/Services Provided: Provide a brief description of the customers who will be surveyed, the services provided by the program conducting the survey, and how these services are provided to customers.

36951. Survey Dates: List the time period in which the survey will be conducted, including specific starting and ending dates. The starting date should be at least 45 days after the submission date. The request for expedited approval, and submission of a complete and accurate approval package, must be made at least 45 calendar days prior to the first day the PI wishes to administer the survey instrument to the public.

36952. Type of Information Collection Instrument: Check the type(s) of information collection instrument(s) that will be used. If other, please explain.

36953. Survey Development: Explain how the survey was developed. With whom did you consult during the development of the survey on content? On statistics? Did you pretest the survey? What actions did you take to improve the survey? What suggestions did you receive for improving the survey? Which of the six topic areas will be addressed? (Note: A description of any pre-testing and peer review of the methods and/or instrument is highly recommended.)

36954. Survey Methodology: Explain how the survey will be conducted. Provide a description of the survey methodology including: (a) How will the customers be sampled? (if fewer than all customers will be surveyed); (b) What percentage of customers asked to take the survey will respond, and (c) What actions are planned to increase the response rate? If statistics are generated, this description must be specific and include each of the following:

- The respondent universe,

- The sampling plan and all sampling procedures, including how individual respondents will be selected;

- How the instrument will be administered;

- Expected response rate and confidence levels; and

- Strategies for dealing with potential non-response bias.

Note: Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by the web.

12. Total Number of Initial Contacts/Expected Number of Respondents: Provide an estimated total number of initial contacts and the total number of expected respondents.

13. Estimated Time to Complete Initial Contact/Instrument: Estimate the time to complete the initial contact and the survey instrument (in minutes).

14. Total Burden Hours: Provide the total number of burden hours. The total burden hours should account for the amount of time required to instruct the respondents in completing the survey, and the amount of time required for the respondent to complete the survey.

15. Reporting Plan: Provide a brief description of the reporting plan for the data being collected. A copy of all survey reports must be archived with PPA. Please note this in the reporting plan.

16. Justification, Purpose and Use: Provide a brief justification for the survey, its purpose, goals, and utility to managers. Specifically, describe how data will be tabulated and what statistical techniques will be used to generalize the results to the entire customer population. Describe how data from the survey will be used. Describe how you will acknowledge any limitations related to the data, particularly in cases where we obtain a lower than anticipated response rate. Note whether or not the survey is intended to measure a Government Performance and Results Act (GPRA) performance measure.

ATTACHMENT 2:

Approval Form for DOI Programmatic Clearance for Customer Satisfaction Surveys (OMB Control Number 1040-0001)

U.S. Department of the Interior Office of Policy Analysis (PPA)

PPA Tracking Number: (for PPA use only) CSS-2

			Date Submitted to PPA:	September 29, 2009
1.	Survey Title:	USGS Colorado Plateau Conference Satisfaction Survey
2.	Bureau:	U.S. Geological Survey Southwest Biological Science Center

3.

Abstract: (not to exceed 150 words) The US Geological Survey (USGS) Southwest Biological Science Center (SBSC) sponsors two natural area conferences that aim to promote research that can be easily integrated into resource management actions. To justify the continued sponsorship of these conferences, USGS would like to conduct post-conference evaluations to measure the satisfaction of conference attendees. We will survey attendees of three USGS-sponsored natural area conferences: (1) the 2008 Colorado River Basin Science and Management Symposium, the 2009 and 2011 Colorado Plateau Biennial Research Conferences. The survey will evaluate the following: perception and satisfaction of various aspects of the conference, e.g. physical features, professional development and networking opportunities, and social aspects of the conference; and socio-demographic and professional make-up of conference attendees. Results will be used to provide feedback to the USGS concerning the utility and keys to the future success of natural area conferences.

4.

Bureau/Office Point of Contact Information

First Name:

Charles

Last Name:

van Riper III

Title:

Bureau/Office:

USGS/SBSC/ Sonoran Desert Research Station

Street Address:

125 Biological Sciences East

City:

Tucson

State:

AZ

Zip code:

85721

Phone:

(520)626-7027

Fax:

(520)670-5001

Email:

[email protected]

5.

Principal Investigator (PI) Information

First Name:

Marty

Last Name:

Lee

Title:

Professor of Wildland Recreation

Bureau/Office:

Northern Arizona University, School of Forestry

Address:

Box 1501

City:

Flagstaff,

State:

AZ

Zip code:

86011

Phone:

(928) 523-6644

Fax:

(928) 523-1080

Email:

[email protected]

6.

Name of Program or Office Conducting Survey:

U.S. Geological Survey, Southwest Biological Science Center, Sonoran Desert Research Station

7.

Description of Customers/ Services Provided:

The customers are attendees of three conferences sponsored by U.S. Geological Survey (USGS). The attendees will represent: state, local, and county government, non-profit agencies, natural resource researchers, and public land managers The services provided include: the pre-conference information, content, logistics, products, and outcomes of each USGS-sponsored conference. Services such as information about the program and registration are provided prior to the conference. Conference content, logistics, socializing opportunities, food, etc. are provided during the conference. Follow-up contact with and among attendees may occur following the conference.

8.

Survey Dates

(mm/dd/yyyy)

to

(mm/dd/yyyy)

1/31/2010

1/31/2012

9.

Type of Information Collection Instrument (Check ALL that Apply)

__Intercept

__Telephone

__Mail

_X_Web-based

Focus Groups

__Comment Cards

__Other

Explain:

10. Survey Development:

The impetus for this collection came from organizers of the 2007 Colorado Plateau Biennial Research Conference and the 2008 Colorado River Basin Science and Management Symposium. The organizers wanted input from the conference attendees concerning their satisfaction of the conferences to date. Two focus groups were organized to discuss the need for and the utility of a conference evaluation. The focus groups consisted of 5 to 7 conference attendees, researchers, and managers. The purpose of the focus groups was to develop a set of questions needed to create a conference satisfaction survey. During each focus group session, participants were asked the following question: “What are the attributes of an excellent conference?” Responses to this question were used during the development of the questions for this collection. We also relied on questions used in other conference evaluation studies reported in the literature.

The questions concerning opportunities for networking and professional development as products of the conferences were considered to be important conference outcomes based on the results of the focus group discussions. We developed these questions in consultation with colleagues conducting similar research in this area as well as reviewing the current literature on the topic.

Names and contact information of individuals consulted with in developing the questions, including appropriate statistical analysis:

Carena Van Riper

Dept. of Recreation, Park & Tourism Sciences

Texas A&M University

2261 TAMU

College Station, TX 77843-2261

Denver Hospodarsky

Associate Professor

School of Forestry

Northern Arizona University

Flagstaff, AZ 86011

(928) 523-7525

[email protected]

The survey was pre-tested, by 5 individuals representing two of the target audiences of the Colorado Plateau conferences (three graduate students and two natural resource managers). In response to their feedback we reworded three questions in the survey (Questions 1, 3, and 8)

Topic areas addressed in the survey:

1. Delivery, quality and value of products, information, and services. The conferences are “products” and “services” provided by USGS to conference attendees. Most of the questions included in the survey fall under this topic area.

2. Interactions with DOI Personnel and Contractors. Asking conference attendees about the organization, structure, and delivery of the conferences relates to DOI personnel who serve as conference organizers, planners, and facilitators.

3. General demographics. Several demographic questions are included to provide a profile of conference attendees.

11. Survey Methodology: (Use as much space as needed; if necessary include additional explanation on separate page).
Respondent Universe		The respondent universe for this collection will be the attendees of three USGS-sponsored conferences: (1) the Colorado River Basin Science and Management Symposium held November 2008, (2) the 10th Colorado Plateau Biennial Research Conference held in October 2009, and (3) the 11th Colorado Plateau Biennial Research Conference held in October 2011. Although we do not know exactly how many people will attend the October 2009 and 2011 Colorado Plateau Conferences, we are estimating, based on previous conference numbers, that approximately 550 non-federal employees will attend each conference (totaling 1100 attendees).
Sampling Plan/Procedure		Conference attendees are required to provide an e-mail address along with other contact information when they register for the conference. We will use these e-mail addresses to survey all conference attendees after the conferences are completed. We will use the conference registration list from the 2008 Colorado River Basin Symposium to conduct the first evaluation. The attendees at this conference were informed during registration that they would receive a post conference evaluation and to expect it in the future.
Instrument Administration		This information collection will be 100% electronic. We will use Survey Monkey™ to serve, collect and store information collected during this evaluation process. After each conference we will send an initial e-mail to all conference attendees. This e-mail will describe the survey and ask for their voluntary cooperation. At that time, the attendee will be given instructions and the Survey Monkey™ URL. Following Dillman’s Tailored Design Method (2007), we will send two follow-up reminder e-mails at weekly intervals following the initial contact. Copies of the initial contact e-mail, the survey instrument, and the follow-up reminder e-mails are attached.
Expected Response Rate Confidence Levels		Participants at the 2008 Colorado River Basin Science and Management Symposium were told, via a general announcement at the conference, of the survey and to expect it in the coming months. We will insert fliers in the registration packets for the October 2009 and October 2011 Colorado Plateau Biennial Research Conferences stating that the survey will be conducted within the coming month. We expect these pre-announcement of the survey will encourage participation and increase our response rate. Additionally, we will adhere to up-to-date follow-up procedures for web surveys outlined in Dillman (2007) in designing the initial contact e-mail, the questionnaire, and the two follow-up reminder e-mails. As a result, we anticipate a 70% response rate.
Strategies for dealing with potential non-response bias		We hope to minimize non-response bias by doing all we can to maximize the number of surveys returned (described above) to reach or exceed an anticipated 70 percent response rate. We will use Dillman’s (2007) methods for web-based surveys to maximize the response rate. Conference attendees will be told to anticipate being asked via e-mail to participate in the survey. Individuals will be contacted in an e-mail explaining the purpose of the survey and directing them to the internet URL to the survey. Two follow-up reminder e-mails will be sent. We will test for non-response bias by sending a shortened version of the survey to a sample of non-respondents via e-mail. The survey will include the following questions taken from the general survey: overall satisfaction with the conference, most positive experience (question 4), most negative experience (question 5), and current employer (question 14). We will compare responses to these four questions with the responses from the respondents who completed the general survey. Any underrepresentation found in this analysis will be reported in reports and articles produced.
Description of any pre-testing and peer review of the methods and/or instrument (recommended)		The questionnaire was pre-tested by three graduate students and two resource managers. These represent two of the target audiences of the Colorado Plateau conferences. We incorporated their suggestions, edits, and comments in the final survey. The respondents also agreed with our estimated burden time of 15 minutes.
12.	Total Number of Initial Contacts/ Expected Number of Respondents		Initial Contacts: 1650 Expected Number of Respondents: 1155 (assuming a 70% response rate) Expected number of respondents to 4 question survey: 75
13.	Estimated Time to Complete Initial Contact/ Instrument (mins.):		5 minutes for initial and follow-up contacts (113 hours) 15 minutes for full questionnaire (289 hours) 5 minutes for 4 question non-response bias test (6 hours)
14.	Total Burden Hours:		428 hours
15. Reporting Plan: Copies of the final report will be given to Colorado Plateau Research Conference coordinator offices and to the Office of Policy Analysis. A manuscript based on results of the survey will be submitted for publication in a peer-reviewed journal.

16. Justification, Purpose, and Use:

Survey Justification and Purpose, Survey Goals

Natural area conferences sponsored by the US Geological Survey Southwest Biological Science Center (SBSC) were established to promote communication between researchers and land resource managers and to present relevant research that can be easily and quickly integrated into resource management actions. Because the USGS currently lacks data on the overall effectiveness of the SBSC-sponsored conferences, we will use this evaluation to assess attendees’ satisfaction with various aspects of the three conferences. The objectives of the USGS Colorado Plateau Conference Satisfaction Evaluation are to:

1. Document attendee satisfaction with each conference.

2. Describe the socio-demographic and professional make-up of conference attendees; and

3. Estimate the satisfaction with the networking and social aspects of each conference

We will use this evaluation to measure conference attendees’ satisfaction with the conference including their desires and expectations regarding conference quality, services, and outcomes. Our main goal is to determine how well the conference performed. The results will enable future conference organizers to direct their conference planning and delivery efforts to those areas of greatest importance to conference attendees. Results will also provide quantitative data on attendee satisfaction against which conference organizers can compare their objectives for the conference, to verify whether or not objectives are being met.

Importance-Performance Analysis (IPA) method will be used to help us evaluate conference attendees’ satisfaction with the services provided during the conference, their preferences for physical, content, social elements or features of the conferences as well as evaluating how well the conference performed on these elements. Importance and performance scores will be measured using a 5 point Likert scale. The results will be plotted on a two-dimensional matrix. The importance scores will be plotted on the vertical axis and the performance scores plotted on the horizontal axis. This method allows for all points to fall into one of four quadrants. The upper left-hand quadrant will include the attributes customers (i.e., conference attendees) perceived as being important but not offered at the desired performance level and will be labeled “Concentrate here.” The lower left quadrant shows attributes that are of less importance and were of average performance and will be labeled “Low priority.” The upper right quadrant will include important attributes that had high performance and will be labeled “Keep up the good work”. The lower right quadrant will show attributes that are of less importance but showed high performance, labeled “Possible overkill.” This analysis will be given to the service providers (i.e., conference organizers) to help them understand which areas of the conference may require more attention to planning and the areas that may not deserve concentrated efforts.

Utility to Managers

Results of this analysis will be very useful to future conference planners in making decisions about conference logistics, content planning and resource allocation. Managers and scientists who attend future conferences will benefit from the results of this research because conference planners will be able to design future conferences based on the quantifiably expressed desires and wants of conference attendees.

These results will provide SBSC and other natural area conference organizers with quantitative knowledge that can ultimately enhance the experience for conference attendees and inform the supporting institutions and/or agencies of their successes and where improvements are needed.

How will the results of the survey be analyzed and used?

The data collected during this study will be coded directly into a computerized database. The data will be analyzed using SPSS®. Data analysis will include several phases. The first will consist of frequency distributions of responses to each question. These will be reported as percentages for each of the conference results. We will use factor analysis using principal component extraction and varimax rotation to reduce the importance/performance variables into a smaller group of factors which will then be plotted across two lines of measurement—importance along the x-axis and performance along the y-axis. Questions regarding attendee networking and building professionalism as outcomes of the conference will be analyzed using factor analysis to explore the relationships among these variables. Cross-tabulation and analysis of variance will be used to compare responses between attendees to the three conferences.

How will the data be tabulated? What Statistical Techniques will be used to generalize the results to the entire customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated response rate, how will you address this when reporting the results? (Use as much space as needed; if necessary include additional explanation on separate page).

What Statistical Techniques will be used to generalize the results to the entire customer population?

We will be surveying the entire population of conference attendees (i.e., we are doing a census survey). We will not be drawing a sample of attendees.

Limitations: We will ask attendees of the Colorado River Basin Science and Management Symposium held in November 2008 to remember a conference they attended a year ago. We expect some individuals may be unable to recall specific details of the conference. We will watch for indications of this problem (e.g., excessive “don’t know” responses to survey questions) and we will describe such limitations of the data in any written and oral presentations of the research results.

Non Response Bias Check: We will test for non-response bias by sending a short survey to a sample of non-respondents via e-mail. The questions will include:

• overall satisfaction with the conference,

• most positive experience (question 4),

• most negative experience (question 5), and

• current employer (question 14).

We will compare responses to those questions with those of respondents who completed the entire survey to test for differences between respondents and non respondents. Any significant differences found in this comparative analysis will be reported in all reports and articles produced as a result of this collection.

Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If so, please include an excerpt from the appropriate document. (Use as much space as needed; if necessary include additional explanation on separate page).

All USGS evaluation activities, including this SBSC-related study, are included under GPRA Section 4. GPRA explicitly reinforces the use of program evaluations to obtain objective measures of program results. This assessment serves to evaluate the performance of this program in order to provide quantifiable and measurable explanations for observed performance outcomes. Performance data provide useful and valuable information to program managers to improve program administration. Evaluations, frequently taking a broader and/or more in-depth approach to program structure and results, yield information that can lead to program improvement strategies and might address administrative changes based on methodologically sound grounds.

ATTACHMENT 3

Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys

X All questions in the survey instrument are within the scope of one of the DOI Programmatic Clearance for Customer Satisfaction Surveys topic areas.

X The approval package is being submitted to the Office of Policy Analysis at least 45 days prior to the first day the PI wishes to administer the survey to the public.

X A qualified statistician has reviewed and approved your request.

X Your bureau/office Information Collection Clearance Officer has reviewed and approved the approval package.

The approval package includes:

X A completed Information Form

X A signed Certification Form

X A copy of the survey instrument

X Other supporting materials, such as:

• Cover letters to accompany mail-back questionnaires

• Introductory scripts for initial contact of respondents

• Necessary Paperwork Reduction Act compliance language

• Follow-up letters/reminders sent to respondents

The survey methodology presented on the Information Form includes a specific description of:

X The respondent universe

X The sampling plan and all sampling procedures, including how respondents will be selected

X How the instrument will be administered

X Expected response rate and confidence levels

X Strategies for dealing with potential non-response bias

X A description of any pre-testing and peer review of the methods and/or the instrument is highly recommended.

X The burden hours reported on the Information Form include the number of burden hours associated with the initial contact of all individuals in the sample (i.e., including refusals), if applicable, and the number of burden hours associated with individuals expected to complete the survey instrument.

X The package is properly formatted (Word) and submitted to the Office of Policy Analysis electronically.

ATTACHMENT 4

CERTIFICATION FORM FOR SUBMISSION UNDER OMB CONTROL NUMBER 1040-0001

This form should only be used if you are submitting a collection of information for approval under the DOI Programmatic Clearance for Customer Satisfaction Surveys.

If the collection does not satisfy the requirements of the Programmatic Clearance, you should follow the regular PRA clearance procedures described in 5 CFR 1320.

Bureau/Office Subgroup or Program Charles van Riper III
Title (Please be specific) USGS Southwest Biological Science Center Sonoran Desert Research Station, Leader
Burden Hour Estimates	Estimate Number of Respondents	Initial Contacts: 1650 Expected Number of Respondents: 1155 Expected number of respondents to 4 question survey: 75
	Hours/Min per Response	5 minutes for initial and follow-up contacts (113 hours) 15 minutes for full questionnaire (289 hours) 5 minutes for 4 question non-response bias test (6 hours)
	Total Burden Hours	428 hours
Bureau/Office Contact (who can best answer questions about content of the submission): USGS/SBSC/ Sonoran Desert Research Station
Name	Charles van Riper III		Phone	(520)626-7027
Certification: The collection of information requested by this submission meets the requirements of OMB control number 1040-0001
Bureau/Office Qualified Statistician Dr. Yeon-Su Kim, Associate Professor, School of Forestry, Northern Arizona University, Flagstaff, AZ					DATE 11 September 2009
Bureau/Office Information Collection Clearance Officer Phadrea D. Ponds					DATE 29 September 2009
Office of Policy Analysis Donald J. Bieniewicz					DATE 26 January 2010
OMB, Office of Information and Regulatory Affairs (OIRA)					DATE

1