Supplemental and Part B Questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB CONTROL NO. 0648--0342)

Reef Smart Participant Satisfaction Survey--NOAA Reef Smart Seminar Series

Biogeography Branch

Center for Coastal Monitoring and Assessment

NOAA National Centers for Coastal Ocean Science

Supplemental Questions for DOC/NOAA Customer Survey Clearance

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

What program office will be conducting the survey?

This survey will be conducted by staff at the Biogeography Branch within the Center for Coastal Monitoring and Assessment (CCMA) of the National Centers for Coastal Ocean Science (NCCOS), National Ocean Service (NOS). The survey will be used to evaluate participant satisfaction with outreach events designed to inform natural resource managers about the status and findings of scientific research being conducted by NOAA’s Coral Reef Conservation Program (CRCP) in the seven U.S. states and jurisdictions containing coral reefs. This outreach initiative is called “Reef Smart.”

What services does this program provide?

The mission of the CCMA Biogeography Branch is to develop information and analytical products through research, monitoring and assessment of the distribution and ecology of living marine resources and their habitats. The Biogeography Branch routinely collects and consolidates spatial and other physical and biological data that are used to produce a variety of biogeographic maps, tools and products. These services are provided to other program offices within NOAA, who are our primary customers.

NOAA CRCP is a cross-Line Office Program that brings together expertise from multiple scientific disciplines in order to increase understanding of coral reef ecosystems, and improve their management. In partnership with NOAA offices and local jurisdictions, the Program seeks to improve the health of coral reefs by mitigating threats and restoring coral ecosystems. The program accomplishes these goals by providing funds for research and management activities, as well as by disseminating information and providing forecasts. The newly launched Reef Smart initiative is a vehicle whereby information generated by CRCP-funded research can be disseminated to key stakeholders within the seven coral reef jurisdictions.

The idea for Reef Smart originated with outreach events that were hosted in 2011. In 2012, the Reef Smart Initiative was formally inaugurated in the CRCP coral reef jurisdiction of Puerto Rico. The initiative is composed of three key events: 1) “Day at Sea”—geared toward policy-makers and their staff; 2) “NOAA Reef Smart Seminar Series”—directed toward natural resource managers working in focal coral reef ecosystems; 3) “Get Reef Smart!”---targeted at middle school children.

The NOAA Reef Smart Seminar Series is a core component of the Reef Smart program. It is during this activity that we propose use of the present survey in the future. In other words, we plan to administer this survey during each “NOAA Reef Smart Seminar Series” event hosted by the Reef Smart project managers in the future. At this time, only 1 NOAA Reef Smart Seminar Series is scheduled for FY2013, which will include a total of 4 seminars over the course of several months.

Who are the customers?

The Reef Smart Initiative aims to provide information about CRCP-funded coral reef ecosystem research to a variety of federal, territorial and local audiences. Primary target audiences include policymakers, natural resource managers and teachers/educators. The customers that will be surveyed with the proposed survey are natural resource managers who participate in the NOAA Reef Smart Seminar Series event.

How are these services provided to the customer?

Reef Smart’s central goal is to improve awareness of CRCP activities and other coral conservation initiatives. Reef Smart’s primary service is the hosting of science forums that combine presentations/demonstrations by scientists about research activities/findings with informational seminars for/from local resource managers about management issues and activities. The schedule of Reef Smart events, generally, is:

Day 1: NOAA Reef Smart Seminar Series---Natural resource managers and other guests hear overviews of the science being conducted related to coral reef ecosystems. This includes briefings on recent research findings and management success stories, as well as the descriptions of the tools used to collect data. Finally, attendees are given time to talk with NOAA researchers about opportunities for collaborative research.

Day 2: Reef Smart Day at Sea—This is a non-technical, interactive event geared toward policymakers. This event will highlight the impact of NOAA coral reef research, identify collaborative research success stories, and provide policymakers with a chance to ask questions about coral reefs, research, etc. This event will take place aboard a NOAA research vessel and will include a short time at sea where equipment demonstrations will occur.

Day 3: Get Reef Smart!—This will be an education program for K-12 students where select students and teachers will come aboard the docked vessel for a rotation of short, reef-centric classes. Each class will focus on a particular topic and will be age appropriate.

A participant satisfaction survey was developed, cleared and implemented in FY2012 for the Reef Smart Day at Sea event. This clearance package is in support of a new survey developed to evaluate participant satisfaction with the NOAA Reef Smart Seminar Series event.

Attendees to the NOAA Reef Smart Seminar Series event will include natural resource managers at the federal, territorial, state, provincial, county, or municipal level. Additionally, attendees could include natural resource managers from businesses or private, non-government organizations. Participants will be asked to voluntarily complete the participant satisfaction survey. The questionnaire will be administered in-person to all invited attendees present at the event. Responses from the questionnaire will be analyzed and findings will be used to improve future outreach activities.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

Explain how this survey was developed.

Staff developed this survey instrument in accordance with the Generic Clearance for Customer Surveys provided by the NOAA Office of the Chief Information Officer, in consultation with individuals of the Biogeography Branch and CRCP who will be responsible for planning and executing the Reef Smart Initiative. Survey design and layout was informed by scholarship published in peer reviewed journals, as well as in non-peer reviewed, “gray” literature on outreach/education best practices and effective methods for evaluating public outreach events (Fedler 2001; Kelly 2004). Literature on the development of effective survey instruments (de Leeuw 2008; Dillman et al 2009) was also used. Additionally, staff consulted literature related to the assessment of respondent ‘awareness’ both generally and in relation to ocean issues (Ciochetto and Haley 1994; Steel et al 2005a; Steel et al 2005b). Additionally, this survey was developed in a manner to ensure appropriate consistency with the Reef Smart “Day at Sea” participant satisfaction survey cleared in FY2011.

With whom did you consult during the development of this survey on content? Statistics?

During development of this survey, we consulted with program staff from the NOAA Coral Reef Conservation Program, the NOAA NOS Biogeography Branch and the NOAA NMFS Coral Reef Ecosystem Division. A draft version of the survey was provided to five people, two with expertise in social research, to provide comments and suggestions. Reviewers were asked to take the survey (i.e., read each item and select a response) and provide comments on a) how long it took them to complete the survey, b) any items that were confusing or unclear and why and c) any other observations or comments that they would like to share. Consultation with a statistician was not undertaken because only descriptive statistics will be used to analyze the data.

What suggestions did you get about improving the survey?

Suggestions from persons who reviewed/tested the survey instrument included:

• General Comment—Can a question be added on the prioritization of research in the jurisdiction?

  • Action: The Coral program presently has a system for soliciting research topics and prioritizing research for the program. Addition of such a question to the present survey would be duplicative of those processes, and beyond the scope of the Reef Smart initiative, ergo the survey.

• Question 2—Add “job” to the assessment item: “The information provided at today’s event was highly relevant to my agency or organization.”

  • Action: Upon consideration of the reviewer’s comments, we decided that substituting “job” for “agency or organization” would actually be preferable. This question follows more closely the flow of the other questions in this section, in that they pertain to the individual’s experience as opposed to the individual’s assessment about their agency or organization. The question was changed to “The information provided at today’s event was highly relevant to my job.”

• Question 2—Consider adding a “Don’t know” response option.

  • Action: No change made. We deemed inclusion of the “no opinion” response to be adequate.

• Questions 3 & 4—The boxes provided are too restrictive, small to write detailed answers. Consider removing the boxes and just leaving a blank space.

  • Action: We removed the boxes to create a larger blank space for respondents to utilize. We rephrased the question accordingly.

• Question 5—What is the difference between “management” and “administration”? Are both words required? Is the word “recovery” appropriate or is this rather a task that is best included under management? Should the word “restoration” be added to the list of organization activities, as a complement to “recovery” because recovery and restoration are different activities?

  • Action: The phrase was changed to:”…natural resource management, recovery/restoration or administration.” We contend that an agency or organization may have an administrative focus or mission without necessarily having a management focus or mission. For example, an agency may administer contracts for a management-oriented activity, while not performing the management activity itself. Similarly, upon consideration, we contend that an organization may have a focus or mission related to restoration/recovery, while not having jurisdiction over the direct management of natural resources. We agreed with the reviewer regarding inclusion of “restoration.”

• Questions 5 & 6---If more space is needed on the survey, consider re-formatting the response items for these questions so that they are laid out horizontally, as opposed to vertically in a column format.

  • Action: To maintain format and flow of the survey questionnaire we opted not to reformat response items in a horizontal manner. With removal of boxes for Questions 3 & 4, the survey appears to be adequately spaced.

References:

Ciochetto, Susan and Barbara A. Haley. 1995. How Do You Measure "Awareness"? Experiences with the Lead-Based Paint Survey. Proceedings of the Section on Survey Research Methods, Alexandria, VA: American Statistical Association, pp. 1163-1168. Available online at: http://www.census.gov/srd/papers/pdf/sm9501.pdf

de Leeuw, Edith. 2008. Self-Administered Questionnaires and Standardized Interviews. Chapter 18 in The Sage Handbook of Social Research Methods, P. Alasuutari, L. Bickman and J. Brannen, Eds. Los Angeles: Sage.

Dillman, Don A., Jolene D. Smyth and Leah Melani Christian. 2009. Internet, Mail and Mix Mode Surveys: the Tailor Design Method, Third Edition. Hoboken, N.J.: Wiley.

Fedler, Anthony J., Ed. 2001. Defining Best Practices in Boating, Fishing, and Stewardship Education. Prepared for the Recreational Boating and Fishing Foundation under Contract #RBFF-00-C-004 by Human Dimensions Consulting, Gainesville, FL.

Kelly, Leah F. Evaluation of Public Participation and Outreach Methods for the North Carolina Coastal Habitat Protection Plan. Thesis. Duke University, 2004. Available online at: http://dukespace.lib.duke.edu/dspace/handle/10161/244

Steel, Brent, Court Smith, Laura Opsommer, Sara Curiel and Ryan Warner-Steel. 2005a. Public Ocean Literacy in the United States. Ocean & Coastal Management 48: 97-114.

Steel, Brent, Nicholas Lovrich, Denise Lach, and Valentina Fomenko. 2005b. Correlates and Consequences of Public Knowledge Concerning Ocean Fisheries and Management. Coastal Management 33: 37-51.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Explain how the survey will be conducted.

Reef Smart staff persons will hand the survey to attendees as they enter the seminar room/auditorium. During the welcome and introductory comments, the seminar host will inform the attendees about the survey, its purpose, importance, and instructions for completion. At the conclusion of the seminar, attendees will be reminded to complete the survey that was provided to them as they entered the room, as well as instructed on how to return the survey to Reef Smart staff. The survey will be self-administered at the venue and returned by participants as they exit the event.

How will the customers be sampled (if fewer than all customers will be surveyed)?

No sampling will be undertaken. All invited participants to NOAA Reef Smart Seminar Series events will be asked to complete the survey. This could include anywhere from 20 to 70 persons per seminar.

What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate?

We anticipate a 100% response rate. The survey has been designed to make its completion as easy and straightforward as possible for participants. Additionally, we will implement several strategies to ensure a 100% response rate. We will distribute the survey as attendees enter the venue, which will increase efficiency and reduce the inconvenience to attendees (i.e., not having to wait at the end of the event for the survey to be distributed). A Reef Smart staff person will introduce the survey at the outset and remind attendees to complete their surveys at the conclusion of the event. We will allocate a period of 10 minutes just prior to the conclusion of the event for participants to complete their surveys. We anticipate that participants will be able to complete the survey in 5 minutes. Finally, we will distribute pens with the surveys so that attendees will have a writing implement readily available to complete the survey. All of these strategies have been devised to ensure a 100% response rate.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Describe how the results of this survey will be analyzed and used.

The data from this survey will be analyzed using the software package SPSS. Data from this survey will be analyzed using only descriptive statistics (e.g., frequency, percent, means, etc.). The purpose of this survey is to provide information to those responsible for planning and executing the Reef Smart Initiative on the level of satisfaction that participants report after having attended a NOAA Reef Smart Seminar event. Information from the survey will be used to assess the success of specific NOAA Reef Smart Seminar Series events, as well as the initiative as a whole in terms of participant satisfaction. The information will be used to improve future events by informing adjustments in content, planning and execution.

If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population?

We do not intend to employ a statistical sampling methodology for this collection. The universe of possible respondents, meaning all invited seminar attendees, will be surveyed. We do not plan to analyze the data collected using inferential statistics.

Is this survey intended to measure a GPRA performance measure?

The present survey is not intended to measure a GPRA performance measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.

Invited ‘NOAA Reef Smart Seminar Series’ participants will be natural resource managers, or their designated representatives, who are from or represent management organizations or agencies in jurisdictions that include, or are in proximity to, focal coral reef ecosystems. A ‘natural resource manager’ includes, but is not limited to, any person, or agent of such person, who has responsibility for promulgating, administering or implementing rules, regulations or management plans, or substantial input into such activities, for any natural resource or spatially designated management unit relevant to coral reef inputs, resources, habitats, or ecosystems. Natural resource managers could be from any level of government, including federal, territorial, tribal, state, provincial, county, parish, city, municipal, or township. Natural resource managers could also be employed in the private/business sector or be a member/employee of a non-government organization. Attendees to the Reef Smart Seminar Series might also include policymakers, scientists, academic researchers, students, or members of the general public whose positions are not strictly defined as “resource management”, if such persons are engaged in activities that are deemed relevant to natural resource management by the managers of the Reef Smart Initiative and their local collaborators.

Invitees to the NOAA Reef Smart Seminar Series event will be purposively selected by the managers of the Reef Smart Initiative and their local collaborators. The purpose of this event is to inform natural resource managers about research activities in the region. Thus, criteria for selection of invitees will not be scientific, but purposeful and intentional. Selection will be based on the professional judgment of program managers and recommendations from their associates, based on knowledge of and familiarity with natural resource managers in the jurisdiction. However, invitees will meet the following criteria:

• Invitees will hold a professional position having responsibility for promulgating, administering or implementing rules, regulations or management plans for any natural resource or spatially designated management unit relevant to coral reef inputs, resources, habitats, or ecosystems in the jurisdiction.

• Invitee will hold a professional position where he or she provides substantial input into the promulgation, administration, or implementation of rules, regulations or management plans for any natural resource or spatially designated management unit relevant to coral reef inputs, resources, habitats, or ecosystems in the jurisdiction.

Because the goal of the survey is to assess our participants’ experience related to engaging in a Reef Smart event, we define our universe of possible respondents as any person who attends a NOAA Reef Smart Seminar Series seminar, as opposed to any natural resource manager in the jurisdiction. Only individuals who attend NOAA Reef Smart Seminar Series will be asked to respond to the survey. The number of natural resource managers attending any one NOAA Reef Smart Seminar Series event will vary, but will never exceed 70 individuals per seminar. At this time, only 1 NOAA Reef Smart Seminar Series is scheduled for FY2013, which will include a total of 4 seminars over the course of several months. However, NOAA Reef Smart Seminar Series events could be planned for any of the 7 coral reef jurisdictions, as defined by the CRCP program, where NOAA research is underway. Seminar events could include anywhere from 1 to 4 seminar sessions per event.

Because the topic of each seminar will be different, each seminar will be treated as a unique event. As such, we will survey each seminar audience for their feedback. It is possible that a manager could be invited to and attend more than one seminar event. However, given the unique nature of each seminar, information collected on questions 1 through 4 will be valid and useful for assessing the value and satisfaction related to the individual seminars, regardless of whether a person has attended more than one seminar. Nevertheless, given the possibility that a person may attend more than one seminar, ergo may be surveyed multiple times, we understand that we may use data from questions 5 through 7 only to assess the individual seminars or compare results for each seminar to each other. In other words, we may not produce and report on summary statistics for these items across the entire seminar series per jurisdiction (e.g., we cannot provide the total number of “City or town government” attendees across the 4 seminars) because these totals may be inflated due to repeat attenders.

Universe of Survey Respondents—Maximum attendees at any given seminar: 70 x 4 = 280	Expected Response Rate
100%	100%

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

We do not intend to employ a statistical sampling methodology for this collection. The universe of possible respondents will be surveyed. We do not plan to analyze the data collected using inferential statistics.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Describe the methods used to maximize response rates and to deal with non-response.

The survey has been designed to make its completion as easy and straightforward as possible for participants. Additionally, we will implement several strategies to ensure a 100% response rate. We will distribute the survey as attendees enter the venue, which will increase efficiency and reduce the inconvenience to attendees (i.e., not having to wait at the end of the event for the survey to be distributed). A Reef Smart staff person will introduce the survey at the outset and remind attendees to complete their surveys at the conclusion of the seminar. We will allocate a period of 10 minutes just prior to the conclusion of the event for participants to complete their surveys. We anticipate that participants will be able to complete the survey in 5 minutes. Finally, we will distribute ink pens with the surveys so that attendees will have a writing implement readily available to complete the survey. All of these strategies have been devised to ensure a 100% response rate. The universe of possible respondents will be surveyed. We do not plan to analyze the data collected using inferential statistics.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

A draft version of the survey was provided to five people, including two social scientists. These five people were asked to take the survey (i.e., read each item and select a response) and provide comments on a) how long it took them to complete the survey, b) any items that were confusing or unclear and why and c) any other observations or comments that they would like to share. Suggestions from persons who tested the survey instrument generally focused on the wording of several questions, as well as the format and layout of the survey. The survey instrument was revised based on comments from these individuals.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

No statistical consultation was obtained for this survey. We do not intend to employ a statistical sampling methodology for this collection, nor do we plan to analyze the data collected using inferential statistics. Data collection will be implemented by Ms. Alicia Clarke, with the Biogeography Branch, Center for Coastal Monitoring and Assessment, National Centers for Coastal Ocean Science, National Ocean Service. Ms. Clarke may be reached by email at [email protected] or telephone by 301-713-3028 x189. Data management and analysis will be completed by Dr. Theresa L Goedeke, Social Scientist, with the Biogeography Branch, Center for Coastal Monitoring and Assessment, National Centers for Coastal Ocean Science, National Ocean Service. She may be reached by email at [email protected] or by telephone at 301-713-3028 x 237.

9