NOAA SEA evaluation supplemental and Part B questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB Control Number 0648-0342, expiration date: 4/20/2015)

National Ocean Service/Coastal Services Center - Training Evaluation

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The National Oceanic and Atmospheric Administration (NOAA) has been developing the Hawai`i Education Program, an initiative to support NOAA’s efforts to develop marine science education programs for Hawaii’s schools. The Hawai`i Education Program is an initiative created to provide the local community with needed educational resources based upon science, technology, engineering, and math (STEM) topics that align with Hawai`i Department of Education benchmarks and national education standards. These resources adapt existing science, data and technology at NOAA into cohesive, grade-level sets of science instruction materials applicable to Hawai`i and throughout the Pacific region. The NOAA team of education professionals, partners and experts developed a series of educational resources, lessons and trainings entitled NOAA Sea Earth Atmosphere (SEA) that are an integral part of the Hawaii Education Program.

This information collection will be conducted by staff at the National Oceanic and Atmospheric Administration (NOAA) Pacific Services Center (PSC) to assess realignment of the NOAA Sea Earth Atmosphere (SEA) curricular resources to some combination of the Common Core Standards or Next Generation Science Standards (NGSS) in addition to what types of products and services should be offered to educators. Sacheen Tavares-Leighton (Program Training Specialist) and Chris Ellis (Social Scientist) will guide this information collection along with John Mitchell (Education Specialist).

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

This survey instrument was developed by reviewing the list of cleared questions for the Generic Clearance, OMB Control No. 0648-0342, and through consultation from Coastal Services Center staff with expertise in the areas of survey research design and evaluation. A list of draft questions was discussed with staff members responsible for and involved in the Hawai`i education program. Suggestions for survey improvement included keeping the list of questions as short as possible and making sure to keep questions focused on the main objective of the evaluation, which is realignment.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The survey will be created using the SurveyMonkey software program and administered via an emailed link, which will direct teachers who have participated in past NOAA SEA teacher workshops and other partners who have used the resource materials, to the SurveyMonkey website. Because this is a call for feedback on how to realign the materials in the most efficient manner, all of these participants should have the opportunity to respond. The number of responses and survey content will be recorded. It will be possible to calculate an actual response rate for reporting purposes. The number of questions will be limited to mostly multiple choice responses in an effort to achieve a higher response rate relative to more lengthy surveys, and based on the program’s informational needs. The total estimated sample size is approximately 200. Based on similar efforts, we anticipate a 50% return rate.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of survey data will be undertaken through basic descriptive statistics only (e.g., percent, mean scores). This information collection seeks to assess how to realign the resource materials and identify additional products and services that should be offered to educators. The respondent universe will include all past NOAA SEA teacher workshop participants and partners who have used the curricular resources. Data from this information collection will not be aggregated with other data for GPRA reporting.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Potential respondents being solicited for the proposed information collectors are formal and informal educators in the State of Hawaii, which are made up of State government employees (HI Department of Education), Private sector (private school and charter school educators), and non-profit/NGOs (informal educators). Following is the breakdown of trainees of interest.

	Total Universe	Estimated Responses based on 50% response rate
State Government	160	80
Non-profit/NGO	10	5
Private Sector	30	15
Total	200	100

This survey will be distributed via email, and will inquire on awareness, use, and application of the SEA curriculum. The estimated time necessary for each respondent to complete the questionnaire is 15 minutes, based on trials with a small (fewer than ten) pilot sample. Total estimated public burden associated with this information collection is 25 hours (100 respondents X 15 minutes). The computer program will keep track of the total number of successful responses.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The survey will be created using SurveyMonkey and administered via an emailed link, which will direct past SEA educators (curricula users over the past 24 months) to the SurveyMonkey website.

Completed surveys received via Survey Monkey will be downloaded to a password protected work space at the Coastal Services Center, only accessible by staff particular to this project. Respondents will be instructed not to provide identifying information on the survey (names, social security numbers, dates of birth, etc.), and any identifying information placed on surveys will be removed.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The intent of this information collection is to assess educator feedback on the utility of SEA education materials in order to inform improvements to the curricula. In order to improve response rates for this information collection, the survey has been made as brief as possible. Nonresponse testing will be a challenge in that no identifying information will be collected that will allow for follow-up activities. Approximately 200 formal and informal educators have received training on the SEA curriculum. All past trainees will be solicited, which represent five Hawaiian islands and a variety of sectors (state government, private sector, and non-profit/NOGs) The intended approach is highly likely to yield a representative sample of the respondent universe, and the information gained will be extremely valuable in making course improvements. This survey allows equal and independent opportunity for all users to provide feedback on SEA education resources.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Draft versions of this survey were circulated for review and comment to nine former SEA curriculum users. Reviewers were asked to offer feedback on the length, appropriateness and clarity of questions, content, or other aspects to improve the questionnaire. Comments from reviewers were helpful and resulted in design, and content changes to clarify questions and simplify instructions.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The implementation of the information collection and data analysis will be completed by Dr. Chris Ellis at the NOAA Coastal Services Center, available by telephone at (843) 740-1195 or by email at [email protected].