Supplemental and Part B Questions

A. Supplemental Questions for DOC/NOAA Customer Survey Clearance
(OMB Control Number 0648-0342)
1. Explain who will be conducting this survey. What program office will be
conducting the survey? What services does this program provide? Who are the
customers? How are these services provided to the customer?
The survey will be conducted by an educational evaluator to examine the effects on student
understanding of the concepts in two NOAA WaterLife games, Sea Turtles and the Quest to
Nest and Where the Rivers Met the Sea. The games were developed by the National Oceanic
and Atmospheric Administration's National Ocean Service (NOS) and are based on research
on effective game design for middle school students. Teachers will receive support in using
the Waterlife games from NOS staff. The games are housed on the NOS website,
http://games.noaa.gov/ and support the education mission of NOAA to build environmental
literacy through products and programs that incorporate the applied science of NOAA and
NOS. The target audience for the games website are students in upper elementary through
high school, but teachers are increasingly using the site as classroom supplements. The
teachers will give the survey link to the students at the beginning and end of their estuary or
sea turtle lessons. Teachers will also be surveyed, and this survey is the only one that would
not be considered a regular part of classroom instruction and testing; therefore only the
burden for this survey will be counted; the rest of the information is background only.
2. Explain how this survey was developed. With whom did you consult during the
development of this survey on content? statistics? What suggestions did you get
about improving the survey?
An outside evaluator developed the survey based on the purpose of the games. The game
developers and NOS education coordinator reviewed the survey. Based on feedback, the
language of some of the questions was refined to be even more developmentally appropriate
for students in grades 5-8 and some questions about additional content were added.
3. Explain how the survey will be conducted. How will the customers be sampled (if
fewer than all customers will be surveyed)? What percentage of customers asked to
take the survey will respond? What actions are planned to increase the response
rate? (Web-based surveys are not an acceptable method of sampling a broad
population. Web-based surveys must be limited to services provided by Web.)
Middle school teachers and informal educators will be recruited who teach the
concepts in one of the games as part of their curriculum. Educators will be recruited
until at least 250 student participants for each game are involved, half for the control
and half for the treatment group for each game. This could result in 5-10 teachers,
depending on the number of classes they teach. The educators who participate will
need to teach more than one section of the subject that includes the concepts, so at
least one group/class can learn with the game, and one without.

Teachers will need to complete the unit of instruction during this academic year, have
students take online pre/post concept tests and the post survey, and complete an
online questionnaire themselves or give an interview about how they used the games
in their instruction (20 minutes).
Students will take the following tests/surveys:
•

•
•

#s 1 and 2: Retrospective student attitude survey (attitude toward the subject; only
two questions about the game itself) - all students, but two questions regarding the
game to be answered by the treatment group only - all questions to be answered after
the unit (10 minutes).
#s 3 and 4: Pre/post understanding performance task – all students (10 minutes each
for pre- and post-test)
#s 5 and 6: Pre/post knowledge test consisting of released items 1 and one open-ended
question - all students (10 minutes each for pre- and post-test).

All the students will be surveyed. No identifying information is being collected from
students.
4. Describe how the results of this survey will be analyzed and used. If the customer
population is sampled, what statistical techniques will be used to generalize the
results to the entire customer population? Is this survey intended to measure a
GPRA performance measure? (If so, please include an excerpt from the appropriate
document.)
The outside evaluator will analyze the results using t-tests for gauging change in student
attitudes and knowledge on the quantitative section. Open-ended question responses will be
analyzed for recurring ideas. An evaluation report on the results will be presented to the NOS
staff who will use it in future decision-making regarding curriculum development and to
further knowledge within NOAA and the field on the effectiveness of these games for
learning and the methods for evaluating them.
This survey is not intended to measure a GPRA performance measure.

1

The knowledge test items have been released/retired by state testing agencies from previous years’ tests.
They have been through a rigorous development and testing process so they are valid and reliable
measures of the concepts.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection method to be used. Data on the number of
entities (e.g. establishments, State and local governmental units, households, or
persons) in the universe and the corresponding sample are to be provided in tabular
form. The tabulation must also include expected response rates for the collection as
a whole. If the collection has been conducted before, provide the actual response.

Game
Where Rivers Meet the
Sea
Quest to the Nest

#Institutions
6

#Students
300

Sample
100%

Expected response rate
100%

6

300

100%

100%

In addition, a maximum of 10 teachers will complete an on-line questionnaire or complete an
interview with the same question, by telephone, at the end of the unit. All teachers are
expected to respond, with a total burden of 3.4 hours.
2. Describe the procedures for the collection, including: the statistical methodology for
stratification and sample selection; the estimation procedure; the degree of accuracy
needed for the purpose described in the justification; any unusual problems requiring
specialized sampling procedures; and any use of periodic (less frequent than annual)
data collection cycles to reduce burden.
Educators will have both control and treatment group students log in to take the online tests.
Teachers are agreeing to involve all their students in the data collection, so no sample
selection will be done. The data will be collected during the same time period as the unit is
taught, to reduce the burden. Teachers will answer questions about their own participation at
the end of the unit.
3. Describe the methods used to maximize response rates and to deal with nonresponse.
The accuracy and reliability of the information collected must be shown to be adequate
for the intended uses. For collections based on sampling, a special justification must be
provided if they will not yield "reliable" data that can be generalized to the universe
studied.
Teachers will give the pretests to the entire class, then conduct the activities over a period of
days or weeks depending on the chosen curriculum and then have the students complete the
posttests and survey on the last day. The tests and activities will be done during the same
time period with the control and treatment groups over the unit’s duration. Some students
may be absent the day of the survey. Teachers will be encouraged to have absent students
make up the tests and survey if appropriate, so that very few, if any, students are expected not
to respond. A response rate close to 100% is expected for teachers, given their interest and
stake in the project, but those not responding within a set length of time will be re-contacted
and reminded. The choice of online or telephone interviews is expected to make responses
less burdensome.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged
as effective means to refine collections, but if ten or more test respondents are involved
OMB must give prior approval.
Locating and using the online student survey will be discussed with the teachers, and then
their understanding will be checked by having them complete the survey. Student
understanding of the concepts will be measured by comparing the post test scores of the two
groups using regression analysis and pre/post gains within each group using Wilcoxon Ttests for matched pairs. Pretest scores will be used to determine the comparability of the
groups. Student perceptions of learning with the game as reported on the end of experience
survey will be compared by group and achievement level for both qualitative and quantitative
data. The teacher responses will be used to augment data gathered from the student surveys
and assist in future use of these games/planning for revised or different games.
5. Provide the name and telephone number of individuals consulted on the statistical
aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or
other person(s) who will actually collect and/or analyze the information for the agency.
Hilarie B. Davis, Ed. D. Evaluator for TLC Inc. 75 Sauga Ave, North Kingstown, RI 02852
401-965-5843.