COTF_genclear_supporting statement DDR 2007 Feb13

COTF_genclear_supporting statement DDR 2007 Feb13.doc

Generic Clearance for Studies to Assess Learning and Flow in Video Games

OMB: 2700-0128

Document [doc]

Download: doc | pdf

The NASA-sponsored Classroom of the Future’s Study of Learning in Games

During fiscal year (FY) 07, FY08, and FY09, the NASA-sponsored Classroom of the Future (COTF) at the Center for Educational Technologies^® (CET) at Wheeling Jesuit University in Wheeling, WV, will study how much people learn (assessment of learning in games) and how people feel (flow) while playing computer games. COTF is planning a number of data collections to inform the study of learning and flow.

A. Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

NASA’s founding legislation, the Space Act of 1958, directs the agency to expand human knowledge of Earth and space phenomena and to preserve the role of the United States as a leader in aeronautics, space science, and technology. High achievement in science, technology, engineering, and mathematics (STEM) education is essential to the accomplishment of NASA’s mission. The Strategic Management of Human Capital Initiative under the President’s Management Agenda requires agencies to “build, sustain, and effectively deploy the skilled, knowledgeable, diverse, and high-performing workforce needed” to meet agency core competencies. NASA’s education investments will contribute to the agency’s human capital needs.

All of NASA’s education efforts are part of an integrated agencywide approach to human capital management. Within the NASA Strategic Plan, education is identified as a cross-cutting function that supports all of the agency’s strategic goals and objectives. NASA delivers a comprehensive agency education portfolio—a collection of investments and strategies, such as research and development, managed to further common goals—implemented by the Office of Education, the NASA mission directorates, and the NASA centers. Through the portfolio NASA contributes to our nation’s efforts in achieving excellence in STEM education. Three outcomes serve to align all agency education activities:

Outcome 1: Strengthen NASA and the nation’s future workforce—NASA will identify and develop the critical skills and capabilities needed to achieve the Vision for Space Exploration. To help meet this demand, NASA will continue contributing to the development of the nation’s future STEM workforce through a diverse portfolio of education initiatives that target America’s students at all levels, especially those in traditionally underserved and underrepresented communities.
Outcome 2: Attract and retain students in STEM disciplines—To compete effectively for the minds, imaginations, and career ambitions of America’s young people, NASA will focus on engaging and retaining students in STEM education programs to encourage their pursuit of educational disciplines critical to NASA’s future engineering, scientific, and technical missions.
Outcome 3: Engage Americans in NASA’s mission—NASA will build strategic partnerships and linkages between STEM formal and informal education providers. Through hands-on, interactive, educational activities, NASA will engage students, educators, families, the general public, and all agency stakeholders to increase Americans’ science and technology literacy.

As the United States begins the second century of flight, the nation must maintain its commitment to excellence in STEM education to ensure that the next generation of Americans can accept the full measure of its roles and responsibilities in shaping the future.

Activities associated with these COTF collections will provide NASA Education research-based results into educational game development. These games will be used to support outcomes 2 and 3, informing NASA’s investment in development of videogames to support increased STEM literacy and pipeline achievement.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The COTF will conduct numerous studies on identifying and assessing learning and tracking flow in videogames. Though the methodology in each study may differ somewhat, the purpose of each collection is similar. The education community depends on quality research- and standards-based products to educate students, and NASA has a responsibility to help provide these educational resources to prepare its future workforce. Without basic research into the assessment of learning in games, NASA Education will have no measurement of how much learning occurs in the games it develops. The results of these studies will be submitted to NASA Education’s Technology and Products Office to inform its development of videogames to support increased STEM literacy and pipeline achievement as well as to educate students of NASA missions and goals. Research results will be disseminated nationally through peer-reviewed publications and conference presentations yet to be determined.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

Most of the data for these activities will be collected online using database technologies. Phone interview respondents will also be noted by staff members, and the responses will be analyzed for research reports. Many of the pre- and posttest questions that will be asked in both focus group and face-to-face interviews will have responses compiled online to aid research efforts. Additionally, almost all of the data collected will be gained through software that tracks user skill and flow in games.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

COTF is conducting original research on learning and flow in educational videogames. This research has never before been conducted, so it is not duplicative. NASA relies on this type of research to inform its investment in developing games for the education community.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden.

To our knowledge, these data collections will have no impact on small businesses or entities.

6. Describe the consequence to federal program or policy activities if the collection is not conducted or is conducted less frequently as well as any technical or legal obstacles to reducing burden.

The NASA Strategic Plan defines plans and goals to help NASA achieve and support the Vision for Space Exploration. Without basic research into assessment of learning in games, NASA Education will have no measurement of how much learning occurs in the games it develops. These games will be used to support NASA Strategic Plan outcomes 1, 2, and 3, informing NASA’s investment in development of videogames to support increased STEM literacy and pipeline achievement.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

* requiring respondents to report information to the agency more often than quarterly;

* requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

* requiring respondents to submit more than an original and two copies of any document;

* requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, ortax records, for more than three years;

* in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

* requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

* that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

* requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

In general, study participants and expert reviewers will volunteer to participate in COTF assessment of learning in games studies. The amount of time between when COTF receives the participant’s request to volunteer and participation will be less than a month. A short turnaround increases responses rates.

CET/COTF is interested in how adolescents and adults learn from games. Exclusionary requirements will deal with being able to read the computer screen and hear the audio files that accompany the game without the aid of additional computer-based assistive technologies (e.g., screen readers). This is basic research and will not be advanced to the point where the project incorporates assistive technologies. Participants will be recruited from our Web site advertisement, through e-mail sent through our education networks (e.g., we contact teachers who have previously worked with us and have indicated a desire to participate in research and training efforts), through word of mouth, and through advertisement such as news releases in the media.

Specifics for individual collections follow.

Study 2—Genre-Learning Outcome Matrix

CET/COTF will make a list of gaming technology and education experts and contact them directly by telephone. Note: COTF staff will conduct this study by collecting information provided by the experts.

Study 3—Real-life Expectancies Based Upon Game-World Interactions

CET/COTF has begun to document publicly published comments by individuals that indicate that those persons have experienced instances in which they apply virtual world interface experiences toward real life. CET/COTF staff will contact these individuals and recruit participation.

Study 4—NASA Game Prototype Usability Testing (Playtesting)

CET/COTF will advertise locally in Wheeling and on the Wheeling Jesuit University campus as well as word of mouth through CET staff networks. Each year we are recruiting only nine usability playtesters for the NASA game and nine for the CET/COTF Selene experimental game (and other possible experimental games, not yet determined). Data from the experimental study will be used to refine game designs, gameplay, and data-mining techniques. Game revisions will require new iterations of game usability testing annually.

Study 5—Experimental Games—Selene—Usability Testing (Playtesting)

CET/COTF will advertise locally in Wheeling and on the Wheeling Jesuit University campus as well as word of mouth through CET staff networks. Each year we are recruiting only nine usability playtesters for the CET/COTF Selene experimental game (and other possible experimental games, not yet determined). Data from the experimental study will be used to refine game designs, gameplay, and data-mining techniques. Game revisions will require new iterations of game usability testing annually.

Study 6—NASA Game Experimental Testing

CET/COTF will use social and professional networks and formal and informal media advertisement to invite adult recruiters. The adult recruiters will invite the youth participants. The process goes much like the following:

I go skiing over the holidays and engage in an animated conversation with a couple who are also skiing. Talk evolves to viewing the moon and the Selene project, and the individuals volunteer to serve as recruiters. These recruiters then will attract youth participants and other responsible adult recruiters.

Study 7—Experimental Games (e.g. Selene) Assessment:

CET/COTF will use social and professional networks and formal and informal media advertisement to attract adult recruiters. The adult recruiters will attract the youth participants. See example in box within the Study 6 collection description above.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.

Specifically address comments received on cost and hour burden. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data,

frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. Consultation with representatives of those from whom information is to be obtained or those who must

compile records should occur at least once every 3 years - even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

N/A.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Usability testers (playtesters) for data collections 4 and 5 will be hired as part-time consultants and will receive a stipend of $280 ($7/hr for 40 hours).

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

All CET/COTF study participants and expert reviewers will review informed consent forms, approved by Wheeling Jesuit University’s Institutional Review Board, which assures them of confidentiality (see the attached consent form for each data collection) in alignment with federal regulations for human subjects research. All data collections will be coded for anonymity. No personally identifiable information (PII) will be collected with study data. This also ensures confidentiality.

Contact information will be collected for respondents in studies 1, 2, 3, 4, and 5 and for adult recruiters in studies 6 and 7. Contact information will be used for administrative purposes, such as scheduling. Contact information will not be linked with other study data. COTF has filed privacy impact assessment PIA paperwork for each of these seven data collections to its Goddard Alternate Privacy Act Manager (APAM).

The file “nasaTripToMoonFlierWeb.pdf” is a sample poster COTF will use to invite responsible adults to recruit youth participants for studies 6 and 7.

Study 1—Commercial Off-the-shelf Games and Learning

Submission: Submission is voluntary.

Data:

Contact information. COTF will collect potential expert reviewer’s contact information (name, phone number, e-mail, postal address). Information use. COTF will use this information to administer the expert review. This is information in identifiable form (IIF) data.
Expert Review. COTF will provide a description of games prototypes and exemplars and learning objectives. Respondent adult experts will be asked to rate the games for their contribution toward each outcome and write a narrative explaining their ratings. Rater names will be removed from narratives and ratings and coded with numbers. Data will be stored on a secure server within the COTF computer system. Data will be accessed by project personnel for statistical and qualitative analyses. All CET security measures will be maintained on these servers. COTF will compile the ratings and analyze the narratives. Information use: These data will be used to study which game types seem to support which type of learning outcome. Dissemination: Study results from gameplay assessments data will be published in professional journals, at conferences, and reported to NASA officials. Participants’ PII will not be disseminated.

IIF is collected to recruit experts and collect their materials. Only contact information and scheduling information will be recorded. Participants’ ratings and narratives data will be recoded as anonymous and not electronically connected to contact information data. No personal information will be collected from children under the age of 13. Contact information will be secured within our onsite and back-up systems according to our organization’s security system. This system is in compliance with NASA security system requirements.

Study 2—Genre-learning Outcome Matrix

Submission: Submission is voluntary.

Data:

Contact information. COTF will collect potential expert reviewers’ contact information (name, phone number, e-mail, postal address). Information use. COTF will use this information to administer the expert review. This is IIF data.
Expert review. COTF will provide a description of games prototypes and exemplars and learning objectives. Respondent adult experts will be asked to rate the games for their contribution toward each outcome and write a narrative explaining their ratings. Rater names will be removed from narratives and ratings and coded with numbers. Data will be stored on a secure server within the CET computer system. Data will be accessed by project personnel for statistical and qualitative analyses. All CET security measures will be maintained on these servers. COTF will compile the ratings and analyze the narratives. Information use. The data will be used to study which game types seem to support which type of learning outcome. Dissemination: Study results from gameplay assessments data will be published in professional journals, at conferences, and reported to NASA officials. Participants’ PII will not be disseminated.

IIF is collected to recruit experts and to collect their materials. Only contact information and scheduling information will be recorded. Participants’ ratings and narratives data will be recoded as anonymous and not electronically connected to contact information data.

Often, CET/COTF will obtain this IIF (contact information, usually for the reviewer’s place of business) from the Internet. Sometimes CET/COTF will acquire it from professional and social networks, and sometimes from the individual reviewer. Adults who are interested in serving as expert reviewers will respond to direct recruitment by CET/COTF staff. Participation by expert reviewers is voluntary. The participant pool will receive information via telephone and e-mail. Participants will complete an informed consent form, approved by the Wheeling Jesuit Institutional Review Board. Participants’ IIF will not be disseminated.

Study 3—Real-life Expectancies Based Upon Game-world Interactions

Submission: Submission is voluntary.

Data:

Contact information. CET/COTF will collect contact information from and for adults who have indicated they transfer experiences from virtual worlds as expectations for the real world. This is IFF. Information use. This information will be used to contact potential participants, to secure agreement to participate, to distribute the informed consent form, and to schedule interviews.

Transcription of interview. CET/COTF will interview adults who have indicated they transfer experiences from virtual worlds as expectations for the real world. The researcher will ask the interviewee to describe how interactions with the virtual world interface may have carried over into real life: Does the participant ever find himself/herself trying to accomplish something in real life that he/she would do in the virtual world? One absurd example might be flying or teleporting. Does the participant ever, for even a brief instant, anticipate flying or teleporting to move from one location to another? If the researcher and the participant identify transfer expectations, then the researcher will ask the participant to describe those transfer expectations, their correlates (the real life expectation and the virtual world activity), and why they might occur. This information will be anonymously recorded. There will be no IIF associated with the protocols. Information use. COTF will use this information to study the effects of transactions with virtual world objects. Study results will be published in professional journals, at conferences, and reported to NASA officials.

IIF is collected only of the adults who have indicated they have applied virtual world expectations toward real-life experience. Only contact information will be IIF. This is a qualitative study, and COTF will collect a maximum of 45 interviews. It will conduct three waves of data collection, 15 in each wave. Wave 1 will be used to identify themes. Waves 2 and 3 will be used to develop information about those themes.

IIF is collected so that CET/COTF can (a) contact potential participants, (b) conduct interviews, and (c) follow federal guidelines for obtaining informed consent for research studies involving human subjects.

CET/COTF will use a pyramid or network approach to build a population of potential subjects. In other words, one participant will refer another. CET/COTF will also use informal and formal contacts with other professionals to form a list of potential participants. Subjects will be informed of the purpose of the data collection, specifically, “This research project studies how experiences in a virtual world, like Second Life, might influence peoples’ expectations for real-world experiences.” The written informed consent form contains a description of the project. CET/COTF staff will also share the details of the study with participants during recruitment and interview phone calls. The Institutional Review Board of Wheeling Jesuit University has already reviewed and approved this research study for individuals who are members of the Second Life virtual community.

Study 4—NASA Game Prototype Usability Testing (Playtesting) and

Study 5—Experimental Games (e.g., Selene) Usability Testing (Playtesting)

Submission: Submission is voluntary because participation is voluntary.

Data:

Contact information. COTF will collect contact information for/from persons who are interested in a short, part-time job playtesting the COTF and NASA games. Information use. COTF will collect potential participant’s contact information (name, phone number, e-mail, postal address) to recruit and schedule research studies.

Signature on informed consent document. Playtesters will sign informed consent documents. It will be necessary to collect contact information for recruiting, scheduling, tracking hours worked, and requests for payment of stipend. This is IIF data. It will be kept separate from usability testing (playtesting) data.

Stipend payment information. Wheeling Jesuit University Sponsored Programs Finance will require playtesters’ Social Security information and mailing address in order to pay stipends to playtesters. CET/COTF will not maintain this information. Information use. This information will be used to pay stipends, to record payment for the federal government, and to mail stipends to playtesters.

Usability testing (Playtesting). Part-time, temporary employees will participate in usability testing of each game developed for CET/COTF to study. COTF will track participants’ gameplay activity during a gameplay. Playtesters will also talk with researchers one on one and in small groups about the games. The data will be anonymous as no PII will be collected. Thus, the data will contain no IIF. Participation as a playtester is voluntary, but playtesters will be paid a $7/hour stipend. Information use. This information will be used to find errors in the game and game database structures and to improve gameplay. Results from usability testing will inform product development. It may also be incorporated into presentations published in professional journals, at conferences, and reported to NASA officials. There will be no IIF information within the data.

Paperwork Reduction Act (PRA) guidelines state that, for testing data collection mechanisms, 10 persons or less should be used. COTF is establishing a playtester population for each game (data collection) of nine persons. This accords with established instructional design practice (Smith & Ragan, 1993) and established game design practice for academic institutions (Fullerton, Swain, & Hoffman, 2004). It also allows COTF to group playtesters as individuals for one-to-one session, in groups of three for small group session, and in a large group of nine. IIF is not collected during usability testing. However, it will be necessary to collect contact information containing IIF for recruiting, scheduling, and requests for payment of stipend.

IIF will be collected to schedule and pay playtesters. It is not connected to the data collection. Each playtester is to be paid $7/hour for about 40 hours of work.

Playtesters will complete an informed consent document, approved by the Wheeling Jesuit University Institutional Review Board. Except (a) as necessary to pay and schedule playtesters and (b) the signature on the informed consent document, this data collection will not involve IIF. The Wheeling Jesuit University Sponsored Programs Finance Office will collect IIF to pay playtesters. CET/COTF will not have access to the personal information that playtesters submit to the Finance Office in order to receive their stipends. Individuals who volunteer to serve as playtesters (usability testers) will be asked to provide their contact information and to sign an informed consent form. Playtesters will be told that their contact information will be used for administrative purposes and to document the informed consent. This will be communicated in person, on hard copies of documents, via e-mail, and/or by telephone. IIF information will not be shared by CET/COTF. Administrative information that participants have completed their 40 hours of playtesting will be submitted to the Wheeling Jesuit University Sponsored Programs Finance Office so that participants can be paid their stipend.

Study 6—NASA Game Experimental Testing and Study 7—Experimental Games (e.g., Selene) Assessment

Submission: Submission is voluntary.

Data Collection 7. Experimental Game (Selene)

Experimental game (Selene) adolescent data collection. COTF will track game participants’ gameplay activity during a game about lunar science concepts of volcanism and impact cratering. The data will be anonymous as no PII about the youths will be collected. Thus, the data will contain no IIF. Participation in this study is voluntary. Parents of all study participants will receive Wheeling Jesuit University Institutional Review Board-approved informed consent forms. Consent forms will contain all details describing the study data collection in accordance with federal guidelines. These will be distributed to parents of interested youth through a network of adults. These adults (recruiters) will obtain permission from parents/guardians for adolescents age 13 and up to participate in the study. This will be a passive consent process, and no personal information containing parents’ or adolescents’ names will be forwarded by recruiters to CET. Children must be 13 and older to participate in this study. Information use. To study the properties of games with embedded assessment instruments while the games are in development in order to refine the game and its embedded assessments.

Experimental game (Selene) adult recruiter data collection. COTF will enlist the efforts of responsible adults to recruit youth as game players. These adults (recruiters) will conduct the passive consent process described above. CET/COTF will build a database of the recruiter network that contains the recruiters’ names and contact information (e-mail, mailing address, phone number, and time available for phone contact). The database will also contain a record of the number of participants registered by the adult recruiter. Information use. COTF will use the contact information to train the recruiters in the informed consent process and to inform the recruiters of project dates.

Experimental game (Selene) adult data collection. COTF will track game participants’ gameplay activity during a game about lunar science concepts of volcanism and impact cratering. The data will be anonymous, as no PII about the adult participants will be collected. Thus, the data will contain no IIF. Participation in this study is voluntary. Participants will receive Wheeling Jesuit University Institutional Review Board-approved informed consent forms. Consent forms will contain all details describing the study data collection, in accordance with federal guidelines. This will be a passive consent process. Participants indicate agreement with the passive consent by clicking a button to proceed and play the game. Information use. To study the properties of games with embedded assessment instruments while the games are in development in order to refine the game and its embedded assessments.

Data Collection 6. NASA Game

NASA game adolescent data collection. COTF will track game participants’ gameplay activity during a game about NASA. The data will be anonymous, as no PII about the youths will be collected. Thus, the data will contain no IIF. Participation in this study is voluntary. Parents of all study participants will receive Wheeling Jesuit University Institutional Review Board-approved informed consent forms. Consent forms will contain all details describing the study data collection in accordance with federal guidelines. These will be distributed to parents of interested youth through a network of adults. These adults (recruiters) will obtain permission from parents/guardians for adolescents age 13 and up to participate in the study. This will be a passive consent process, and no personal information containing parents’ or adolescents’ names will be forwarded by recruiters to CET. Children must be 13 and older to participate in this study. Information use. To study the properties of games with embedded assessment instruments while the games are in development in order to refine the game and its embedded assessments.

NASA game adult recruiter data collection. COTF will enlist the efforts of responsible adults to recruit youth as game players. These adults (recruiters) will conduct the passive consent process described above. CET/COTF will build a database of the recruiter network that contains the recruiters’ names and contact information (e-mail, mailing address, phone number, and time available for phone contact). The database will also contain a record of the number of participants registered by the adult recruiter. Information use. COTF will use the contact information to train the recruiters in the informed consent process and to inform the recruiters of project dates.

NASA game adult data collection. COTF will track game participants’ gameplay activity during a game about NASA. The data will be anonymous, as no PII about the adult participants will be collected. Thus, the data will contain no IIF. Participation in this study is voluntary. Participants will receive Wheeling Jesuit University Institutional Review Board-approved informed consent forms. Consent forms will contain all details describing the study data collection in accordance with federal guidelines. This will be a passive consent process. Participants indicate agreement with the passive consent by clicking a button to proceed and play the game. Information use. To study the properties of games with embedded assessment instruments while the game is in development in order to refine the game and its embedded assessments.

Ultimate information use: Study results from adolescent and adult gameplay data will be published in professional journals, at conferences, and reported to NASA officials. Adult recruiter contact information data will not be disseminated. CET may report descriptives of aggregate numbers of youth participants per recruiter.

Concluding information use: Study results from adolescent and adult gameplay data will be published in professional journals, at conferences, and reported to NASA officials. Adult recruiter contact information data will not be disseminated. CET may report descriptives of aggregate numbers of youth participants per recruiter.

IIF is collected only from the responsible adult recruiter network individuals. Only contact information and number of recruited adolescents will be recorded. CET is not collecting any IIF about the adolescents who will participate in the study. Adult recruiter information will not be tagged to adolescent gameplay data and will not be disseminated. COTF has targeted about 1,000 complete and usable player records for each game study (NASA game and experimental game) each year. CET/COTF will continue to invite and register recruiters for each iteration of the study (each year) until the targeted number of player records is achieved.

Adolescent participants in this project will not provide any IIF. Adolescents will complete an informed consent document approved by the Wheeling Jesuit University Institutional Review Board.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

Study 1—Commercial Off-the-shelf Games and Learning: N/A

Study 2—Genre-learning Outcome Matrix: N/A

Study 3—Real-life Expectancies Based Upon Game-world Interactions: N/A

Study 4—NASA Game Prototype Usability Testing (Playtesting): N/A

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting): N/A

Study 6—NASA Game Experimental Testing and Study 7—Experimental Games (e.g. Selene) Assessment:

COTF will ask study 6 and study 7 participants to provide demographic information. Submission of this information is optional, as is participation in the study. This information is important in order for us to study the relationships between gaming parameters, learning, and learner characteristics.

Additionally, for reporting purposes and to ensure diversity, NASA must track and report this type of information. The NASA education portfolio is established upon an overarching philosophy of cultivating diversity. The cultivation of diversity is both a management philosophy and core value for all NASA education efforts. Diversity of skills and talents needed in its future workforce is critical to NASA’s success. Potential at both the individual and organizational levels will be maximized by fostering awareness, understanding, and respect for individual differences. The knowledge, expertise, and unique background and life experiences—including ethnic, gender, racial, religious, and cultural identity—of each individual strengthen the agency (NASA Education Strategic Coordination Framework: A Portfolio Approach, p. 6).

12. Provide estimates of the hour burden of the collection of information. The statement should:

* Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour

burden on respondents is expected to vary widely because of differences in activity, size, or

complexity, show the range of estimated hour burden, and explain the reasons for the variance.

Generally, estimates should not include burden hours for customary and usual business practices.

* If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

* Provide estimates of annualized cost to respondents for the hour burdens for collections of

information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 13.

Data Collection	Name	Start	End	Purpose	Sampling Method	Population	Number in Sample	Burden Hours
1	Commercial Off-the-shelf (COTS) Games and Learning	FY08	FY 09	Develop methodology for assessment of learning due to COTS games.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults.	120, 60/yr	120/yr
2	Genre-learning Outcome Matrix	FY07	FY 09	Identify the match between learning outcomes and game genres.	Expert sampling—modal purposive nonprobability sampling	Experts (adults).	45, 15/yr	150/yr
3	Real-life Expectancies Based Upon Game-world Interactions	FY07	FY 09	Support the hypothesis that people project their virtually embodied experiences (such as using a “mouse click” to identify an avatar’s social history in a game world) as expectations for real-world experience.	Snowball purposive nonprobability sampling	Adult players of games that occur in virtual worlds.	45, 15/year	15/yr
4	NASA Game Prototype Usability Testing (Playtesting)	FY07	FY09	Refine the properties of the NASA game embedded assessment instruments while the game is in development.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	The playtester pool—adults who are recruited and employed to come to our COTF lab or participate over the Internet to test game interfaces and assessment tools.	27, 9/yr	360/yr
5	Experimental Games (e.g. Selene) Usability Testing (Playtesting)	FY07	FY09	Refine the properties of the experimental games embedded assessment instruments while the game is in development.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	The playtester pool—adults who are recruited and employed to come to our COTF lab or participate over the Internet to test game interfaces and assessment tools.	27	360/yr
6	NASA Game Experimental Testing	FY07	FY09	Assess changes in learning and flow due to the NASA game.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults and minors. Adults complete an anonymous informed consent form. Minors are overseen by an adult volunteer team lead (such as a Scout leader). Team lead manages parental permissions; passive informed consent is used to play the game. No identifying information is gathered about either adult or minor participants.	3,000	1,000/yr
7	Experimental Games (e.g. Selene) Assessment	FY07	FY 09	Assess changes in learning and flow due to the experimental game/s.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults and minors. Adults complete an anonymous informed consent form. Minors are overseen by an adult volunteer team lead (such as a Scout leader). Team lead manages parental permissions; passive informed consent is used to play the game. No identifying information is gathered about either adult or minor participants.	4,500	1,500/yr

Per study:

Study 1—Commercial Off-the-shelf Games and Learning. Our data collection will require about two hours for assessment of learning (pretest and posttest). We estimate 120 participants, 60/yr. This totals 120 hours/yr.

Study 2—Genre-learning Outcome Matrix. Forty-five participants will each spend 10 hours or less completing the learning outcome-genre matrix and narrative (See COTF_outcomes_genre_info.doc.) Only 15 people a year will participate, so the yearly burden is 150 hours.

Study 3—Real-life Expectancies Based Upon Game-world Interactions. Forty-five participants, 15 per year, will each spend one hour or less talking with COTF staff on the telephone. The researcher will ask the interviewee to describe how interactions with the synthetic world interface may have carried over into real life: Does the participant ever find himself/herself trying to accomplish something in real life that he/she would do in a synthetic world? If the researcher and the participant identify transfer expectations, then the researcher will ask the participant to describe those transfer expectations, their correlates (the real-life expectation and the synthetic-world activity), and why they might occur. The participant will also be asked to provide references that refer other synthetic-world community members known to have had similar transfer expectations.

Study 4—NASA Game Prototype Usability Testing (Playtesting). Nine participants will each work approximately 40 hours on playtesting. COTF has used game design (Fullerton et al., 2004) and instructional design literatures (Smith & Ragan, 1993) to set the number of usability testers (playtesters) at nine. According to Fullerton, playtesting is the single most important game design activity (p. 196). Technically, playtesting is not the same as usability testing. Playtesting allows the designer additional insight into how the players experience the game. It is a component of iterative design (test, evaluate, and revise). Both group and individual dynamics are necessary to test game effectiveness. Historically, four individuals can provide sufficient one-to-one (player to researcher) information in early states of instructional design. Field trials are conducted with about 30 individuals representing each targeted audience. COTF budgeted to employ nine playtesters. This will allow the researchers the flexibility to study individuals at play within the game, focus groups of three players, and larger group (nine members) open discussion. Given the time (only about two weeks) constraints, responsible use of budget dollars, and financial resource constraints, nine playtesters will allow COTF grouping flexibility while maintaining a manageable sample size that will provide informative data without overwhelming the analysis with too much data to be processed, evaluated, and reported to the game design team at Georgia Tech. A playtester pool of nine affords breakout into smaller groups of three. This practice accords with the federal policy to test instruments with 10 persons or fewer to reduce the burden on the public and enhance the quality of the information obtained through the data collection. (83i justification A.12. “Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample [fewer than 10] of potential respondents is desirable.”)

Study 5—Experimental Games Usability Testing (Playtesting). Nine participants will each work approximately 40 hours on playtesting. COTF has used game design (Fullerton et al., 2004) and instructional design literatures (Smith & Ragan, 1993) to set the number of usability testers (playtesters) at nine. According to Fullerton, playtesting is the single most important game design activity (p. 196). Technically, playtesting is not the same as usability testing. Playtesting allows the designer additional insight into how the players experience the game. It is a component of iterative design (test, evaluate, and revise). Both group and individual dynamics are necessary to test game effectiveness. Historically, four individuals can provide sufficient one-to-one (player to researcher) information in early states of instructional design. Field trials are conducted with about 30 individuals representing each targeted audience. COTF budgeted to employ nine playtesters. This will allow the researchers the flexibility to study individuals at play within the game, focus groups of three players, and larger group (nine members) open discussion. Given the time (only about two weeks) constraints, responsible use of budget dollars, and financial resource constraints, nine playtesters will allow COTF grouping flexibility while maintaining a manageable sample size that will provide informative data without overwhelming the analysis with too much data to be processed, evaluated, and reported to the game design team at Georgia Tech. A playtester pool of nine affords breakout into smaller groups of three. This practice accords with the federal policy to test instruments with 10 persons or fewer to reduce the burden on the public and enhance the quality of the information obtained through the data collection. (83i justification A.12. “Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample [fewer than 10] of potential respondents is desirable.”)

Study 6—NASA Game Experimental Testing. In this study 1,000 participants will spend one hours each, totaling 1,000 hours. This study will be repeated with new iterations of the game in FY 07, FY08, and FY09. This totals 3,000 hours.

Study 7—Experimental Games (e.g. Selene) Assessment. In this study 1,500 participants per year will spend one hour each, totaling 1,500 hours/yr, FY 07-FY09.

13. Provide an estimate for the total annual cost burden to respondents or recordkeepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

* The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services component. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling and testing equipment; and record storage facilities.

* If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10), utilize the 60-day pre-OMB submission public comment process and use existing economic or regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate.

* Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

N/A.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

For each year for studies 4 and 5, there will be nine testers hired to work on a part-time basis for the COTF. The testers will be paid $7/hour. For each study, testers will work approximately 40 hours.

$7/hour x 40 hours = $280 per tester

$280 x 9 testers = $2,520 per study (Total $15,120)

We are budgeted by NASA in one-year increments with funding to cover one year of work. We anticipate that subsequent years will fol1ow the FY07 budget allocations. For the current fiscal year, we anticipate 6,739 hours will be spent by our computer programmers, graphic artist, instructional designers, educational researchers, TV production team, and content area experts (scientists) to prepare, conduct, analyze, and report the studies listed for FY07. We are also supporting an external game design team and an external statistician. We estimate that this number of hours for the playtesters (detailed above), external consultants, and the staff involved will cost $413K (including indirects and cost centers).

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.

N/A.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Data collection	Name	Start	End	Tabulation, methodology, and analytical techniques	Report	Publication	Conferences
1	Commercial Off-the shelf (COTS) Games and Learning	FY08	FY 09	Descriptives, regression, t-tests, ANOVAs	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
2	Genre-learning Outcome Matrix	FY07	FY 09	Qualitative and quantitative: descriptives and grounded theory	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
3	Real-life Expectancies Based Upon Game-world Interactions	FY07	FY 09	Qualitative and grounded theory	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
4	NASA Game Prototype Usability Testing (Playtesting)	FY07	FY09	Qualitative and grounded theory: lists, notes, data mining¹	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
5	Experimental Games (e.g. Selene) Usability Testing (Playtesting)	FY07	FY09	Qualitative and grounded theory: lists, notes, data mining¹	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
6	NASA Game Experiment	FY07	FY09	Data mining¹	Annual report to NASA	As accepted through peer review process	As accepted through peer review process
7	Experimental Games (e.g. Selene) Assessment	FY07	FY 09	Data mining¹	Annual report to NASA	As accepted through peer review process	As accepted through peer review process

¹ Data mining—Classical statistical methods that may be used include but are not limited to ANOVA, multivariant regression, principal component analysis, and k Nearest Neighbor clustering. We suspect that nonlinear methods may be required given the inherent complexity of the learning process. The nonlinear methods most likely to yield information fall into the broad category of self-organizing systems. These systems, in general, cluster data based on commonalities within the data itself without regard to outcome or result. The clusters can then be examined for common factors but not included in the data driving the clustering. These data mining methods will be pilot tested by SELENE during spring and early summer of 2007 using the Selene prototype game.

Study 1—Commercial Off-the-shelf Games and Learning. The project will begin in FY2008. Protocols will be scored for learning gains using repeated measures analyses. Learning gains will be included within COTF-WJU annual reports to NASA Education annually. COTF will also write results and methods for publication in peer-reviewed professional journals and at conferences yet to be determined.

Study 2—Genre-learning Outcome Matrix. The project will begin FY07 and continue through FY09. COTF will calculate a mean and a standard deviation score for each learning outcome/genre dyad. The results of these studies will be submitted to the NASA Education Technology and Products Office annually and disseminated nationally through peer-reviewed publications and conference presentations yet to be determined.

Study 3—Real-life Expectancies Based Upon Game-world Interactions. This data collection will begin in FY07 and continue through FY09. COTF will use NVivo, which is software developed by QSR International to help researchers manage, shape, and analyze information to identify themes within interviewee protocols. COTF will record frequencies of occurrence for each theme. The results of these studies will be submitted annually to the NASA Education Technology and Products Office and disseminated nationally through peer-reviewed publications and conference presentations yet to be determined.

Study 4—NASA Game Prototype Usability Testing (Playtesting). This is not a study, but usability testing of the NASA game (see answer to item 12 for explanation). The project will begin FY07 and continue through FY09. In other words, COTF is prepared to playtest a NASA game each year. COTF will examine the results for patterns, but the small sample size will not support most inferential statistical analyses. The results of the study will be used by the NASA game prototype project team to refine the game and its embedded assessments. Results will be included in a report submitted annually to the NASA Education Technology and Products Office and disseminated nationally through peer-reviewed publications and conference presentations.

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting). The project will begin FY07 and continue through FY09. This is not a study but usability testing of the NASA game (see answer to item 12 for explanation). COTF plans to test one game per year. This could be the Selene game about lunar geology and its subsequent enhancements, revisions, and modules, or it could be a different game aligned with enhancing STEM achievement and the NASA mission.

Study 6—NASA Game Experimental Testing. The project will begin FY07 and continue through FY09. COTF will use statistical software to identify patterns in the data (e.g., data mining, reliability, correlations, factor analysis, and Rasch Modeling). The results of these studies will be submitted annually to the NASA Education Technology and Products Office and disseminated nationally through peer-reviewed publications and conference presentations yet to be determined.

Study 7—Experimental Games (e.g. Selene) Assessment. Selene is a game about lunar geology based upon the findings of the NASA lunar missions and satellites. Dr. Charles A. Wood, planetary scientist and geologist, formerly of Johnson Space Center and currently a Cassini mission scientist studying Saturn’s moon Titan, is the subject matter expert for this game. The game is being developed in collaboration with the game studies program at Georgia Institute of Technology’s Literature, Communication, and Culture department. COTF will use statistical software to identify patterns in the data (e.g., data mining, reliability, correlations, factor analysis, and Rasch Modeling). The results of these studies will be submitted annually to the NASA Education Technology and Products Office and disseminated nationally through peer-reviewed publications and conference presentations yet to be determined.

Educational Game Research Publication and Presentation, In General

The research/conference presentation cycle typically takes about one year from proposal to presentation. Publication in peer-reviewed journals and edited books may take from one to two years. COTF regularly presents at the American Educational Research Association (AERA), which is the premier educational research conference, and the Association for Educational Communications and Technology (AECT) conference. Educational gaming research is a timely topic, at the cutting edge of contemporary education research. Although COTF is targeting AERA and AECT journals, COTF theoretical research (research conducted by COTF educational researcher Dr. Debbie Denise Reese through synthesis of the literature in many education-related fields) has led to the following peer-reviewed conference presentations and publication opportunities:

Reese, D. D. (2007, March). Designing Selene: Theory-based game design and data-mining. Paper presented at the Serious Games Summit, San Francisco.

Fullerton, T., Swain, C., & Hoffman, S. (2004). Game design workshop: Designing, prototyping, and playtesting games. San Francisco: CMP Books.

Reese, D. D. (2006). Foundations of serious games design and assessment (No. COTF/LVP/Sep-2006). Wheeling, WV: Center for Educational Technologies, Wheeling Jesuit University.

Smith, P. L., & Ragan, T. J. (1993). Instructional design (1st ed.). New York: Merrill.

COTF will continue to pursue these and other presentation and publication opportunities to disseminate NASA Education-funded research.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

NA—We will add the OMB number to registration screens—or some other screen—in the software; for verbal collections, such as interviews in person or via phone, the OMB number will be added to the script and relayed to participants.

18. Explain each exception to the certification statement identified in Item 19, "Certification for

Paperwork Reduction Act Submissions," of OMB Form 83-I.

None

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the Supporting Statement to the extend that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Data collection	Name	Start	End	Purpose	Sampling method	Population	Number in sample	Burden hours
1	Commercial Off-the shelf (COTS) Games and Learning	FY08	FY09	Develop methodology for assessment of learning due to COTS games.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults	120	120/yr
2	Genre-learning Outcome Matrix	FY07	FY09	Identify the match between learning outcomes and game genres.	Expert sampling: Modal purposive nonprobability sampling	Experts (adults)	45	150/yr
3	Real-life Expectancies Based Upon Game-world Interactions	FY07	FY09	Support the hypothesis that people project their virtually embodied experiences (such as using a “mouse click” to identify an avatar’s social history in a game world) as expectations for real-world experience.	Snowball purposive nonprobability sampling	Adult players of games that occur in virtual worlds.	45	15/yr
4	NASA Game Prototype Usability Testing (Playtesting)	FY07	FY09	Refine the properties of the NASA game embedded assessment instruments while the game is in development.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	The playtester pool—Adults who are recruited and employed to come to our COTF lab or participate over the Internet to test game interfaces and assessment tools.	27	360/yr
5	Experimental Games (e.g. Selene) Usability Testing (Playtesting)	FY07	FY09	Refine the properties of the experimental games’ embedded assessment instruments while the games are in development.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	The playtester pool—Adults who are recruited and employed to come to our COTF lab or participate over the Internet to test game interfaces and assessment tools.	27	360/yr

6	NASA Game Experimental Testing	FY07	FY07	Assess changes in learning and flow due to the NASA game.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults and minors. Adults complete an anonymous informed consent form. Minors are overseen by an adult volunteer team lead (such as a Scout leader). Team lead manages parental permissions; passive informed consent is used to play the game. No identifying information is gathered about either adult or minor participants.	About 3,000 game players.	1,000/yr
7	Experimental Games (e.g. Selene) Assessment	FY07	FY09	Assess changes in learning and flow due to the experimental game/s.	Accidental, haphazard, or convenience (volunteers) nonprobability sampling	Adults and minors. Adults complete an anonymous informed consent form. Minors are overseen by an adult volunteer team lead (such as a Scout leader). Team lead manages parental permissions; passive informed consent is used to play the game. No identifying information is gathered about either adult or minor participants.	Around 4,500 game players.	1,500/yr

Study 1—Commercial Off-the-shelf Games and Learning

Recruitment and selection methods:

How participants recruited: Local media, other local organizations’ publicity, within the Wheeling Jesuit University campus.
Factors considered in screening candidates and selecting participants.
Instruments used for recruiting participants.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants. Their responses to written or electronic pre/posttests written to measure the game’s learning outcome.
Where is this data retained?

Within the Center for Educational Technologies’ data information system. This system is described in the CET security document. Three types of backup storage are used at the CET. Near-line backups are stored in the server room close to the servers for instant restores. Daily backups are stored in a fireproof safe on another floor separate from the server room. The monthly backups are stored in another building on the Wheeling Jesuit University campus.

How do we access it?

Through our desktop and laptop computers and within file cabinets in locked offices. Any individual hard copies are locked in the data room during a study and then archived within a locked storage room. All stored boxes are marked with dates and directions for disposal. The Wheeling Jesuit University Institutional Review Board requires data retention for three years after a study has been completed. Contact data is maintained until the CET changes its mission from education or disbands—whichever occurs first.

Study 2—Genre-learning Outcome Matrix

Recruitment and selection methods:

Participants are recruited from lists of knowledgeable professionals from fields related to gaming and learning science.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants.

Their responses on the learning outcomes matrix (see file COTF_outcomes_genre_infoDDR 07Jan02 ) and the narrative they write to accompany their ratings.

Where is this data retained?

Data is retained within the Center for Educational Technologies’ data information system. This system is described in the CET security document. Three types of backup storage are used at the CET. Near-line backups are stored in the server room close to the servers for instant restores. Daily backups are stored in a fireproof safe on another floor separate from the server room. The monthly backups are stored in another building on the Wheeling Jesuit University campus.

How do we access it?

Data is accessed through staff desktop and laptop computers and within file cabinets maintained within locked offices. Any individual hard copies are locked in the data room during a study and then archived within a locked storage room. All stored boxes are marked with dates and directions for disposal. The Wheeling Jesuit University Institutional Review Board requires data retention for three years after a study has been completed. Contact data is maintained until the CET changes its mission from education or disbands—whichever occurs first.

Study 3—Real-life Expectancies Based Upon Game-world Interactions

Recruitment and selection methods:

Participants are recruited whenever the researchers hear or read a comment about application of virtual world expectancies to the real world.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants.

A qualitative record of their explanation of how, what, when, where, and why they apply virtual world affordances to the real world. Oral data is transcribed during the interview session by a member of the research team.

Where is this data retained?

How do we access it?

Study 4—NASA Game Prototype Usability Testing (Playtesting)

Recruitment and selection methods:

Participants are recruited from families, local high schools and colleges, and the Wheeling Jesuit University faculty, staff, and student body and their extended community.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants.

Their responses to the oral discussion sessions, their one-on-one responses, small group responses, their data-mined data.

Where is this data retained?

How do we access it?

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting)

Recruitment and selection methods:

Participants are recruited from families, local high schools and colleges, and the Wheeling Jesuit University faculty, staff, and student body and their extended community.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants.

Their responses to the oral discussion sessions, their one-on-one responses, small group responses, their data-mined data.

Where is this data retained?

How do we access it?

Study 6—NASA Game Experimental Testing

Recruitment and selection methods:

Participants are recruited through social and professional networks. All adult recruiters are screened by our CET research assistants, who serve as gatekeepers that recruiters are adults and that they understand their voluntary responsibilities as a recruiter. Interested youths, ages 13-18, are invited by the recruiters, who are simply looking for interested participants. Please see our two recruitment posters and frequently asked questions. (Adult: nasaTripToMoonFlierWEB and Youth: nasaTripToMoonFlierPlayerRecruitment, FAQs for Selene Project).

How contacting participants (e.g., names, phone #, e-mail addresses)?

Yes, but only for recruiters. No personal information is collected for any youth study participant.

Data:

Data collected on participants.

Gameplay responses and demographics.

Where is this data retained?

How do we access it?

Study 7—Experimental Games (e.g. Selene) Assessment

Recruitment and selection methods:

Participants are recruited from families, local high schools and colleges, and the Wheeling Jesuit University faculty, staff, and student body and their extended community.

How contacting participants (e.g., names, phone #, e-mail addresses)? Yes.

Data:

Data collected on participants.

Their responses to the oral discussion sessions, their one-on-one responses, small group responses, their data-mined data.

Where is this data retained?

How do we access it?

Study 6—NASA Game Experimental Testing

Recruitment and selection methods:

Participants are recruited through social and professional networks. All adult recruiters are screened by our CET research assistants, who serve as gatekeepers to confirm that recruiters are adults and that they understand their voluntary responsibilities as a recruiter. Interested youths, ages 13-18, are invited by the recruiters, who are simply looking for interested participants. Please see our two recruitment posters and frequently asked questions. (Adult: nasaTripToMoonFlierWEB and Youth: nasaTripToMoonFlierPlayerRecruitment, FAQs for Selene Project).

How contacting participants (e.g., names, phone #, e-mail addresses)?

Yes, but only for recruiters. No personal information is collected for any youth study participant.

Data:

Data collected on participants.

Gameplay responses and demographics.

Where is this data retained?

How do we access it?

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Study 1—Commercial Off-the-Shelf Games and Learning: accidental, haphazard, or convenience (volunteers) nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 2—Genre-learning Outcome Matrix: expert sampling—modal purposive nonprobability sampling

* Statistical methodology for stratification and sample selection. None

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 3—Real-life Expectancies Based Upon Game-world Interactions: snowball purposive nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 4—NASA Game Prototype Usability Testing (Playtesting): accidental, haphazard, or convenience (volunteers) nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting): accidental, haphazard, or convenience (volunteers) nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 6—NASA Game Experimental Testing: accidental, haphazard, or convenience (volunteers) nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

Study 7—Experimental Games (e.g. Selene) Assessment: accidental, haphazard, or convenience (volunteers) nonprobability sampling

* Statistical methodology for stratification and sample selection. None.

* Estimation procedure. None for sampling.

* Degree of accuracy needed for the purpose described in the justification. We will recruit until the number of usable datasets equals the prespecified sample size.

* Unusual problems requiring specialized sampling procedures. None.

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden. Participants are asked to participate only one time.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Recruitment details for each study were detailed in item 1.

Study 1—Commercial Off-the-shelf Games and Learning. We will continue to recruit until we obtain the target number of 120 participants.

Study 2—Genre-learning Outcome Matrix. We will continue recruiting until we obtain 45 participants (15/year).

Study 3—Real-life Expectancies Based Upon Game-world Interactions. We will continue recruiting until we obtain 45 participants (15/year).

Study 4—NASA Game Prototype Usability Testing (Playtesting). We will continue recruiting, based upon referrals, until we obtain nine testers. These testers will become employees of COTF during the testing period.

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting). We will continue recruiting, based upon referrals, until we obtain nine testers. These testers will become employees of COTF during the testing period. This process will be repeated each year, FY07-FY09, for a total of 27 playtesters for each iteration of game testing.

Study 6—NASA Game Experimental Testing. We will recruit participants through COTF Web sites and affiliates (e.g., Challenger Learning Center, 4-H, colleague networks within the education and learning sciences communities who advertise the study. We will use the Jesuit network (e.g., Boston College, Georgetown University as brokered by Wheeling Jesuit University’s Vice President of Sponsored Programs Davitt McAteer) to recruit both adult and minor participants.

Study 7—Experimental Games (e.g. Selene) Assessment. We will recruit participants through COTF Web sites and affiliates (e.g., Challenger Learning Center, 4-H, colleague networks within the education and learning sciences communities who advertise the study. We will use the Jesuit network (e.g., Boston College, Georgetown University as brokered by Wheeling Jesuit University’s Vice President of Sponsored Programs Davitt McAteer) to recruit both adult and minor participants.

Recruiting notices:

Adult, see nasaTripToMoonFlierWEB
Youth, see nasaTripToMoonFlierPlayerRecruitment
All, see FAQs for Selene Project

Consent forms:

Study 1—Commercial Off-the-shelf Games and Learning. See game collection 1 informed consent.

Study 2—Genre-learning Outcome Matrix. See game collection 2 informed consent.

Study 3—Real-life Expectancies Based Upon Game-world Interactions. See game collection 3 informed consent.

Study 4—NASA Game Prototype Usability Testing (Playtesting). See game collection 4 informed consent.

Study 5—Experimental Games (e.g. Selene) Usability Testing (Playtesting). See game collection 5 informed consent

Study 6—NASA Game Experimental Testing. See game collection 6 informed consent.

Study 7—Experimental Games (e.g. Selene) Assessment. See game collection 7 informed consent, game collection 7 recruiter TOS, and game collection 7 player TOS.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

Study 1. Each game assessment will have assessment items developed to measure learner growth in the targeted learning objectives. The entire collection of data consists of participants’ pre and posttest responses to these assessment questions. We will submit any testing of these methods/procedures as a separate data collection.

Study 2. COTF conducted an in-house test of the learning outcome-game genre matrix. Computer and learning science staff managers formed teams and used the game genre-learning outcome matrix to review selected videogames.

Study 3. COTF will follow up on conversations or written comments by contacting the author and requesting a telephone conversation. Prospective participants will receive an informed consent form via e-mail and schedule an appointment for the interview. Once the informed consent form is returned, COTF will conduct the interview. The interview team will transcribe interviewee’s comments during the conversation. These comments will be analyzed using NVivo software. All protocols will be coded with a participant alias. COTF will not pretest this interview question. COTF will conduct one telephone interview with each participant. The interview will last less than an hour.

Study 4. Playtesting by nine playtesters is the pretesting (testing before the actual study) of the embedded assessment (learning and flow), the database system, and game mechanics. Playtesters will arrive for scheduled sessions of game playing, one-on-one, small group, or entire group session. Gameplay data will be recorded in the Selene database. COTF researchers will take notes of the small group, one-on-one, and entire group session comments. See Background_COTF_game_research.doc.

Study 5. Playtesting by nine playtesters is the pretesting (testing before the actual study) of the embedded assessment (learning and flow), the database system, and game mechanics. Playtesters will arrive for scheduled sessions of game playing, one-on-one, small group, or entire group session. Gameplay data will be recorded in the Selene database. COTF researchers will take notes of the small group, one-on-one, and entire group session comments. See Background_COTF_game_research.doc.

Study 6. Adolescents ages 13-18 will use a recruiter’s access code to enter the game site. The NASA game database will assign each player a password. Each player will select a unique user name. Each player will play the game. The game will track players’ gameplay decisions about NASA science. See Background_COTF_game_research.doc.

Study 7. Adolescents ages 13-18 will use a recruiter’s access code to enter the game site. The Selene database will assign each player a password. Each player will select a unique user name. Each player will play the game. Each player will be assigned to one of four conditions:

Game, instruction, game: Level 1 play game, watch instructional video, level 2 play game.
Game, game: Level 1 play game, level 2 play game.
Demonstration game, instruction, game: Level 1 observe game, watch instructional video, level 2 play game.
Demonstration game, game: Level 1 observe game, level 2 play game.

Figure 1. Double transfer paradigm as applied in SELENE studies.

Adapted from “Inventing to Prepare for Future learning: The Hidden Efficacy of Original Student Production in Statistics Instruction,” by D. L. Schwartz and T. Martin, 2004, Cognition and Instruction, 22(2), p. 149. Copyright 2004 by Lawrence Erlbaum Associates Inc.

Learning Goal: Targeted Domain Knowledge

		Learning resource:
Game Level 1		Instruction		Game Level 2
Preconceptual model knowledge	transfer in	Model knowledge	transfer out	Model knowledge
	→		→
Model building		Model building		Novel scenario
				Assessment

Figure 2. The double transfer paradigm as applied to design and assessment of learning in prepare for learning educational games targeting complex concept learning outcomes.

See Background_COTF_game_research.doc.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Dr. Debbie Denise Reese, 304-243-4327.

Page 18 of 34

File Type	application/msword
File Title	PRA
Author	Debbie Denise Reese, Ph.D.
Last Modified By	Walter Kit
File Modified	2007-03-07
File Created	2007-03-07