Outcomes

NASA-sponsored Classroom of the Future Outcomes and Matrix Information

COTF will also build on expert review by nine COTF staff conducted in FY06 that matched commercial, off-the-shelf (COTS) gameplay with learning outcomes. Games experts will be trained and then asked to study COTF definitions for learning types and game types and then rate each type of game genre for its ability to promote each type of learning objective. Experts will write a rationale for why they evaluated the games as they did. The document entitled “COTF_outcomes_genre_info.doc” contains the chart that experts will complete and a sample, completed expert narrative prepared.

The Expert Review

COTF extracted direct and paraphrased descriptions from primary and secondary sources to compile a definition for each of Robert Gagné’s learning outcomes. These, combined with the three outcomes based on the preceding section, form the elaborated set of learning outcomes. The COTF team prepared a matrix of game genres by learning outcomes for use by experts in evaluation the match between game genres and learning outcomes. Experts will use the matrix to rate each game genre/learning outcome dyad. Experts will write a narrative to explain why they gave the rating scores.

Developing the Job Aid

Job aids are verbal information, usually lists of steps and decisions. They are designed to help people complete a task when a set of complex procedures must be executed and memorization is not required (Smith & Ragan, 1993). The key component of a job aid is that it is a brief summary of a longer organized discourse. The COTF team created a learning outcome job aid that it used by the experts to review selected game genres and complete the learning outcome/genre matrix. The job aid is a table with brief descriptions for each learning outcome. The job aid serves two purposes. First, the brief descriptions support experts’ group discussion and consensus building about the meaning of each learning outcome. The process allows a set of experts to construct a shared understanding and common discourse. Second, as a job aid, the table of descriptions is a quick reference the experts can use to evaluate games while completing the genre by learning outcomes matrix.

The Learning Outcomes Job Aid

The Learning Objects Job Aid (see Table 1) is an annotated table of learning outcomes and brief descriptions. It was developed during consensus-building discussion by the team, based upon a set of notes from the literature about each of the learning outcome (see Appendix A). The job aid is designed as a quick reference for use with the genre descriptions and genre/learning outcome matrix included within this report (see Table 4). The matrix is used by experts to rate genre efficacies in support of learning goals. COTF developed an elaborated set of learning outcomes. While most of the learning outcomes listed derive directly from Gagné’s (Gagné, Briggs, & Wager, 1992) classification system, COTF derived three additional outcomes to address contemporary learning science perspectives. Any variations from Gagné’s classification system (non-Gagnéan learning outcomes) are noted within the table.

Table 1.

Job Aid Listing Learning Outcomes

Learning Outcome	Brief Description
Verbal information	The kind of knowledge people can state, knowing “what.”
Labels and names	Consistently naming an object or object class, such as associating "hat" with a hat or naming the object in a picture as a triangle. Ability to name the hat or triangle doesn't mean you know the meaning or purpose of a hat or definition or concept of a triangle.
Facts and lists	Relation between two or more objects or events. Some examples are multiplication facts, reciting Newton's Laws of Motion, that water boils at 100°C, or the list of English prepositions. Again, this doesn't mean a learner understands the facts/lists.
Organized discourse	A body of related information/knowledge that can be stated. Examples: the periodic table, a set of the facts and events within the context of a period of history. Two game examples: World of Warcraft - the background information for the game context or the background information on an island. Car racing game - the statistics on a car.

Table 1.

Job Aid Listing Learning Outcomes and Brief Descriptions Culled from the Narrative (cont.)

Learning Outcome	Brief Description
Intellectual Skill: Procedural Knowledge	The kind of knowledge people can do, knowing “how.”
Discrimination	Ability to distinguish between stimuli that differ along one or more dimensions. A learner can determine if two things are the same or different. Examples: Recognizing that the palette of two wines is different. Recognizing two vowel sounds as different. Recognizing that a rectangle is different from a triangle. Games: If teams are represented by two colors, you recognize there are two teams. You recognize the difference between different land form covers but don’t necessarily know what they represent. Recognizing that two types of vibrations by the controller are different.
Concept	Sets of objects, symbols, or events grouped together based upon shared characteristics. Concepts are mental representations of categories of objects, events, or other entities. Concepts can be considered classifying rules.
Concrete	Identify a class of something by its object properties; i.e., group things together by the physical characteristics they share. Examples: Identify the triangles in a group of objects by familiarity with their shape. A fence is a concrete concept.
Defined (Abstract)	A particular class of objects, relations, or events that USUALLY cannot be pointed to. A concrete object can also be a defined concept: (a triangle is a closed plane figure formed by the intersection of three line segments at three points). A “boundary line” is a defined concept (contrast this with the concrete concept “fence”).

Table 1.

Job Aid Listing Learning Outcomes and Brief Descriptions Culled from the Narrative (cont.)

Learning Outcome	Brief Description
Rule
Relational (e.g., if/then, cause/effect)	Relationship between two or more concepts. Examples: If clouds are heavy and dark, then bring an umbrella. Economic considerations were one cause of the Civil War.
Procedural	Steps to conducting a process. Examples: Problem-based learning (PBL) might be considered a series of steps. One could conduct the steps without understanding the concepts behind them. There might be steps to follow in troubleshooting a computer problem.
Problem-Solving	Inventing a solution by applying concepts and rules: Example: How do I save coral reefs?
Declarative Knowledge (Concepts-in-Use): Non-Gagnéan outcome	Concepts are an integrated network of subconcepts and the relations between them. This outcome reflects the process of learning concepts over time through applying them. Within this perspective, one’s knowledge of a concept begins as naïve theories and progresses over time as one tests and refines theory about what the concept is.
Context (Triarchic Model) Non-Gagnéan outcome	In what situation will learning be learning the knowledge and in what way, in a real situation would an authentic practitioner use this knowledge. In the game SWAT, when you play the game—or in America's Army—the activity you are doing in the game is like what a real person would be doing. It models reality.

Table 1.

Job Aid Listing Learning Outcomes and Brief Descriptions Culled from the Narrative.

(cont.)

Learning Outcome

Brief Description

Cognitive Strategy

A cognitive strategy is a control process the learner exerts over his/her personal learning to make learning more effective. This is different from a learning strategy in which someone else teaches a learner how to learn more effectively. The distinction between learning and cognitive strategies is the source of the strategy. If the source is the learner, it is a cognitive strategy. There are cognitive strategies for learning and cognitive strategies for thinking. A cognitive strategy is when the learner consciously selects and modifies ways of attending, learning, remembering, and thinking. It involves reflecting on how to apply a strategy to make use of one’s learning strengths and compensate for ones learning weakness. It is knowing one’s own characteristics as a learner and taking steps to be successful. Students use self-regulation to set their own goals and to estimate their success. This process is a component of cognitive strategies. Metacognition is also a cognitive strategy and deals with a person's cognitive, executive control function. With metacognition, he or she monitors personal learning and memory and decides when they are inadequate. Example: I enhance my comprehension when I read by taking notes, underlining, and discussing what I read with colleagues. Game example: In some games, you forget the key controls, so you lay out the instruction manual to help you.

Table 1.

Job Aid Listing Learning Outcomes and Brief Descriptions Culled from the Narrative.

(cont.)

Learning Outcome	Brief Description
Psychomotor skill Note: subset items are not distinct outcomes, but the team thinks they are important psychomotor aspects to consider within game learning.	Coordinated muscular movements combining knowledge of how and when to move (usually procedural knowledge) with movement skill.
Discrete	A single or a few steps with a beginning and end. Example: Using a key to unlock a door.
Discrete: Serial	A number of steps combined to form a major skill. Examples: Parking a car, writing words, tracking a target and shooting.
Continuous	Beginning and ending is subtle and determined by performer. Examples: Steering a car, dribbling a basketball.
Closed Skill	Psychomotor skill performed without active influence from the environment. Example: A lone player practicing at shooting baskets. Hitting the tennis ball against the backboard.
Open Skill	Psychomotor skill performed with active influence from the environment. Example: Playing a basketball or tennis game with other players.

Table 1.

Job Aid Listing Learning Outcomes and Brief Descriptions Culled from the Narrative.

(cont.)

Learning Outcome	Brief Description
Attitude	An internal state determining a choice of personal action/behavior, such as whether to approach or avoid a thing, person, or situation. Attitudes determine behavior. The stronger an attitude, the more it determines behavior. Attitudes have three components: Cognitive – Knowing how to execute a behavior. Affective – Knowing why to engage in a behavior. Behavioral – Engaging in the behavior. Game clarification: We are evaluating attitude toward the game content, and not toward the game itself. For example, a game might effect a disposition toward baseball or military action of science, technology, engineering, and mathematics (STEM).
Sociocultural/identity Non-Gagnéan` outcome	A way of defining oneself based upon the game OR the game and the other players. A player’s sense of self can be determined by the game—or the player’s sense of self as a player is determined by the game and interactions with other players.

The Learning Outcomes-Game Genre Matrix

COTF developed a five-point Likert rating scale to review the alignment between game genres and learning outcomes (see Table 2). COTF developed quasi-quantitative equivalents for qualitative rating scale determiners to provide a rule of thumb by which raters could gauge their responses. A rating of four, for example, would mean that a game genre provided opportunities to achieve the specified learning outcome during most of the gameplay. Most of the gameplay means that about half of the gameplay would support the learning outcome.

Table 2.

The Rating Scale for the Learning Outcomes-Game Genre Matrix.

General Guidelines

Spirit of the Scale

When applying this scale, think about the spirit in which it was developed. In general, the numbers go up based upon two things: the amount of gameplay supporting a type of learning outcome and the direct connection between the gameplay and the learning outcome. As a rule of thumband only thatyou may think of the terms in the scale qualifiers as percentages. The following percentages are general guidelines to help you interpret the meaning of the rating scale qualifiers. (Focus on the overall experience, not the clock.)

Meanings of Terms

Term Description

Minimal Less than about 5% of the gameplay.

Few From about 5% to about 10% of the gameplay.

Many From about 11% to about 49% of the gameplay.

Most Above about 50% of the gameplay.

Rating Scale

Rating Description

NR Not rated

0 Opportunity to achieve this learning outcome not available in genre gameplay.

1 Genre gameplay provides minimal opportunity to achieve this learning outcome.

2 Genre gameplay provides a few opportunities to achieve this learning outcome.

3 Genre gameplay provides many opportunities to achieve this learning outcome.

4 Most gameplay within genre provides opportunities to achieve this learning

outcome.

Table 1.

^{The Learning Outcomes-Game Genre Matrix}

Action

Adventure

Arcade

Fighting (Beat 'm ups)

Gambling

Puzzle

Racing/ Driving

Role-Playing

Simulation: Building

Simulation: Flight, etc.

Sports

Strategy

Average by Learning Outcome

Verbal information

Labels and names

Facts and lists

Organized discourse

Intellectual Skill: Procedural Knowledge

Discrimination

Concept

Concrete

Defined (Abstract)

Rules

Relational (if/then)

Procedural

Problem-solving

Declarative knowledge: Concepts-in-use

Triarchic model: Context

Cognitive strategy

Psychomotor skill (average of 5 skill types below)

Discrete

Discrete: serial

Continuous

Closed skill

Open skill

Attitude

Sociocultural/identity

Average by Genre2

¹ Scoring: 0=opportunity to achieve learning outcome not available in gameplay, 1= minimal opportunities, less than about 5% of gameplay, 2= a few opportunities, from 5% to about 10% of gameplay, 3 = many opportunities, from about 11% to about 49% of gameplay, 4= most gameplay provides opportunities, above about 50% of gameplay. ² genre averages calculated using average psychomotor skill.

Sample Team Narrative: Review of All Game Genres by Learning Outcomes

Verbal Information

Almost all game genres offer this learning opportunity for most of the gameplay except for the arcade and puzzle game types because you don’t need to know labels and names, facts and lists or have any organized discourse in order to play or complete these types of games. Basically, in an arcade or puzzle game, you move but don’t necessarily need to know about what it is that’s in front of you.

Intellectual Skill: Procedural Knowledge

Almost all are discriminatory for most of the gameplay and those rated high on concrete concepts are action, arcade, gambling and strategy because there are specific things you have to do in order to win the game or in order to work your way through the game. Abstract concepts are rated higher in the simulation and strategy types of games where you need to use your own thinking in order to move forward. Relational rules has a lot to do with action, simulations, sports, and strategy, and those were rated high because the action and the strategy games pose if/then situations all the time. For example, if you shoot and miss, you will get shot back at. Or in any simulations, if you don’t pull up on the throttle, then you will go down. But the procedural rules are found in the arcade games, fighting games, and strategy, where these game types are more based on procedural tasks. For example, you must do certain tasks in order to move forward through the board. There is an order/ hierarchy of tasks therefore making these genres procedural in nature. Problem-solving is found more in “higher level” games such as simulations, strategy, sports where thinking at a higher level must be incorporated in order to achieve the goal.

Declarative Knowledge: Concepts-in-Use

Again, this includes the more “thinking” games, such as simulation and strategy games where you need to put multiple concepts together to achieve a goal.

Triarchic Model: Context

This includes simulations and strategies and only certain action games where you can put together all your information and concepts to relate to, basically, “real life.”

Cognitive Strategy

These are games that would require or incorporate the need for you to think about the game, take notes, and pull out information about the game that would help in playing and winning the game. They might include games where you would need some type of help on the controls, or where you’d need to write things down to understand the game better. Basically, cognitive strategy games provide the learning opportunity where you are able to know your limitations and to compensate as needed. The game genres that would allow you to compensate and use this type of strategy and provide many opportunities would include all the higher end games. Those that provide few opportunities would include the lower end games, such as arcade, fighting, gambling, and puzzle games, where there is no learning curve involved in these game types. For these game types, you would just jump on and play (you don’t need to use a manual to play these games nor do you need an understanding of the game itself). If I want to be able to play a higher end game well, then I would usually need to go through an introduction and some type of manual of instructions, which would provide many opportunities to use a cognitive strategy.

Psychomotor

Games such as arcade games are based on the discrete psychomotor skills. That is most of the gameplay for this type of game. Discrete: serial psychomotor skills would also include the arcade and racing/driving because you’re adding multiple movements to accomplish your goal. Continuous skills include the arcade, fighting, or racing where you would have more of repetitive movements to accomplish you goal. Closed skills include the arcade, fighting, gambling, and puzzle games where you are not influenced at all by your surroundings and there is nothing else that comes into the play to influence the outcome, which is just the opposite of open skills games that include role-playing, simulations, sports, strategy games where your environment would come into the play, such as a flight simulator where you have daytime, nighttime, fog, rain, etc. These are games that would force you to consider certain factors when deciding on how you would go about accomplishing your task.

Attitude

These include games such as role-playing, simulations, sports, and strategy where you are more involved with a group, are provided more opportunity to have discussions within a group, or there are outside influences that need to be considered. Attitude games are unlike arcade games, puzzles, fighting, racing/driving, where these games are based on “raw” skill, and there’s no opportunity for any influence of a group or any outside influences to change attitude or even to influence your identity. The way that a role-playing game changes your attitude is through your identity. An action game or adventure game doesn’t change attitude because it doesn’t depend on any of the outside, cultural influences or provide for interaction.

Sociocultural/identity

These include genres such as role-playing, simulations, sports, strategies, and games where you would be involved with a group or games where outside influences need to be considered in the gameplay should enhance sociocultural/identity.

References

Gagné, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design. New York: Harcourt Brace Javanovich College Publishers.

Smith, P. L., & Ragan, T. J. (1993). Instructional design (1st ed.). New York: Merrill, an imprint of Macmillan Publishing Company.



Page 12 of 12