Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP) Survey Assessments Innovations Lab (SAIL)

For English Language Arts (ELA)

Collaboration and Inquiry Study 2017

OMB# 1850-0803 v. 176

October 2016

Table of Contents

1) Submittal-Related Information 1

2) Background and Study Rationale 1

3) Sampling and Recruitment Plans 3

4) Data Collection Process 5

5) Consultations Outside the Agency 7

6) Assurance of Confidentiality 7

7) Justification for Sensitive Questions 8

8) Estimate of Hourly Burden 8

9) Estimate of Costs for Paying Respondents 9

10) Costs to Federal Government 9

11) Schedule 9

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803), which provides for NCES to conduct various procedures (such as field tests and cognitive interviews) to test new methodologies, question types, or delivery methods to improve assessment instruments for future administrations of the National Assessment of Educational Progress (NAEP).

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute of Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to collect questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.

As part of NAEP’s development process, systems of delivery and assessment items are pretested on smaller numbers of respondents before they are administered to larger samples in pilot or operational administrations. The NAEP Survey Assessments Innovations Lab (SAIL) initiative is a research program set up to explore the potential value to NAEP in the development of innovative technology-based item types. This project makes use of the SAIL Virtual World for English Language Arts (ELA) Assessment.

This exploratory study seeks to develop and research ways of capturing aspects of collaboration as pairs of students work with each other in the context of a virtual world designed to assess students’ ability to gather, process, evaluate and synthesize online information from multiple sources and to construct arguments or explanations supported by information drawn from those sources. This project extends ongoing work from SAIL¹ to develop and research a virtual world that provides evidence of these practices (e.g., inquiry from multiple sources) within an ELA task. Using an iterative research and development process, we will examine how to capture students’ collaborative processes and how to use that evidence to support assessments of collaborative inquiry skills. As part of this process, we will develop a version of the ELA task that enables two students to work collaboratively, and then put the task and data capture systems through iterative testing activities, including play testing and tryouts. The iterative testing phases are especially important given unknown factors associated with using platforms for innovative technology-based items for assessments of collaborative inquiry.

Volume I describes the design, data collection, burden, cost, and schedules of the research activities for the aforementioned projects; Volume I Appendices provide recruitment and communication materials; and Volume II provides protocols and questions used in the research sessions.

Types of Research Methods

The following sections describe the different types of research methodologies that will be used. Given that SAIL projects involve technology-based platforms, all of the research activities will be conducted using technology (e.g., tablet or computer)².

Play Testing

In play testing, a process adapted from the game-design industry, a diverse set of students in groups of two will work through and discuss activities, problems, and tasks with one another, either in-person or through a virtual platform. An observer/facilitator will give overviews of the activities to students and provide guidance on what students should reflect on. Play testing will take place early in the test development process using preliminary versions of the virtual systems. The purpose of play testing is to gather student views on early versions of the interactive technology and begin to understand the range of ways in which students use them. The primary goal is to evaluate and refine the platform and activities. Accordingly, two phases of play testing will be conducted:

• A preliminary “face-to-face” implementation, in which students work together using a shared task interface to engage in collaboration, with prompts provided by the researcher; and

• A virtual, “remote” implementation, in which students collaborate via a digital platform with embedded instructions and prompts to manage the collaboration without the need for ongoing researcher intervention. This digital platform includes channels for text chat and audio/video chat; once the ELA task is implemented in the collaborative platform, all communication with the partner (talk aloud or text chat) will be mediated through the computer interface.

During play testing, students will be audio/video recorded as they work together to complete the collaborative task. Students will be encouraged to talk to each other about issues they confront as they work through the collaborative task, while observers note reactions to and potential problems with content or format (e.g., prompts eliciting the expected conversation/negotiation). Observers will query students to draw out student understandings (or misunderstandings), facilitate deeper reactions, or probe areas of possible confusion. Through play testing, researchers will be able to identify construct-irrelevant features in tasks, such as inaccessible language, technical issues with the collaborative platform, difficult interactions, ambiguous or ineffective prompts or instructions, or uninteresting or unfamiliar activities that result in poor student engagement. Play testing early in the research and development cycle allows for refinements to the system that can be tested in subsequent, more intensive tryouts. Notably, the face-to-face implementation will be used to inform and refine the design of the remote implementation, thus providing an opportunity to test out and refine collaborative prompts and modes of collaboration in the task, before embedding it in a digital software platform designed to support remote collaboration.

Play testing studies produce largely qualitative data in the form of verbalizations made by students while working through the task or in response to interviewer probes. Some informal observations of behavior are also gathered, given that typically a second observer (in addition to the interviewer) is involved. Behavioral observations may include such things as nonverbal indicators of affect, suggesting emotional states such as frustration or engagement, and interactions with the task, such as ineffectual or repeated actions suggesting misunderstanding or usability issues. Play testing sessions will conclude with oral questions and/or a brief survey instrument designed to evaluate students’ impressions of the task, the platform, and the collaboration itself (e.g., perceptions of the collaborator, or of the results of the collaboration). In addition to these data sources, we may also be able to extract preliminary logfile data from the system (i.e., time-stamped action logs of clickstream data revealing students’ sequences of actions taken in the environment), which would then also be analyzed and refined to prepare for collecting data logs during the subsequent tryout study.

Small-Scale Tryouts

During small-scale tryouts, students will be seated in separate rooms (or opposite sides of a large room) and will be asked to work uninterrupted in pairs, using the remote implementation in the digital collaborative platform, through a selected set of activities, problems, or tasks. The strength of using a tryout methodology on a small scale is that it allows data to be gathered about student responses and actions during normal, uninterrupted performance. The objective is to explore how students are thinking and what cognitive processes they are using as they work through tasks collaboratively and make use of the virtual collaboration platform. The primary goal at this stage is to understand how students think with the systems, and explore what kinds of evidence of student cognition and interaction the systems can elicit, under normal conditions (i.e., with minimal researcher intervention). In contrast to traditional cognitive interview techniques, where students may be instructed to “think aloud” or respond to in-the-moment questions from the interviewer (i.e., using verbal probing techniques to elicit verbalizations of moment-by-moment cognition), the tryout approach being proposed for this study will provide an opportunity to analyze students’ verbalizations as they occur (a) in conversations engaged in by the participants over the course of the task, and (b) in response to specific probing techniques that are built into the collaborative assessment (e.g., “Discuss with your partner and come to agreement before submitting your answer”)³. Note that, as described above, any communication between the partners (i.e., text chat or actual verbal communication) will be mediated by the digital collaborative platform (i.e., students will not sit and talk directly, but will use the features of the digital platform to communicate). Specifically, the remote implementation using the digital platform allows the potential for several streams of ongoing data capture, including an audio/video channel (i.e., audio/video chat), a written communication channel (i.e., text chat tool) built into the remote collaboration platform, in addition to any external audio/video capture and/or screen capture methods that might also be implemented to maintain a record of the tryout session. Thus, several sources of “process data” generated in the moment will be collected and analyzed in relation to the student team’s performance on the inquiry task. This verbal and behavioral data will be combined with students’ response data gathered from the logfile data collected within the task, in addition to other background variables, to conduct mixed qualitative and quantitative analyses of: (1) performance in the small-scale tryouts, and (2) relationships among performance, evidence of the collaborative process (e.g., behaviors revealed in the ongoing discourse between the partners), and team composition.

A similar combination of allowing students to verbalize their thought processes in an unconstrained way, supplemented by specific and targeted probes from an interviewer, has proven to be productive in previous NAEP developmental studies⁴. The tryout approach provides a small-scale snapshot of the ranges of responses and actions that the systems are meant to elicit, but with fewer resource implications than formal piloting. Previous experience, for example with the NAEP Technology Engineering Literacy Assessment⁵, shows that tryout-based insights are very informative, especially for the refinement of scoring rubrics (e.g., for examining, characterizing, and grouping the types of actions and responses that students provide and allocating appropriate scoring levels accordingly) and for finalizing or revising decisions about student actions that are to be captured during pilot or national implementations.

3. Sampling and Recruitment Plans

Play Testing Studies

Educational Testing Service (ETS) will administer the play testing sessions. Students will be recruited from near the ETS campus, in Princeton, New Jersey, for scheduling efficiency and flexibility. ETS will recruit students, representing a range of demographic groups, using existing ETS contacts with individual parents/guardians, and through the ETS intranet website to generate interest from ETS staff who may have friends or family who are eligible to participate. In some cases, ETS will directly contact parents/guardians of students who have previously participated in ETS research and who are known to fit the targeted range of grade level, gender, race/ethnicity, socioeconomic background, and district type (urban, suburban, rural). In other cases, information on the study may be posted online for parents to respond. During these communications, the parent/guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort, and the activities that it entails. Confirmation e-mails and/or letters will be sent to participants. Only after ETS has obtained written consent from the parent/guardian will the student be allowed to participate in a play testing session. See appendices A and I-O for recruitment, confirmation, consent, and thank you materials.

Up to 20 students total (i.e., 10 pairs) from grades 9 and 10 will be recruited, some of whom will be testing the collaborative features of the virtual world environment in person (face-to-face implementation) and others the use of the platform for online collaboration (remote implementation). A small sample is sufficient at the play testing stage given that the key purpose is to identify usability errors and other construct-irrelevant issues.⁶ Although the sample will likely include a mix of student characteristics, the analysis will not explicitly measure differences by those characteristics.

Small-scale Tryouts

Three options will be used for recruitment for small-scale tryouts, in order to ensure adequate participation:

• Option A: Will focus on recruiting school and/or community sites where tryouts can be conducted in an afterschool setting. For school sites, project staff from the University of Rhode Island or the University of Arizona (see Section 5) will recruit and conduct the tryouts at schools near the project staff location (e.g. near the University of Arizona). For community sites, ETS will recruit and conduct the tryouts at sites near the ETS Princeton, New Jersey campus. Staff will use existing local contacts to begin recruitment (see appendices B-E and O). Under this option, school and community sites will be selected that are deemed reasonably convenient for participants. Sites with existing afterschool programs will be targeted.

• Option B: Similar recruitment methods to those used for play testing will be used in Option B. Students will be recruited from near the ETS Princeton, New Jersey campus for scheduling efficiency and flexibility. ETS will recruit students using existing ETS contacts with teachers and staff at local schools and afterschool programs for students via emails or letters. Paper flyers and consent forms for students and parents will be distributed through teacher and staff contacts (see appendices F – O).

• Option C: Eureka Facts will recruit and conduct the tryouts at their facility under Option C. EurekaFacts will use various outreach methods to recruit students to participate, with the bulk of the recruitment conducted by telephone using targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach/contact methods and resources such as newspaper/Internet ads, outreach to community-based organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), social media, and mass media recruiting (such as postings on the EurekaFacts website). To ensure that the sample population is representative of different geographical areas (urban, rural, and suburban), students will be recruited from the District of Columbia, Maryland, Virginia, Delaware, and Southern Pennsylvania (see appendices F-O).

Project staff will aim to recruit students from the following demographic populations:

• A mix of race/ethnicity (Black, Asian, White, Hispanic, etc.);

• A mix of socioeconomic background; and

• A mix of urban/suburban/rural.

Although the sample will likely include a mix of student characteristics, the results will not explicitly measure differences by those characteristics, given that sample sizes will not provide enough statistical power to do so.

Interested participants will be screened to ensure that they meet the criteria for participation in the pretesting session (e.g., their parents/guardians have given consent and they are from the targeted demographic groups outlined above). When recruiting participants, staff will first speak to the parent/guardian of the interested minor before starting the screening process. The parent/guardian will be informed about the objectives and participation requirements of the data collection effort and about the activities it entails. After confirmation that participants are qualified, willing, and available to participate in the research project, they will receive a confirmation email/letter and/or phone call. Informed consent from parents will be obtained for all respondents. A minimum of 20 and a maximum of 40 students total from grades 9 and 10 will participate (i.e., between 10 and 20 pairs of students).

4. Data Collection Process

Play Testing

Play testing will take place at the ETS campus, in a dedicated research laboratory that is set up with recording equipment and working space for observers, facilitators, and one or more students (suitable for individual or small group sessions). Participants will first be welcomed and introduced to the facilitators/observers (assessment specialists, cognitive scientists, research assistants/associates, or task designers), and will be assured that their participation is voluntary (see Section 6). Observers will then give an overview of the planned activities to students and provide guidance about what students should focus on. Observers will take notes on what students say and on any noteworthy actions or behaviors occurring in the session. The sessions will be audio/video recorded (to capture students’ conversation and manner of interaction with the task and platform) and, where feasible, screen-capture will be used to record the actions occurring on the screen. These recordings can be replayed or analyzed later, to see how a given student progressed through the task and what actions they took. If logfile capture is available, all student actions with the system will also be recorded in a data file; this will not provide any identifiable data, since students will be coded with an anonymous ID number.

For the first round of play testing, students will work with their partners face-to-face, using a shared task interface (i.e., the scenario-based task will be delivered and team responses will be input using a single computer). For the second round of play testing, students will be located in separate rooms (or on opposite sides of a large room, so that students can only interact via the digital platform) and collaborate with their partners using the remote platform. For the most part, students will be allowed to explore and interact with the task and activities in pairs with little intrusion on the part of the observer. However, at a few strategic points, observers may introduce prompts to elicit collaborative activity (e.g., advising the student to discuss with their partner and reach an agreement before submitting their response), or questions meant to explore students’ reactions to the task, areas of confusion, and ways of thinking about answers to the questions in the tasks and/or items. Examples of such questions are:

• Is this difficult? Why or why not?

• Do you find the problem in this task interesting – why or why not?

• Do the prompt(s) used in the system help you to [think together/work] with your partner or not?

Prior to each play testing session, ETS staff may identify some key focus areas for activity or for the system that students will be using. If students do not provide sufficient comments on targeted parts during the interaction with the task, an observer may ask students if they had any thoughts about the particular sections, using questions such as those described above, but focused on specific places or issues in the task or activities or system. Observers may pose additional targeted questions to students upon completing the task, for example, questions about the perceived quality of the collaboration (e.g., “Do you believe that your collaboration was successful? Why or why not?”) or the feasibility of completing the activities (“Did you have enough time to complete the activities? Do you think you could have done better at the task if you were given more time?”). Students will also complete a brief questionnaire regarding demographic information, computer experience, familiarity with the research skills tested in the assessment, and other questions about their relevant prior experiences, perspectives, and knowledge. See Volume II, Part B for the protocol used in the play testing studies.

Analysis Plan

Since play testing is a more informal process that generates relatively unstructured information, no formal quantitative analyses of these data will be performed. However, the data will be compiled for the purpose of qualitative analyses, which will seek to pick out themes or individual observations that are important for developing the system and tasks going forward.

The general analysis approach will be to compile the different types of data to facilitate identification of patterns of responses for specific tasks or activities, such as patterns of responses or behaviors, or types of actions observed from students at specific points in a given task. This overall approach will help to ensure that the data are analyzed in a way that is thorough, systematic, and that will enhance identification of problems with the systems or tasks and provide recommendations for addressing those problems (e.g., modifications to instructions, prompts, or system design).

Small-Scale Tryouts

All sessions will be conducted either during afterschool hours in a school or community site setting (Option A) or in a research facility (Options B and C). Pairs of students seated either in separate rooms (or in distant locations within a large room so that they cannot directly interact) will work independently to complete the collaborative inquiry task using the remote implementation, which allows for audio/video and chat-based communication among the partners throughout the task; these data streams will be captured and stored for analysis. In addition, we may screen capture student actions as they appear on screen using software such as CamStudio^TM. The core strength of such screen recording capabilities is their facility for capturing students’ interactive behaviors as they happen, while one or more observers can later record text comments that are time-locked to each student’s actions observed in a logfile. Students’ actions with the system will also be automatically recorded in a logfile. Where there are discrete actions to be captured (e.g., button presses, taps on a tablet screen), the logfile will capture and identify interactions with timestamps.

In the small-scale tryouts there will be no think-aloud or verbal probing component requiring direct researcher intervention, although verbal data will be collected from students’ unfolding conversations as they work through the task, and in response to specific prompts for discussion that will be embedded in the task. Therefore, audio and video data capture will also be part of the tryout study. Students will complete brief pre-test measures to elicit information about their relevant prior experiences, perspectives, and knowledge and other variables of interest related to evaluating the quality of the collaboration and the composition of the student pairs. Following use of the collaborative inquiry task, students will be asked to complete a questionnaire about their experiences with the task and the collaborative activity, and may be asked a general evaluative question to get their overall impressions of tasks or activities with the system after they have completed the session. The goal of tryouts is to gather authentic, uncontaminated task performance and interaction data. Therefore, student pairs will work through tasks and selected items at their own pace and without interruption from researchers/observers. This protocol is described in Volume II, Part C.

Analysis Plan

Student responses to items will be compiled into spreadsheets to allow quantitative and descriptive analyses of the performance data. Screen captures, video/audio data, and observation notes will be used for qualitative analysis of behavioral data to characterize the range of behaviors observed across various pairs of students (i.e., more versus less successful collaborators). Once a coding scheme is established and applied to relevant segments of verbal data (e.g., conversational dialogue, chat box input), a basic quantitative analysis will provide frequency counts and, where relevant, sequence information for different behaviors or actions observed from each pair of students. The log files will be analyzed to examine frequencies, categories, and orders of actions as appropriate for the research and development goals at this stage of the project; much of this analysis will use descriptive analyses, but some inferential statistical tests may also be used. The compiled dataset (including task performance data, verbalization data, and background characteristics/other variables as obtained from questionnaire measures) can be submitted to cognitive and psychometric analyses that focus on examining (1) how student pairs perform on the collaborative inquiry assessment; (2) relationships among student/pair/task characteristics and actions and team performance, and (3) any apparent effects of presentation mode (i.e., features of the remote platform) on the collaborative processes observed.

5. Consultations Outside the Agency

Educational Testing Service (ETS) is the Item Development and Data, Analysis, and Reporting contractor for NAEP, who will develop prompts and modes of collaboration for the items and activities in the Virtual World task and will analyze the results. For play testing and the small-scale tryout studies to be conducted at community sites in the Princeton, NJ area (Option A) or at the ETS Princeton campus (Option B), ETS staff will perform recruitment and data collection activities, and carry out the necessary research studies. Additionally, ETS contracted an interdisciplinary team of investigators across several institutions, including the University of Rhode Island, the University of Arizona, and the University of Jyväskylä in Finland.⁷ This team will participate in recruitment activities and conducting of small-scale tryouts (under Option A), and will collaborate with ETS on data analysis. Lastly, EurekaFacts will serve as a subcontractor to ETS under small-scale tryout Option C, recruiting participants for small-scale tryouts and administering the tryout studies at their facilities.

6. Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that without the permission of their parent or guardian, their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)]. Participants will also be notified that, with their parent or guardian’s permission, some responses or clips from videos may be selected for use in research reports or presentations.

Written consent will be obtained from parents or legal guardians of students who are under the age of 18. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study, and will be destroyed after the final report is completed. While sessions will be audio/video recorded, the only identification included with the files will be the unique ID assigned to each participant by the interviewer. The recorded files will be secured for the duration of the study and will be destroyed when the research is complete.

7. Justification for Sensitive Questions

Throughout the item and task development process, as well as the process of developing interview protocols, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have attempted to identify and minimize potential bias in questions.

8. Estimate of Hourly Burden

The estimated burden for play testing recruitment assumes attrition throughout the process⁸. The anticipated total number of student participants for play testing is 20 (i.e., a maximum of 10 pairs). Play testing sessions are expected to last up to 150 minutes for all students. The estimated burden for small-scale tryouts recruitment also assumes attrition throughout the process⁹. The anticipated number of student participants who will complete the collaborative inquiry task and related questionnaires is a minimum of 20 and a maximum of 40 (i.e., 10–20 pairs of students). Tryout sessions are expected to last up to 150 minutes for all students.

Table 1. Specific Burden for Play Testing studies¹⁰ and Tryouts

Respondents	Hours per respondent	Number of respondents	Number of responses	Total hours (rounded up)
Parent or Legal Guardian for Student Recruitment (Play testing)
Initial contact	0.05	250	250	13
Completion of online screening form or phone screening	0.15	50*	50	8
Consent form completion and return	0.13	25*	25	4
Confirmation to parent via email or letter	0.05	25*	25	2
Participation (Play testing)
Students	2.5	20	20	50
Sub-Total		270	370	77
Student Recruitment via Teachers and Staff (where appropriate)(Tryouts)
Initial contact with staff: e-mail/phone calls	0.5	8	8	4
Coordination and planning with participating schools/organizations	5	4	4	20
Parent or Legal Guardian for Student Recruitment (Tryouts)
Initial contact	0.05	208	208	11
Completion of online screening form or phone screening (where appropriate)	0.15	83*	83	13
Consent form completion and return	0.13	50*	50	7
Confirmation to parent via email or letter (where appropriate)	0.05	50*	50	3
Participation (Tryouts)
Students	2.5	40	40	100
Sub-Total		260	443	158
Total		530	813	235

* Subset of initial contact group, not double counted in the total number of respondents.

9. Estimate of Costs for Paying Respondents

For play testing and small-scale tryout studies, to encourage participation and thank participants for their time and effort, a $25 gift card will be offered to each participating student, plus a $25 gift card to a parent or legal guardian bringing the student to and from the testing site (if appropriate). If a school or community site is used for the study, the school or organization will receive a gift card equivalent to $10 per student participating in the study.

10. Costs to Federal Government

The estimated cost to the federal government for the virtual world play testing and small scale tryouts described in this submittal, including designing, preparing for, and conducting play testing sessions, recruitment, incentive costs, data collection, and summary of findings is $135,000.

11. Schedule

Table 2 depicts the high-level schedule for the various activities. Each activity includes recruitment, data collection, analyses, and reports.

Table 2. High-Level Schedule of Milestones

Activity	Dates
Play Testing	November 2016 – September 2017
Tryouts	August 2017 – December 2017

1 Sparks, J.R. (2014, November). Assessing students’ inquiry and information gathering skills in an immersive environment. Invited presentation to NCES/NAEP SAIL Network Meeting, Washington, DC.

2 For the ease of description, the term “computer” has been used in the recruitment materials.

3 This method is sometimes referred to as an “interaction approach” to data collection (see Miyake, 1986).

4 For example, NAEP Science Pretesting Activities (OMB #1850-0803 v.73, October 2012) and NAEP 2011 Cognitive Interview Studies of NAEP Cognitive Items (OMB #1850-0803 v.45, March 2011).

5 Technology and Engineering Literacy Pre-Assessment Studies: Tryout and Usability Studies (OMB #1850-0803 v.66, February 2012).

6 Nielson, J. (1994). Estimating the number of subjects needed for a think aloud test. In J. Human-computer Studies. 41, 385-397. Available at: http://www.idemployee.id.tue.nl/g.w.m.rauterberg/lecturenotes/DG308%20DID/nielsen-1994.pdf

7 While staff from the University of Jyväskylä will participate in the planning and analysis of the studies, they will not administer any tryout sessions. The University of Rhode Island and the University of Arizona will administer all tryout sessions under Option A.

8 Assumptions for approximate attrition rates are 80 percent from initial contact to screening form completion, 50 percent from submission of screening form to confirmation, and 20 percent from confirmation to participation.

9 Assumptions for approximate attrition rates are 60 percent from initial contact to screening form completion (if appropriate for the Option of recruitment used), 40 percent from submission of screening form to confirmation (if appropriate for the Option of recruitment used), and 20 percent from confirmation to participation.

10 The burden estimates in this table reflect the maximum burden for recruitment if students do not participate in multiple play testing sessions.