NAEP Website Usability Study 2013 Volume I

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB # 1880-0542) that provides for the Department to carry out collections of routine customer feedback.

Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute for Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to also collect survey questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.

The NAEP public site, http://nces.ed.gov/nationsreportcard/, is a key means by which NCES disseminates information to the general public. The public site is used by public policy officials, educators, parents, and other general audiences to access NCES data and learn about education progress and the assessments in general.

NCES has designated Fulcrum IT (see Section 5), the web/technology development, operations and maintenance contractor for NAEP, to conduct a study to inform the redesign of the public website. This redesign is geared towards bringing more users to the site, and enhancing the users’ experience while on the site. The study described in this submission is aimed at assisting in that effort, and is based on the following criteria:

Improving the visual design of the website utilizing all available modern technology;
Improving the way the content of the website is structured so that users have an easier time finding information they need; and
Improving the Public Relations aspect of the website to help users get a clearer sense of NAEP’s mission and goals.

Recruitment

The study will include 40 participants representing groups of people that have used or might have reason to use the NAEP public website. Past usability studies conducted by Fulcrum IT have sustained participation rates of 75% - 100% of recruited potential participants. To ensure sufficient numbers of participants for each population profile, we will be recruiting roughly 150% of the desired targets. This means that 60 potential participants will be contacted for participation in the study. If more than 5 people for a group agree to participate, only the first 5 will be used for the first phase of the study. The remainder will be asked to participate if they are needed to replace someone who drops out of the study before it is completed. Each target group, composed of 5 individuals, along with the specific recruiting strategy, is listed below.

Researchers (5) & Media (5)

Researchers and media are included to determine the best way to package information from NAEP studies to facilitate the use of these data in media stories and research articles. Qualified participants are those who have either used NCES data in the past or written articles that may have used NCES data. Prospective participants will be identified through online searches of media and scholarly journals, and through referrals made by NCES and NAEP contractors who are familiar with authors who meet the requirements stated above. Individuals will then be contacted directly and asked to participate (see appendix A1).

Grade 4 students (5), grade 8 students (5), and one parent of each student (10)

Students who could be involved in a NAEP assessment and their parents are included. This is to ensure that pre- and post-assessment information that could motivate and facilitate participation in NAEP assessments is readily available on the NAEP public website. Due to confidentiality issues, we will not be using actual NAEP assessment participants in this usability study. Rather, we will include participants who fit the profile of potential public site users.

Fulcrum IT has developed a database of teachers and other adults who have expressed a willingness to participate in usability studies. They will be contacted (see appendix A5) and asked to send Fulcrum IT the names and contact information of families of 4th and 8th graders that might be willing to participate in this usability study. Because this study will occur in the summer, some student participants will be “rising” 5^th and 9^th-graders, while others will be rising 4^th and 8^th graders. Students will not be recruited directly. Parent-child pairs will be recruited through direct contact with the parent. As with adult participant tasks, student tasks will be scheduled at mutually convenient times, through conversations with the students’ parents (see appendix A2 for parent/student recruitment e-mail).

Generally NAEP includes a broad cross-section of racial and socio-economic groups in their studies. However, the focus of this usability study is the website, rather than the students, so in that sense, race and SES are far less important than familiarity with web conventions. Though students with limited internet experience are not easy to find, a question about internet access is included in the student interview. To the degree that there is variability in student internet experience, this information will help us determine which website elements may be barriers to using the site for some children.

Education Policymakers (5)

Education policymakers are included to help determine the best way to package NAEP assessment results and best practices recommendations for use in shaping education policy. NAEP State Coordinators (see Section 5) are a crucial link in this information pathway and will be used as a source of referrals to state education policymakers (see appendix A3 for policymaker recruitment e-mail).

Educators (5)

Teachers and school principals are included to determine how information about best practices and novel approaches to assessment can be made relevant to them. Fulcrum IT has developed a database of educators who have expressed a willingness to participate in usability studies. Invitations to participate will be extended to these contacts (see appendix A4 for educator recruitment e-mail).

All participants

Participants who respond to our e-mail/phone requests will be sent a follow-up e-mail (see appendix B) regarding participation and the required consent forms (appendices F, G, and H). Once consent forms are received, a verification e-mail (see appendix C) will be sent to arrange a mutually convenient time for the phone interview portion of the study.

Study design and data collection

There will be three separate user testing phases over the course of the study. This will take place over the course of several months, as each phase of the study will be conducted using content created during different phases of the development process. The tasks and analyses for each phase are described below:

Phase 1 - User Interviews

Interviews will be conducted over the phone and will take roughly 40 minutes for adults, and 10 minutes for students. Mutually convenient times will be set up by phone or over e-mail, and a research associate will place the call to the participant at the agreed-upon time. Interview questions (see Volume II, section 1) will focus on how participants have used the site in the past, what would be beneficial to them in the future, and what other sources they use to get similar information (if any).

Interviews will generally follow the scripted questions found in the Interview Questionnaire. However, interviewers will ask follow-up questions if participants bring up new areas of interest. Since these areas of interest cannot be predicted, these particular questions cannot be scripted. However, interviewers will ask questions only to gain a clear understanding of what the participants are trying to convey in the interview.

This phase is designed to obtain qualitative information from participants about what features would make them more likely to use the NAEP public website. The interview results will consist of lists of desired features, and will be used by the development team to guide the website design process.

Phase 2 - Card Sorting

Card Sorting, an activity involving grouping information from a list of pieces of web content, offers keen insight into how users think a website should be organized. Each participant will be presented with a list of website content categories that is deemed to be relevant to the user group they represent. These content categories will be developed by analyzing the information gained in the user interviews and comparing that to the current content of the website. Content categories will be similar to those shown in the sample content list (Volume II, section 2), but may be different, reflecting the new information obtained from participants in the User Interviews portion of the study. The participants will then sort the “cards” according to perceived similarity to one another.

Card sorting will be completed by participants using the internet device of their choice (computer laptop, tablet, etc.). This activity is likely to take around 45 minutes, and can be completed over the course of a week at the convenience of the participants. Instructions and procedural links will be sent over e-mail (see appendix D). This task will not be conducted using the student group of participants.

This task is designed to obtain perceived categories of information that will be available on the website. Card sorting data from all participants will be combined using a cluster analysis. This analysis displays all the content of the website in a chart that uses proximity to convey perceived similarity. This similarity chart will then inform the organization of information on the website.

Phase 3 - User Testing

User testing will involve all participants, and will consist of completion of assigned tasks on a pre-release version of the NAEP public website. While phases 1 and 2 are to help guide the development of website interaction, this phase will take place after the development team has created working interaction elements for users to test. A working prototype of the website is expected to be ready for user testing roughly 2 months after completion of the Card Sorting task in phase 2 of the usability study. Instructions and related information will be sent over e-mail (see appendix E).

The tasks participants will be asked to perform will be determined by the specific interaction elements created for this website, many of which will stem directly from information obtained in the user interviews to be performed in the first phase of this study. The directions and tasks presented in Volume II section 3 are similar to those that users will be presented in this phase. After each set of tasks is completed, participants will be asked to answer an ease-of-use question about those tasks. They will also be asked for any comments they have about the task they just completed. All comments will be recorded and provided to the developers to help guide development.

This task can be performed on any computer to which participants have access. Teleconference software will be used so that the research associate can talk to participants during completion of their tasks and record their computer screens and audio over the internet. Fulcrum IT uses Cisco’s WebEx for web screen sharing conference calls. The participants will login on the WebEx website to join the teleconference. The WebEx teleconference will be recorded off the researcher’s computer using Techsmith’s Morae screen recording software, which captures audio, video, on-screen activity, and keyboard/mouse input during a research session.

At no time will there be any video recorded of the participants or their surroundings. These sessions will take roughly 90 minutes for adults and 30 minutes for students (who will have far fewer tasks to complete) and will be conducted at a mutually convenient time agreed upon via e-mail or phone.

This task is designed to determine if the redesigned website meets usability targets set for this effort. Users’ success or difficulty in completing assigned tasks will be analyzed to determine which information or control elements are missing or insufficient to allow successful completion of anticipated user tasks. As with all sessions for this study, the participants are not being tested or graded. It is the website that is being evaluated, and all results will be used to improve the website, rather than to make any inferences about the participants’ abilities.

Results will be analyzed in terms of percent of participants who successfully completed each task and task element. An example finding would be: “40% of participants found and clicked on the PRINT button without assistance.” This finding would be used to determine that the PRINT button needed to be made more visible to users in order to be used successfully 100% of the time.

After completion of the user testing task, adult participants will be directed to the usability website to fill out the 10-item standardized System Usability Scale questionnaire (see Volume II, section 4).

Consultations outside the Agency

Fulcrum IT Services Company LLC (Fulcrum IT)

Fulcrum IT is the NAEP contractor responsible for the development and ongoing support of NAEP computer-based assessments for NCES.

NAEP State Coordinators

The NAEP State Coordinator serves as the liaison between the state education agency and NAEP, coordinating NAEP activities in his or her state.

Paying Respondents

Adult participants will be given an e-gift card (American Express) that can be used for purchases from any vendor that accepts American Express. The amount of the gift card will be contingent on the number of phases in which the person participates, at the rate of $25 per phase. This allows for fair compensation for participants who complete only one or two of the phases. Students participating in phase 3 will be compensated with a similar e-gift card in the amount of $20, which is commensurate with their reduced time burden relative to the adult participants. Due to the very short time burden for students in phase 1, participation in that phase will not be compensated.

Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].

Written consent will be obtained from legal guardians of minor students and the various categories of adult participants before interviews are conducted. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study, and will be destroyed after the final report is released.

The user testing task (phase 3) will be recorded. The only identification included on the files will be the participant ID. In addition, the researchers may wish to use some recordings in scientific presentations as a means of describing the methodology used for this study. Any recordings used would not be paired with the participant’s name, address, or likeness.

Justification for Sensitive Questions

No sensitive questions will be asked.

Estimate of Hourly burden

Estimated hourly burden for the participants is distributed among the three phases according to table 1 below.

Table 1: Estimate of Hourly Burden

Task	Number of Participants	Individual Participant Burden	Total Burden
Referral e-mail (sent to teachers to obtain parent/student leads; appendix A5)	20 Adults	15 minutes	5 hours
Initial recruitment e-mails (appendices A1-A4)	50 Adults	5 minutes	4 hours
Follow-up contact for Instructions/Consent	30* Adults	15 minutes	8 hours
Phase 1- Interviews	30* Adults 10 Students	40 minutes 10 minutes	20 hours 2 hours
Phase 2 - Card Sorting	30* Adults	45 minutes	23 hours
Phase 3 - User Testing	30* Adults 10* Students	90 minutes 30 minutes	45 hours 5 hours
Total	80 Participants		112 Hours

*Subset of recruited group

Cost to federal government

The following table provides the overall project cost estimates.

Table 3: Estimate of Costs

Activity	Provider	Estimated Cost
Recruit participants; develop measurement instruments; perform the testing; record, analyze, and report the results	Fulcrum IT	$39,168.21
Participant Compensation (including associated fees)	Fulcrum IT	$3,200
Total		$42,368.21

Project Schedule

Table 4: Schedule

Date	Event
July-August 2013	Recruit participants
September-October 2013	Prepare for and conduct Phases 1 & 2
December 2013-February 2014	Prepare for and conduct Phase 3
March 2014	Final Study Report to NCES

Recruitment e-mails

The following e-mails will be sent to different user groups in an effort to recruit participants for the 2013 National Assessment of Educational Progress (NAEP) public website redesign usability study (http://nces.ed.gov/nationsreportcard/).

A1. Media/Researchers

Subject line: NAEP Usability Study Date:

Text:

Greetings! The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is redesigning the National Assessment of Educational Progress (NAEP) website (http://nces.ed.gov/nationsreportcard), which provides information to thousands of users about the state of education in the U.S. We are seeking qualified participants for a paid usability study to help us determine what website content would be of interest to researchers and the media.

Would you be willing to participate in a usability study to help in these efforts? The study consists of three tasks that will be performed on the internet and over the phone over the course of the next few months.

The first task will be a phone conversation where we get your opinions about the type of information that might be useful to you. If you have used the NAEP website before, we will also want to know what you like or don’t like about the site. This activity will take roughly 40 minutes.
The second task will be an activity known as “card sorting.” We will give you a list of content and you will group them in a way that makes the most logical sense to you. This activity will take roughly 45 minutes and can be performed online at your convenience.
The third task will be an activity where you try out the new website and tell us what you think of it. This phase will be conducted as a teleconference, and will take 60-90 minutes.

If you think you might be interested in participating, please reply to this e-mail or give us a call.

Thank you!

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

A2. Parents and Students

Subject line: NAEP Usability Study Date:

Text:

Greetings! The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is redesigning the National Assessment of Educational Progress (NAEP) website (http://nces.ed.gov/nationsreportcard), which provides information to thousands of parents just like you about our assessment process. We are seeking participants for a paid usability study to help us determine how we can provide the most useful information to parents and students.

Would you (and your child) be willing to participate in a usability study to help us in these efforts? The study consists of three tasks that will be performed on the internet and over the phone over the course of the next few months.

The first task will be a phone conversation where we get your opinions about the type of information that might be useful to you. If you have used the NAEP website before, we will also want to know what you like or don’t like about the site. This activity will take approximately 10 minutes for your child and 40 minutes for you.
The second task will be an activity known as “card sorting.” We will give you a list of content and you will group them in a way that makes the most logical sense to you. This activity will take roughly 45 minutes and can be performed online at your convenience. Note: your child will not be participating in this portion of the study.
The third task will be an activity where you try out the new website and tell us what you think of it. This phase will be conducted as a teleconference, and will take roughly 30 minutes for your child and 60-90 minutes for you.

As compensation for your participation, you will receive a $75 e-gift card for completing all three phases of the study. Gift amounts will be prorated at $25 per session if you are unable to participate throughout the entire study. Student participants will also receive their own $20 e-gift card for their participation in the study. If you think you might be interested in participating, please reply to this e-mail or give us a call.

Thank you!

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

A3. Education Policymakers

Subject line: NAEP Usability Study Date:

Text:

Greetings! The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is redesigning the National Assessment of Educational Progress (NAEP) website (http://nces.ed.gov/nationsreportcard), which providesinformation to thousands of users about the state of education in the U.S. We are seeking interested participants for a paid usability study to help us determine what website content would be of interest to education policymakers.

Would you be willing to participate in a usability study to help us in these efforts? The study consists of three tasks that will be performed on the internet and over the phone over the course of the next few months.

The first task will be a phone conversation where we get your opinions about the type of information that might be useful to you. If you have used the NAEP website before, we will also want to know what you like or don’t like about the site.
The second task will be an activity known as “card sorting.” We will give you a list of content and you will group them in a way that makes the most logical sense to you. This activity will take roughly 45 minutes and can be performed online at your convenience.
The third task will be an activity where you try out the new website and tell us what you think of it. This phase will be conducted as a teleconference, and will take 60-90 minutes.

If you think you might be interested in participating, please reply to this e-mail or give us a call.

Thank you!

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

A4. Educators

Subject line: NAEP Usability Study Date:

Text:

Greetings! The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is redesigning the National Assessment of Educational Progress (NAEP) website (http://nces.ed.gov/nationsreportcard), which provides information to thousands of users about the state of education in the U.S. We are seeking interested participants for a paid usability study to help us determine what website content would be of interest to educators like you.

The first task will be a phone conversation where we get your opinions about the type of information that might be useful to you. If you have used the NAEP website before, we will also want to know what you like or don’t like about the site.
The second task will be an activity known as “card sorting.” We will give you a list of content and you will group them in a way that makes the most logical sense to you. This activity will take roughly 45 minutes and can be performed online at your convenience.
The third task will be an activity where you try out the new website and tell us what you think of it. This phase will be conducted as a teleconference, and will take 60-90 minutes.

If you think you might be interested in participating, please reply to this e-mail or give us a call.

Thank you!

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

A5: Educator Referral E-mail

Subject line: NAEP Usability Study Date:

Text:

We need your help in recruiting participants. We are recruiting 4th and 8th grade students and their parents to help us determine how we can provide the most useful information to parents and students. They will be asked to participate in three tasks during the course of the study:

The first task will be a phone conversation where we get opinions about the type of information that might be useful to users. If users are familiar with NAEP, we will also ask them what they like and don’t like about the website.

The second task will be an activity known as “card sorting.” We will give participants a list of content and they will group it in a way that makes the most logical sense to them. This activity will take roughly 45 minutes and can be performed online at their convenience. Note: Students will not be used for this phase.

The third task will be an activity where they try out the new website and tell us what they think of it. This phase will be conducted as a teleconference, and will take 60-90 minutes for adults and 30 minutes for children.

As compensation for their participation, parent participants will receive a $75 e-gift card for completing all three phases of the study. Gift amounts will be prorated at $25 per session if they are unable to participate throughout the entire study. Student participants will receive a $20 e-gift card for their participation.

If you think know of anyone that may be interested in assisting with this study, please reply to this e-mail or give us a call. You can also give our information directly to those you feel may be willing to participate.

Thank you!

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

Follow up contact phase of recruitment

People who e-mail a response showing interest in participating will be sent the following e-mail. People who respond by phone will be thanked for their interest and told that they will be receiving the following e-mail.

Thank you for your interest in participating in the NAEP website usability study. As the usability team, our role is to make sure that the website is easy to use and meets the needs of the people who visit it. Your participation in this usability study is one way how we do that and we appreciate your willingness to participate.

We have attached the following forms to this e-mail:

Research Participant Informed Consent form
Audio/Video Recording Release form
Student Informed Consent form (sent only to parent/student participants)

Please read the attached documents carefully. If you have any questions about the attached documents, don’t hesitate to call or e-mail us for answers or clarification. If you agree to participate in this study, please sign the forms and return them to us.

The forms can be e-mailed, faxed or sent via postal service to:

Usability Team

Fulcrum IT Services

5870 Trinity Parkway, suite 400

Centreville, VA 20120

Fax: 703-543-2978

E-mail: [email protected]

Once we receive your signed forms, we will e-mail you to set up a time for the first task in this study, the telephone interview.

Thank you again for your interest in helping the National Center for Education Statistics with this important task.

Sincerely,

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

Phase I Instruction e-mail

This e-mail will be sent to participants after their signed consent and release forms have been received by Fulcrum IT.

Dear ___________,

Thank you for agreeing to participate in the NAEP website usability study. As indicated in our previous contact with you, there are three tasks you need to complete in this study. The first task is a phone interview, which we would like to complete within the next week. The phone conversation should last 20-40 minutes (10 for children) and can be conducted at any time that is convenient for you.

The second phase won’t be started for another couple of weeks. We will send you an e-mail explaining the phase two task after completion of the telephone interview.

Please let us know at your earliest convenience when you would like to be contacted for the interview and thanks again for your participation.

Sincerely,

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

Phase II Instruction e-mail

Dear ___________,

Thank you for agreeing to participate in the NAEP website usability study. As indicated in our previous contact with you, there are three tasks for you to complete in this study. We are now ready to begin the next phase, where you will be completing a card sorting task. We have set up a web page for the card sorting task. This task can be done on your own time by visiting the following link:

[URL]

All of the instructions for completing the task are on the web page. If anything is unclear, please use the online help or contact one of us for assistance. The card sorting task will take roughly 40 minutes. When you have completed the task, we will be automatically notified by our system. This task needs to be completed by [7 days from e-mail date]. If you have not begun the task by [5 days from e-mail date] we will send you this e-mail again as a gentle reminder. If you will not be able to complete this task in the allotted time (or at all), please let us know at your earliest convenience.

Thank you again for your participation in the NAEP website usability study.

Sincerely,

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

Phase III Instruction e-mail

Dear ___________,

Thank you for agreeing to participate in the NAEP website usability study. As indicated in our previous contact with you, there are three tasks for you to complete in this study. We are now ready to begin the final phase, which will be conducted using a teleconference website called WebEx. We need to schedule a 90-minute block of time in which to complete this task. At the agreed-upon time, we will both login to the WebEx site. All the instructions you will need to join the teleconference are on the website.

During the teleconference we will be able to talk to one another and I will be able to see your computer screen. In addition, I will be recording your computer screen and our conversation throughout the teleconference. The task will involve you going to our website to find information to answer some questions I will be asking you. We are not testing your ability to answer these questions – we just want to see if our website makes it easy to find the information it contains. After you find (or don’t find) the information, I will ask you to rate how easy that task was to complete.

I have attached a document to this e-mail that contains all the questions I will be asking as well as the rating scale for telling me how easy the task was to complete. Please call or e-mail me with a good time for us to perform this task. After I have scheduled the teleconference, you will get an automated e-mail from WebEx with the link for the web page to join the conference.

If you have any questions or will not be able to complete this task in the near future (or at all), please contact me at your earliest convenience.

Thank you again for your participation in the NAEP website usability study.

Sincerely,

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

Audio & video release form

As part of the National Association of Educational Progress’ efforts to redesign the public website (http://nces.ed.gov/nationsreportcard/), participants will have their voice and screen interactions recorded by Fulcrum IT.

The purpose of this form is to inform you that audio and computer screen recordings will be made of each participant during phase 3 of this study. By signing this form, you are granting permission for your computer screen and teleconference conversation during phase 3 of this study to be recorded. In addition, the researchers may wish to use some recordings in scientific presentations as a means of describing the methodology used for this study. Any recordings used would not be paired with the participant’s name, address, or likeness. If you would be willing to allow your recording to be used for that purpose, please indicate by initialing the appropriate box below. Otherwise, your recording will be deleted upon completion of the study and summarization of the results. Granting or denying use of your recording will have no bearing on your participation in the study, or the compensation you will receive for your participation.

Initials
	I do grant permission for my screen and audio recording to be used in scientific presentations about this usability study.
	I do NOT grant permission for my screen and audio recording to be used after completion of the study, at which point my recording will be deleted permanently.

By signing this form I acknowledge that I have completely read, and fully understand the above release, and agree to the stipulations listed therein.

Full Name:
Phone:
E-mail:
Print Name:
Signature:		Date:
Parent Name: (if participant is a minor)
Parent Signature:

Student informed consent form

Title of Study

National Association of Educational Progress (NAEP) Small Scale Usability Study to Redesign the NAEP Public Website (http://nces.ed.gov/nationsreportcard/)

Description of Study

The National Association of Educational Progress (NAEP) is redesigning its public website in order to make the site more user friendly, and to better relay NAEP’s mission and goals to the public. This study will assist in that effort by incorporating the students’ feedback into the redesign process.

For student participants, the study will take place in two parts. Here is a brief description of each part:

Part 1: The student will answer a few simple questions based on a hypothetical scenario that their school is about to participate in a NAEP assessment. The students will be asked what things they would like to see on the public website that they would find useful as an assessment participant. This interview will take roughly 10 minutes.
Part 2: The student will log in to a teleconference with a research associate. They will then use a browser to interact with the NAEP public website. They will be asked to look up some information and perform basic tasks on the website, while talking through their process with the research associate. This conversation as well as the student’s computer screen activity will be recorded for later analysis. This task will take roughly 30 minutes.

Compensation

Student participants will receive compensation of $20 in the form of an American Express e-gift card upon completion of the 2^nd part of this study.

Confidentiality and Anonymity

The researchers will collect no identifying data on participants. Each participant will be identified only by a participant number during and after the study. Responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

Voluntary Nature of the Study

Participation in this study is strictly voluntary. Participants may withdraw from participation at any time without consequence.

Contacts and Questions

Participants are given opportunities to ask questions before, during, and after the study. If you have any questions about the process, please contact a member of the usability team at Fulcrum IT.

Gregory Anderson, Senior Usability Expert

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer

703-543-2878

If you have concerns about the conduct of the research associates during the course of this study, please contact the Project Manager, Harlan Messinger (703-543-2949).

Agreement (Statement of Assent)

I hereby accept the terms described above and give permission for my child to participate voluntarily in the NAEP usability study.

Participant Name:		Signature:		Date:
Parent/Guardian Name:		Signature		Date:

Please sign and return this form to Fulcrum IT. It can be e-mailed, faxed, or sent via postal service.

Usability Team

Fulcrum IT Services

5870 Trinity Parkway, suite 400

Centreville, VA 20120

Fax: 703-543-2978

E-mail: [email protected]

research participant informed consent form

Dear <Educator, Policymaker, Parent, Researcher, Media>: <DATE>

Purpose of Study

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Fulcrum IT is conducting a voluntary usability study with teachers and school administrators, parents, researchers, policymakers, and the media about redesigning the National Assessment of Educational Progress (NAEP) website (more information about the NAEP program is available online at: (http://nces.ed.gov/nationsreportcard/).

Description of Study

There are three tasks in the study that will take a total of approximately 3 hours. The tasks you will perform include a one-on-one telephone interview, an online card sorting task, and a user testing session. The user testing session will involve using our website to retrieve information while giving feedback to our research associates. These user testing sessions will be recorded for later analysis. Recordings will include audio of our conversation and video of your computer screen.

Compensation

Participants will receive compensation of $75 in the form of an American Express e-gift card for completion of all three phases of this study. Participants who do not complete all three phases of the study will be compensated at $25 for each phase of the study completed.

Confidentiality and Anonymity

Voluntary Nature of the Study

Your participation in this study is entirely voluntary, and you may withdraw at any time without consequence. Please sign this form and return it to Fulcrum IT before participating in any of the study tasks.

Contacts and Questions

Participants are given opportunities to ask questions before, during, and after the study. If there are any questions about the process, participants may contact a member of the usability team at Fulcrum IT.

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

If you have concerns about the conduct of the research associates during the course of this study, please contact the Project Manager, Harlan Messinger (703-543-2949)

Agreement (Statement of Assent)

I hereby accept the terms described above and agree to participate voluntarily in the NAEP usability study

Print Name:
Signature:	Date:

Please sign and return this form to Fulcrum IT. It can be e-mailed, faxed or sent via postal service.

Usability Team

Fulcrum IT Services

5870 Trinity Parkway, suite 400

Centreville, VA 20120

Fax: 703-543-2978

E-mail: [email protected]

Study completion e-mail

Dear ___________,

Thank you for participating in the usability study for the NAEP website. Your input has given us useful insight that will help us continue to improve this website so that the valuable information is readily available to anyone who is interested in education and the NAEP program.

We have purchased an American Express e-gift card for you to use online at any place that takes American Express credit or gift cards. You will be receiving an automated e-mail from American Express with your e-gift card information and instructions on how to use it to make purchases.

If you do not receive an e-mail from American Express in the next couple of days, please contact one of us, and we will make sure you receive your compensation for participation in our study.

Thanks again for your participation.

Sincerely,

Gregory Anderson, Senior Usability Expert, Fulcrum IT

703-543-2877

Joseph W. Quam, Usability Analyst / Technical Writer, Fulcrum IT

703-543-2878

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Operational Analysis Customer / Stakeholder Survey Checklist
Subject	Operational Analysis
Author	Fulcrum IT
File Modified	0000-00-00
File Created	2021-01-31

NAEP Website Usability Study 2013 Volume I