National Center for Education Statistics
National Assessment of Educational Progress
Appendices
Usability Study for Use of Touch-Screen Tablets
in NAEP Assessments
OMB# 1850-0803 v.87
October 30, 2013
Volume I: Appendices
Appendix A: Usability Study Sample E-mail Contact with Participating Schools 1
Appendix B: Usability Study Sample School Contact Script 2
Appendix C: Usability Study Sample School Contact Script-Subsequent Sessions 5
Appendix D: Usability Study Parent/Guardian Notification Letter 8
Appendix E: Thank You Email to Participating School 9
E-mail Subject Line: Arrangements for NAEP Usability Study Date:
Dear [NAME of School Coordinator],
[NAME of NAEP State Coordinator] forwarded your name to me to work out arrangements for my colleagues and me to visit [Name of School] in the next few weeks. We are requesting to work with [number of students] as part of a usability study to test our system on new touch-screen tablets. This study will be conducted by Fulcrum IT, on behalf of the National Center for Education Statistics in regards to the National Assessment of Educational Progress, or NAEP.
In the study, students will be asked to complete tasks on a tablet and answer questions about how easy or hard it is to accomplish computer functions, answer questions in different formats, and understand what the system expects them to do next. Before working on the tablets, students will be asked about their familiarity and frequency of use of laptops, smart phones, and tablets. Student information may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Any student may decline participation for any reason.
There are a few logistical matters to address before we conduct the study. I will summarize them in this email, and then follow-up with a phone call to you to finalize these arrangements, unless you prefer to finalize the arrangements over email.
Study Date(s)
We expect each user testing session to take approximately 75 minutes to complete. We could come to your school the week of [Date]; or we can arrange different dates with you.
Student Selection for Study
Please identify up to [number of students] [appropriate grade] students to participate in the study - approximately half boys and half girls. The students should be generally representative of your school makeup (e.g., race, SES, familiarity with computers). The purpose of this study is to examine the usability of tablets for a wide range of students. Therefore, if possible, it is important to include some students that have limited familiarity with tablets and/or computers.
Space Requirements
When we visit to conduct the study, we will need to use a conference room, classroom, or space in your media center or library, if private, with room for up to three students with tablets and 4 research associates with laptops. We will bring the tablets and related equipment, and set up everything that we need. We would like to meet with up to three students at a time for up to 75 minutes each, but we are flexible depending on your class schedules and space considerations.
Parent/Guardian Notification
Parents/guardians of the students participating in the study should be notified by letter. A sample letter is attached to this email; please provide this letter to the parents/guardians.
Other Logistics
It would be helpful to know your school day schedule so that we can plan to arrive 45 minutes before the first testing session. In addition, please let me know where we should park, and where we should check in with school staff upon arrival.
I will call you within the next few days to determine which dates might work for you and to answer any questions you might have. Feel free also to contact me by email or phone, as indicated below. We greatly appreciate your school’s cooperation, and will offer a $50 gift card to a school supply store as a token of our appreciation. Thank you for your help.
Sincerely,
Gregory Anderson, Ph.D.
Senior Usability Expert
Fulcrum IT Services, LLC
703-543-2877
Appendix B: Usability Study Sample School Contact Script
NAEP 2013 Tablet Usability Study
School Contact Script
Note: Information in grey boxes is not to be read; rather, these are instructions to the Fulcrum IT staff.
Hello, _________________________ (name of school coordinator). My name is __________________ and I am calling from Fulcrum IT, which has been contracted to conduct a study on behalf of the National Center for Education Statistics about the National Assessment of Educational Progress, or NAEP. I am following up on an email I sent to you a few days ago describing the study. This call should only take a few minutes. Would now be a good time to finalize arrangements for the study?
|
Yes |
|
No |
If not, when would be a good time? ___________________________
1. In my email, I proposed the week of [Date] for us to conduct the study. Is there a day during that week that works for you?
|
Yes |
Date 1: ________________ |
|
No |
Arrange new date: _____________________________________ |
2. Now let’s discuss the study logistics. We would like to work with [number of students] total, divided into 2-3 groups, for 75 minutes each. Students will be paired with a staff member from my company who will facilitate the study session. We’ll need a quiet space with desks or tables, in which to set up tablets and laptop computers for the students and facilitators.
|
Date:_____________________________
|
|
Students Group #1 |
Time: ___________ |
Location: ______________ |
Students Group #2 |
Time: ___________ |
Location: ______________ |
Students Group #3 |
Time: ___________ |
Location: ______________ |
My team and I will need about 45 minutes before the first group is scheduled to set up and prepare. What time should we arrive? ______ (time), is that OK? May we have access to __________________ (the testing location) at that time?
3. Were you able to identify [number of students] [specify grade] students, generally representative of your school makeup (e.g., race, familiarity with computers)?
|
Yes |
Continue |
|
No |
Provide assistance in selecting as diverse a group of students as possible. |
As mentioned earlier, we’ll be working with the selected students a few at a time. When I arrive at your school on the morning of the usability study, please provide me with a list of the selected students, along with information about when the students are available.
4. Now, let’s review the status of parent/guardian notification. Have the parents and guardians been notified?
|
Yes |
Continue |
|
No |
Skip to question 5. |
Please keep a copy of the letter for me that I’ll collect on the morning of the usability study.
Were notifications sent to only the parents and guardians of the selected students or to the parents and guardians of all the students in the [specify grade] grade?
|
Parents/Guardians of selected students only |
Skip to question 6. |
|
Parents/guardians of all students in the [specify grade] grade. |
On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible. |
Skip to question 6. |
5. As a reminder, the parents and guardians of the selected students must be notified before the usability study can take place. I’d like to call you back to confirm the letter has been sent. When is a good time for me to make that call?
Callback date _______________________ Time ________________ |
Also, I’ll need a copy of the letter for my records, which I’ll collect from you on the morning of the usability study.
Do you plan to notify the parents/guardians of only the selected students or the parents and guardians of all the students in the [specify grade] grade?
|
Parents/Guardians of selected students only |
Skip to question 6. |
|
Parents/guardians of all students in the [specify grade] grade. |
On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible. |
6. I have just a few other logistical questions to ask you, so that the study runs as smoothly as possible.
When my team and I arrive at your school on the morning of the study, where should we park?
_______________________________________________________________________________________
We will check in at the main office. Are there other procedures or protocols specific to your school that we need to follow?
_______________________________________________________________________________________
Is there anything scheduled that might interrupt the usability session, such as a fire drill or assembly?
_______________________________________________________________________________________
How should we contact the office from the study location if we need assistance should there be an emergency?
_______________________________________________________________________________________
How do you announce school closings or delays, such as a snow day or weather delay? By what time are these decisions made?
_______________________________________________________________________________________
Our team lead will want to let you know when we are leaving, and will have a gift card for the school as a thank you for participating in our study. With whom should he speak?
_______________________________________________________________________________________
7. Those are all of the questions I have for you at this time. Do you have any questions for me?
If you have any questions before the usability study date, I can be reached at ________________________ (telephone number) or this e-mail address _________________________ (e-mail address).
Thank you for your time.
NOTES:
Appendix C: Usability Study Sample School Contact Script- Subsequent Sessions
NAEP 2013 Tablet Usability Study
School Contact Script
Note: Information in grey boxes is not to be read; rather, these are instructions to the Fulcrum IT staff.
Hello, _________________________ (name of school coordinator). My name is __________________ and I am calling from Fulcrum IT, which has been contracted to conduct a study on behalf of the National Center for Education Statistics about the National Assessment of Educational Progress, or NAEP. I am calling to make arrangements for our next study session at your school. This call should only take a few minutes. Would now be a good time to finalize arrangements for the study?
|
Yes |
|
No |
If not, when would be a good time? ___________________________
1. Earlier, we had talked about the week of [Date] for us to conduct this study session. Is there a day during that week that works for you?
|
Yes |
Date 1: ________________ |
|
No |
Arrange new date: _____________________________________ |
2. Now let’s discuss the study logistics. We would like to work with [number of students] total, divided into 2-3 groups, for 75 minutes each. Students will be paired with a staff member from my company who will facilitate the study session. Last time we were here, we worked in (location). Would you like us to use the same space this time?
|
Yes |
|
No |
If not, where would you like us to work? ___________________________
|
Date:_____________________________
|
|
Students Group #1 |
Time: ___________ |
Location: ______________ |
Students Group #2 |
Time: ___________ |
Location: ______________ |
Students Group #3 |
Time: ___________ |
Location: ______________ |
My team and I will need about 45 minutes before the first group is scheduled to set up and prepare. What time should we arrive? ______ (time), is that OK? May we have access to __________________ (the testing location) at that time?
3. Were you able to identify [number of students] [specify grade] students, generally representative of your school makeup (e.g., race, familiarity with computers)?
|
Yes |
Continue |
|
No |
Provide assistance in selecting as diverse a group of students as possible. |
As mentioned earlier, we’ll be working with the selected students a few at a time. When I arrive at your school on the morning of the usability study, please provide me with a list of the selected students, along with information about when the students are available.
4. Now, let’s review the status of parent/guardian notification. Have the parents and guardians been notified?
|
Yes |
Continue |
|
No |
Skip to question 5. |
Please keep a copy of the letter for me that I’ll collect on the morning of the usability study.
Were notifications sent to only the parents and guardians of the selected students or to the parents and guardians of all the students in the [specify grade] grade?
|
Parents/Guardians of selected students only |
Skip to question 6. |
|
Parents/guardians of all students in the [specify grade] grade. |
On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible. |
Skip to question 6. |
5. As a reminder, the parents and guardians of the selected students must be notified before the usability study can take place. I’d like to call you back to confirm the letter has been sent. When is a good time for me to make that call?
Callback date _______________________ Time ________________ |
Also, I’ll need a copy of the letter for my records, which I’ll collect from you on the morning of the usability study.
Do you plan to notify the parents/guardians of only the selected students or the parents and guardians of all the students in the [specify grade] grade?
|
Parents/Guardians of selected students only |
Skip to question 6. |
|
Parents/guardians of all students in the [specify grade] grade. |
On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible. |
6. I have just a few other logistical questions to ask you, so that the study runs as smoothly as possible.
When my team and I arrive at your school on the morning of the study, where should we park?
_______________________________________________________________________________________
We will check in at the main office. Are there other procedures or protocols specific to your school that we need to follow?
_______________________________________________________________________________________
Is there anything scheduled that might interrupt the usability session, such as a fire drill or assembly?
_______________________________________________________________________________________
How should we contact the office from the study location if we need assistance should there be an emergency?
_______________________________________________________________________________________
How do you announce school closings or delays, such as a snow day or weather delay? By what time are these decisions made?
_______________________________________________________________________________________
Our team lead will want to let you know when we are leaving, and will have a gift card for the school as a thank you for participating in our study. With whom should he speak?
_______________________________________________________________________________________
7. Those are all of the questions I have for you at this time. Do you have any questions for me?
If you have any questions before the usability study date, I can be reached at ________________________ (telephone number) or this e-mail address _________________________ (e-mail address).
Thank you for your time.
NOTES:
Appendix D: Usability Study Parent/Guardian Notification Letter
NAEP PARENT/GUARDIAN NOTIFICATION LETTER
Usability Evaluation of Touch-Screen Tablets
[School Letterhead] [Date]
Dear Parent or Guardian:
We are pleased to inform you that [school name] has been selected to participate in the Student Usability Evaluation of touch-screen tablets for the National Assessment of Educational Progress (NAEP). This study will be used to prepare for delivery of future NAEP assessments using touch-screen tablets instead of laptop computers.
NAEP is the largest continuing and nationally representative assessment of what our nation’s students know and can do. The results of NAEP are released as The Nation’s Report Card, which provides information about student achievement to educators, parents, policymakers, and the public.
In our school, the usability study will be conducted on [date]. Your child may be selected to participate. The study involves observing individual students working for 75 minutes on touch screen tablets and asking them questions about how easy or hard it is to accomplish tablet functions, answer questions in different formats, and understand what the system expects them to do next. Their comments and actions will be recorded for later analysis. Before working on the tablets, students will be asked about their familiarity and frequency of use of laptops, smart phones, and tablets.
It will take about 75 minutes for most students to complete the study. The screens of the tablets will be recorded, along with the audio from the session. No cameras will be used, and no images or video of your child or their surroundings will be captured. The researchers will collect no identifying information on the students. Each student will be identified only by participant number in the study. All the information your child provides may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].
Your child will represent many other students, so participation is very important. However, your child’s participation in the study is voluntary. In addition, your child may decide not to participate or end his or her participation before the study is complete. If you do not want your child to participate, please notify me in writing by [date].
There is no need to prepare for this study. As a token of appreciation for participating in the study, your child will get to keep a set of earbuds given to them to plug in and listen to the tablet.
Students are given an opportunity to ask questions before, during, and after the study. If you have any questions before or after the study, contact the lead field researcher, Gregory Anderson, Fulcrum IT Services, LLC, 703-543-2877 ([email protected]). If you would like additional information about the NAEP assessments, please visit http://nces.ed.gov/nationsreportcard/. If you have any other questions, please contact me at [telephone number] or via e-mail at [e-mail address].
Sincerely,
School Principal
Appendix E: Thank You Email to Participating School
THANK YOU EMAIL
Usability Evaluation of Touch-Screen Tablets
[Date]
Dear [Head of School]:
I wanted to write you personally to thank you for participating in our NAEP usability study. The results of this study will be used to improve the delivery of NAEP assessments, and make participating in these assessments a positive experience for everyone involved. If you have any feedback for us about your experience participating in this usability study, don’t hesitate to contact me or the project manager at Fulcrum IT.
Sincerely,
Gregory Anderson, Ph.D.,
Senior Usability Expert, Fulcrum IT
(703) 543-2877
Project Manager: Harlan Messinger
(703) 543-2949
File Type | application/msword |
File Title | Appendices (A-R) |
Author | mgisbert |
Last Modified By | Kashka Kubzdela |
File Modified | 2013-10-31 |
File Created | 2013-10-29 |