National Center for Education Statistics
National Assessment of Educational Progress
Volume I
Supporting Statement
Focus Groups for Transitioning Reading and Math Survey Questionnaires to Technology Based Assessments
OMB# 1850-0803 v.96
February 7, 2014
1) Submittal-Related Information 2
2) Background and Study Rationale 2
3) Sampling and Recruitment Plans 3
5) Consultations Outside the Agency 5
6) Assurance of Confidentiality 5
7) Justification for Sensitive Questions 5
8) Estimate of Hourly Burden 6
This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which allows for NCES to conduct various procedures (such as focus groups) to test new methodologies, question types, or delivery methods to improve survey instruments and procedures.
The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute for Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to also collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and school administrators to provide context for the reporting and interpretation of assessment results.
Over the next few years, NAEP will fully transition from paper-and-pencil assessments to technology-based assessments (TBAs). The current schedule for NAEP mathematics and reading assessments is to be operationally administered as TBAs in 2017 and piloted in 2016. Understanding the role of technology in reading and mathematics is crucial for a successful transition of the NAEP assessments to TBAs in order to capture the most relevant contextual factors.
To help inform the item development process for the survey questionnaires, data collection activities such as focus groups and cognitive interviews are planned prior to the 2016 pilot and 2017 operational assessments. Focus group research is especially important given unknown contextual factors associated with technology-based assessments. The main purpose of the proposed focus group research is to better understand the following constructs in the context of reading and mathematics:
Familiarity with technology, especially as it relates to mathematics and reading (e.g., e-readers, tablets, software, apps, and Internet resources);
Access to technology relevant for learning at home and at school, including patterns of access across student groups;
Ways technology (especially tablets, smart phones, and Internet resources) change students’ experiences with text, numbers, and calculations;
How the use of technology facilitates instruction and learning, both in and outside of the classroom; and,
Shifts in the nature and type of skills required of students as they read and/or do math in technology-rich environments.
The focus group method is a qualitative, group approach to data collection. It involves holding group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of an issue or topic. The method is useful for going beneath the surface of a response. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social issues such as challenges and issues surrounding cultural issues and education.
The results from this study will be used to set priorities for new item development to ensure that the 2017 survey questionnaires capture the most relevant contextual factors. Following the focus groups, draft items will be written and undergo cognitive interview testing, prior to being pilot tested in 2016. The versions of the items for both the cognitive interviews and the pilot test will be submitted to OMB for review prior to the commencement of these activities.
Volume I of this submittal contains descriptions of the design and sampling, as well as burden, cost, and schedule information for the study. Volume II contains the protocols for the various focus groups. The appendices contain notifications, consent forms, screening checklists, informational flyers, phone scripts, and thank you documents.
Focus groups with students will be conducted face-to-face, and focus groups with teachers and school administrators will be web-based. CRP, a consulting firm experienced in conducting focus groups has been selected by NCES to conduct this study (See Section 5). To ensure that appropriate and relevant information is obtained, 80 respondents will participate across the respondent types and grades, as shown in Table 1.
Table 1. Sample Size for Focus Groups
Respondent Group |
Grade 4 |
Grade 8 |
Total |
Students (Primarily Reading Questions, Some Additional Math Questions) |
8 |
8 |
16 |
Students (Primarily Math Questions, Some Additional Reading Questions) |
8 |
8 |
16 |
Students Sub-total |
16 |
16 |
32 |
Mathematics teachers |
8 |
8 |
16 |
Reading teachers |
8 |
8 |
16 |
Teachers Sub-total |
16 |
16 |
32 |
School administrators |
8 |
8 |
16 |
School Administrators Sub-total |
8 |
8 |
16 |
Overall Total |
40 |
40 |
80 |
Within these categories, CRP will recruit a mix of participants with respect to school type (e.g., public, private), community type (e.g., urban, rural), gender, race/ethnicity, familiarity with technology, and years of teaching/administrating experience to ensure a diverse sample.
Plan for Recruitment of Students
CRP will recruit students from the Washington, D.C. area. Students will participate in an in-person focus group at a location in Alexandria, VA, in order to equalize the travel distance for both prospective D.C. “urban students” and “rural students” in counties surrounding the focus group site.
First, CRP will send an email of introduction about the focus group research to various elementary and middle school principals in the D.C. area (see Appendix A) with the student flyers, (representative flyers in Appendices B and C), student demographic form (Appendix D), and parent/guardian consent form (Appendix I) attached. CRP will then follow up with the principal or a school administrator at each school via phone (transcript in Appendix E) to confirm their willingness to allow CRP to recruit students in their school. Principals that agree to participate will be asked to disseminate the flyers along with the consent form to students to bring to their parents/guardians. Principals, or their designee, will also be asked to select a subset of 4 to 6 students who volunteer to participate. The principal will be asked to complete the demographic form (Appendix D) for each student participant to ensure that the 4 to 6 students selected are a diverse sample as per the aforementioned criteria. Principals will then be asked to send the completed demographic forms to CRP via email.
Using the contact information provided by the principals, parents will first be sent a letter or email of introduction (Appendices F), and then a CRP staff member will follow up with the parent via phone (Appendix G). CRP will confirm participation, and the date and time of the focus group session (Appendix H). Parents/guardians will be required to sign informed consent forms for student participation prior to the focus group session (Appendix I).
Plan for Recruitment of Teachers and Administrators
Eight states will be randomly selected by CRP to participate. The largest population center or school districts (urban/suburban) and a random rural population area will be chosen from information derived from state websites. Teacher and school administrator recruitment will be augmented by utilizing national organizations’ databases of administrators and faculty.
Teachers and administrators will first be sent a letter or email of introduction and the Teacher/School Administrator consent form (Appendices J, K, and O), then a CRP staff member will follow up with the teachers and administrators via phone (Appendices L and M) and ask participants to provide demographic information to ensure a diverse sample is selected as per the aforementioned criteria. CRP will confirm participation, and the date and time of the focus group session (Appendix N). All participants will be required to sign informed consent forms for teacher/administrator participation prior to the focus group session (Appendix O).
Prior to the study, CRP will contact cooperating teachers, schools, and parents of participating students to confirm participation and make logistical arrangements. CRP staff experienced with focus groups will conduct the study. On the day of the focus group, participants will be asked to participate in focus group discussions (see Volume II). The focus groups are scheduled for 90 minutes. Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to help answer questions about mathematics and reading learning and the role of technology in learning/instruction. Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only (see Section 6). As part of the introduction process, the moderator will explain to participants that their responses will be audio recorded. The moderator will ask each participant to speak clearly and distinctly so that each response will be captured by the recording. For the web-based teacher and administrator focus groups, the moderator will explain the technology and describe the tools the participants may use, such as muting their phone and asking questions.
The moderator from CRP will be tasked with keeping participants engaged by asking probe questions (see Volume II), soliciting responses from less talkative participants, and asking follow-up questions directed to the group (e.g., “what do you all think about… [repeat focus group probe]”, or “that’s interesting, could you tell me a little bit more about that”). Interviewers may also take additional notes during the student focus groups, including behaviors (e.g., the participant’s facial expressions indicated they are confused) and if extra time was needed to answer certain questions.
Analysis Plan
After the session, the notes and audio recording will be summarized to report themes and illustrative statements that will be analyzed by the NAEP questionnaire development team. The focus group results will be used to set priorities for new item development to ensure that the 2017 survey questionnaires capture the most relevant contextual factors.
Educational Testing Service (ETS)
ETS serves as the Item Development (ID) contractor on the NAEP project, developing cognitive and survey items for NAEP assessments. As such, ETS will be responsible for the management of all activities described in this package.
CRP, Incorporated (CRP)
CRP is a minority- and woman-owned professional services and management consulting firm located in Silver Spring, Maryland that provides technical consulting primarily in the fields of education and behavioral health. CRP is a small business and serves as a subcontractor to ETS on the survey item development work.
Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].
Written consent will be obtained from participants and from parents or legal guardians of students. Teachers and school administrators will be asked to complete and return the consent form to CRP via email, fax, or mail before the web-based focus group session. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study and will be destroyed after the final report is completed.
The interviews will be recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is submitted.
Throughout the item, task, and interview protocols development process, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have attempted to identify and minimize potential bias in questions.
Estimate of Hourly Burden
The estimated burden for recruitment assumes attrition throughout the process.1 All focus groups will be scheduled for no more than 90 minutes. Table 2 details the estimated burden for the survey questionnaire focus groups.
Table 2. Estimated Burden for Recruitment of Focus Group Participants
Respondent |
Hours per respondent |
Number of respondents |
Total hours |
Principal or School Administrator for Student Recruitment |
|||
Initial contact |
0.05 |
36 |
2 |
Follow-up & Identify students |
1.0 |
24* |
24 |
Sub-Total |
|
36 |
26 |
Parent or Legal Guardian for Student Recruitment |
|||
Initial contact |
0.05 |
96 |
5 |
Follow-up via phone |
0.15 |
64* |
10 |
Consent & Confirmation |
0.15 |
32* |
5 |
Sub-Total |
|
96 |
20 |
Teacher and School Administrator Recruitment |
|||
Initial contact |
0.05 |
144 (96+48) |
7 |
Follow-up via phone or e-mail |
0.15 |
96*(64+32) |
14 |
Consent & Confirmation |
0.15 |
48*(32+16) |
7 |
Sub-Total |
|
144 |
28 |
Participation (Focus Groups) |
|
|
|
Grade 4 Students |
1.5 |
16 |
24 |
Grade 8 Students |
1.5 |
16 |
24 |
Teachers |
1.5 |
32* |
48 |
School Administrators |
1.5 |
16* |
24 |
Sub-Total |
|
32 |
120 |
Total Burden |
620 responses |
308 |
194 |
*Subset of initial contact group
To encourage participation and thank for their time and effort, a $25 VISA gift card will be offered to each participating student, plus a $25 VISA gift card to a parent or legal guardian bringing the student to and from the testing site, along with a thank you letter (see Appendix P) for allowing the child to participate in the study. Teacher and school administrator focus groups participants will be mailed a $40 VISA gift card and a thank you email (see Appendix Q) for taking part in the study.
Costs to Federal Government
The estimated cost for the survey questionnaire focus group activities in this submittal is $557,477 as delineated in Table 3.
Table 3. Estimate of Costs
Activity |
Provider |
Estimated Cost |
Design, preparation, and analysis for mathematics and reading survey questionnaire focus groups |
ETS |
$ 77,000 |
Preparation and conduct of mathematics and reading survey questionnaire focus groups (including recruitment, incentive costs, data collection, analysis, and reporting) |
CRP |
$ 480,477 |
Table 4 depicts the high-level schedule for the reading and mathematics focus group activities. The commencement of activities is contingent upon OMB approval.
Table 4. High-Level Schedule of Milestones
Dates |
Activity |
March–April 2014 |
Recruitment process |
April-June 2014 |
Data Collection, Analysis, and Reporting |
1 Assumptions for approximate attrition rates for direct participant recruitment are 33 percent from initial contact to follow-up, 50 percent from follow-up to confirmation. Note: The initial principal contact for student identification attrition rate is 33 percent from contact to confirmation.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Debby E. Almonte |
File Modified | 0000-00-00 |
File Created | 2021-01-28 |