Justification

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803), which provides for NCES to conduct various procedures (such as field tests, cognitive interviews, and feasibility studies) to test new methodologies, question types, or delivery methods to improve assessment instruments and data collection methodologies and expand on NCES’s educational data analyses that can be provided to the public. This request is to examine the feasibility of conducting a Middle School Transcript Study (MSTS), which would collect transcripts of middle school students in conjunction with the regular administration of the National Assessment of Educational Progress (NAEP).

Background and Study Rationale

NAEP is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, part of the Institute for Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in various subject areas, and to collect survey questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.

The goal of MSTS will be to examine middle school coursetaking patterns. The MSTS feasibility study will be the first transcript study to be conducted entirely using electronic records submissions and the first to focus on middle grades. It will be limited to select large urban districts, referred in NAEP as Trial Urban District Assessment (TUDA) jurisdictions and, as a feasibility study, will not be conducted at the state level. As a component of NAEP, the MSTS feasibility study will be conducted in conjunction with the 2017 NAEP grade 8 mathematics and reading assessments. The MSTS feasibility study data will be linked to the NAEP assessment scores, as well as school and student background questionnaire data, so that any relationships between student coursetaking and performance may be evaluated. While similar transcript studies have been conducted at the twelfth-grade level, this study will investigate the feasibility of providing educators and educational researchers with valuable information that will allow for the analyses and reporting of national trends in middle school coursetaking.

Conducting MSTS 2017 as a feasibility study with a limited number of TUDAs will help to evaluate and streamline operations for potential future NAEP TUDA-based middle or high school transcript studies, as well as other electronic data-based transcript studies. The MSTS 2017 feasibility study plans to collect middle school student coursetaking information in at least six voluntary TUDA jurisdictions that are part of the NAEP 2017 grade 8 sample, including data on courses taken, credits earned, and grades received. There will be no known burden on the schools because data will be collected at the district level. Furthermore, there will be little burden on districts, given that the data will be collected entirely via the submission of electronic records^¹. In previous high school transcript studies, schools were given the option to submit transcripts electronically, but relatively few had the capacity to use this option. Over the last decade, however, more student-level educational data are being stored electronically by schools, districts, and states. This study will rely on electronic transcript data sources at the district level.

One of the main objectives of an operational MSTS would be to supplement NAEP data with valuable coursetaking information, which would provide TUDAs with more contextual data to explain their eighth-grade students’ NAEP assessment scores. What courses students took and their performance in those courses, combined with their attitudes toward mathematics and reading courses as asked in the student questionnaire, would provide a more complete picture of how students perform on the eighth-grade NAEP. An operational MSTS would allow comparisons of student coursetaking information within and across TUDA jurisdictions at the middle school level. These comparisons would provide more insights about student achievement in critical middle school grades, while the addition of other demographic information, data sources, and questionnaire data would identify differences between the participating TUDAs.

Analyses of the MSTS 2017 feasibility study will mimic those that would be performed in an operational MSTS. As such, they will focus on academic tracking, identification of potential student dropouts, performance gaps, and the impact of unequal access to technology on students. An additional objective of this feasibility study is to promote the use of electronic records in school districts, since district participation is contingent on having records in this format (see Appendix A for how TUDAs can benefit by participating in the MSTS 2017 feasibility study).

Study Design

NCES has contracted with Westat to carry out recruitment, data collection, course catalog and transcript coding, analyses, and reporting for the MSTS feasibility study. NCES has also contracted with Educational Testing Service (ETS) to prepare student NAEP assessment scores for analyses with data collected through the MSTS feasibility study. The study will proceed in three phases, including selection of TUDAs for participation, data collection, and subsequent coding and analysis.

In the first phase, Westat will prepare materials and solicit TUDA jurisdictions for participation, ensuring that the possible participants understand their responsibilities and requirements for the study. Eight TUDAs will be selected. Additional jurisdictions will be considered for inclusion based on projected time and cost. Appendix B lists the 27 TUDAs participating in NAEP 2017, which represents the universe of the jurisdictions, all of which might be given the option to participate in the feasibility study.

In the second phase, district-level course catalogs and student transcripts will be collected electronically. Course-level catalogs will be collected soon after a TUDA agrees to participate in the study, while student transcripts will be collected during the following summer. Participating TUDA districts will be responsible for transferring the electronic files to secure NAEP file transfer protocol (FTP) sites hosted at Westat. The coding of course catalogs and student transcript information will be conducted by coders who will be trained by experienced High School Transcript Study (HSTS) supervisors, with quality control checks and mechanisms for inter-coder reliability put in place.

In the third phase, the NAEP assessment scores of the students will be adjusted using conditional variables and weights provided by Westat to ETS using secure protocols already developed for NAEP. Those variables and weights will be used by ETS to generate scores (known as plausible values) for the students in the MSTS feasibility study. Westat will also produce preliminary analyses using transcripts and analytically derived variables on hand. These analyses, informed by prior experience with the HSTS, will focus on credits earned, grade point averages, highest level courses taken, measures of different academic tracks, identifiers of potential student dropouts, and measures involving student access to computers and the Internet in and outside of classrooms.

The results of the MSTS feasibility study will be published as an NCES Research and Development (R&D) report², incorporating all participating TUDAs’ results. In addition, individualized data summaries will be delivered to each TUDA, however they will be caveated as not containing official NCES statistics given that this first collection is only a feasibility study designed to evaluate the possibility of full scale MSTS data collections in the future. As such, the TUDA data summaries will be for TUDAs’ internal use only, and TUDAs will not be allowed to publish or shared them outside of each TUDA’s district office. Appendix C contains a list of analyses that are expected to be included in the R&D report and the individual TUDA data summaries. The R&D report, as well as feedback regarding the study process, will inform NCES’ decision whether MSTS should be conducted in the future.

Sampling and Recruitment Plans

Recruitment letters will be mailed to the 27 TUDA districts participating in the 2017 NAEP assessments (see Appendix B for complete listing) seeking six or more volunteer participants (see Appendix D for an example of the recruitment letter; Appendix E for recruitment brochure text; and Appendix F for the letter of interest to be sent to districts). A follow-up web meeting with interested TUDA coordinators or representatives will be conducted to further explain the objectives of the study, data collection process, and field questions. If more than eight TUDAs wish to sign on, NCES will consider the probable extra time and cost of accepting additional participants.

If the number of interested TUDAs is larger than can be feasibly included in the study, selection criteria will be developed by Westat and NCES, focusing on:

Geographic diversity among TUDA districts;
Demographic diversity among TUDA districts; and
The quality of student records (in particular, whether they are complete and come in digitally readable file formats).

Upon selection for the MSTS feasibility study, interested TUDAs will be informed about their participation status (see Appendix G for the letter to be sent to participating TUDAs; and Appendix H for the letter to be sent to TUDAs that are not selected for the study).

Data Collection Process

Unlike prior transcript studies, data for the MSTS will be collected entirely through the electronic transfer of files. Individual schools will not be involved in the data collection process. Appendix I lists the data that will be collected from the TUDAs. Participating TUDAs must be able to submit electronic student transcripts and the course catalog of their state or jurisdiction. The data transfer is already covered under the auspices of NAEP, but some jurisdictions may ask for additional information or a special data sharing agreement. Since the MSTS is a feasibility study under the NAEP program, the collection of electronic student transcripts falls under the auspices of the overall NAEP data collection procedures and protocols (described in OMB #1850-0928 v. 1-2) and does not require a separate MSTS specific data sharing agreements. However, if a district finds that a data sharing agreement is necessary to comply with state rules and regulations, we will work with the district to execute a data sharing agreement (see Appendix J for a data sharing agreement template to be used if a special data sharing agreement is requested).

Course Catalogs

Upon being selected to participate in the MSTS feasibility study, the NAEP TUDA coordinator will be contacted to collect information for the study and acquire the district-level course catalog. Appendix K contains the telephone protocol that will be used when contacting the NAEP TUDA coordinators. The catalog should contain a list of secondary school courses offered to students in the TUDA between sixth and eighth grades. Each course must have a unique ID code that corresponds with the course ID codes found on the electronic student transcripts. TUDAs will be encouraged to include course descriptions, which focus on the topics taught in the course, with the catalog if they are available, but they are not required.

Transcripts

After the 2016-2017 school year, Westat will again contact the NAEP TUDA coordinator to request the electronic submission of the student transcript records. Appendix L contains the telephone protocol that will be used when contacting the TUDA coordinators concerning the student transcript records. The student transcripts will record all courses taken by students between sixth and eighth grades, including the unique ID code that links to courses in the course catalog. Based on the 2015 NAEP HSTS Electronic Transcript Data Collection Feasibility Study (OMB# 1850-0803 v.113), if districts maintain their own electronic files, transferring them should take only 1 to 2 weeks. If an offsite vendor maintains their files, it may take up to 6 weeks. If requested by the TUDA, there may be some reimbursement to offset these costs (as discussed in section 9 of this document).

Electronic Submission Process

TUDA representatives will upload both course catalogs and student transcript records to secure NAEP FTP sites hosted at Westat. Westat personnel will collect and download the electronic data files from the secure FTP sites for all participating TUDAs.

Westat has approved systems in place for the secure exchange of data between contractors and jurisdictions. FTP sites have been successfully used for the exchange of NAEP data, including Personally Identifiable Information (PII), between Westat and states. For the MSTS feasibility study, secure FTP sites will be used. Each TUDA will be assigned its unique FTP site. The TUDA Coordinator will be provided with the username and password to access the site. When the user receives the instruction and uploads the data, other data will not be visible to the user. The upload process will be a simple process. When the file is uploaded, the MSTS contract administrator will receive an email to download the data. The data will be stored according to NCES security standards and deleted from the FTP site and securely stored according to NCES data security protocols (see section 7 below).

NCES requires that TUDAs not know which of their students were sampled during NAEP assessments, so the TUDAs will need to submit transcripts without realizing the student sample. Linking students in the 2017 NAEP and the MSTS feasibility study will be done using a unique student ID that was assigned to all students sampled for the NAEP. From there, one of two methods can be used:

The TUDA could provide transcripts for all eighth grade students from the sampled NAEP schools within their jurisdiction. While this may be very efficient with electronic records, some TUDAs may be reluctant to provide more data than is absolutely necessary for the study.
The TUDA could provide transcripts for all eighth grade students sampled for the NAEP 2017 assessments, plus an additional 10 percent of students who were not sampled. This option was used in the 2015 NAEP HSTS Electronic Transcript Data Collection Feasibility Study.

Consultations Outside the Agency

Each NAEP TUDA Coordinator serves as the liaison between a TUDA education agency and NAEP, coordinating NAEP activities in his or her TUDA. NAEP TUDA Coordinators from the participating TUDAs will work with appropriate district staff to collect the information needed for the study, and to provide electronic submissions of the district-level course catalog and student transcript records.

Westat is the Sampling and Data Collection (SDC) contractor for NAEP. Westat will recruit the TUDAs to participate in the study, coordinate the catalog and transcript collection and transmittal process, and produce the data sets and analysis reports for the study. Educational Testing Service (ETS) is the Data Analysis and Reporting (DAR) contractor for NAEP, providing psychometric and statistical analyses to effectively model current and future assessment data. ETS will recondition the NAEP eighth-grade assessment scores for participating TUDAs in the MSTS.

Confidentiality and Data Security Procedures

NAEP has policies and procedures that dictate student data security and confidentiality in compliance with legislation (Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347, the National Assessment of Educational Progress Authorization Act, and the Education Sciences Reform Act of 2002, 20 U.S.C §9573). Specific to the NAEP program, the legislation ensures that privacy, security, and confidentiality policies and procedures are in compliance with the Privacy Act of 1974 and its amendments, NCES Confidentiality Procedures, and the Department of Education Automated Data Processing (ADP) Security manual.

NCES ensures that all current contractor policies and procedures are in compliance with all NAEP security and confidentiality requirements. In addition, all NAEP contractor staff members with access to confidential NAEP information are required to sign affidavits of nondisclosure affirming, under severe penalty for unlawful action, that they will protect NAEP information from non-authorized access or disclosure. Appendix M contains NCES Affidavit of Nondisclosure that all personnel who work on NAEP are required to sign.

NAEP complies with the Family Educational Rights and Privacy Act (FERPA). Student transcript disclosure is permitted under FERPA’s exception to the general consent requirement that permits disclosures to authorized representatives of the Secretary for the purpose of evaluating Federally supported education programs (34 CFR §§ 99.31 (a)(3)(iii) and 99.35).

Confidential information is protected during transmission to and from NAEP systems by the use of robust secure file transfer protocol (FTP) platforms and of data encryption technologies, such as secure sockets layer (SSL), secure shell (SSH), and digital certificates and signatures that encrypt data, validate data integrity, and authenticate the parties in a transaction.

Estimate of Hourly Burden

The MSTS feasibility study will not result in any burden to students, teachers, or schools. There will be limited burden on TUDA district administrative and information technology staffs to review, approve, and implement the request for electronic transcript data, along with electronic supporting documentation. Table 1 lists the hourly estimates of response burden for the maximum possible number of participating TUDAs (this study aims for a minimum of six participating TUDAs).

Because the MSTS is a feasibility study part of the NAEP program, the collection of electronic student transcripts falls under the auspices of the overall NAEP data collection procedures and protocols and, as such, does not require a separate MSTS-specific data sharing agreements. However, if a district finds that a data sharing agreement is necessary to comply with state rules and regulations a data sharing agreement will be executed.

Table 1. MSTS response burden estimate based on maximum possible number of TUDA participants

Activity	Maximum Number of Respondents	Maximum Number of Responses	Hours per Response	Maximum Total Hours
Negotiate data sharing agreement for electronic transcript transfer¹	27	27	20	540
Interview to collect initial MSTS information	27	27	1	27
Collection of requested information after the interview	27	27	2	54
Uploading course catalog to secure FTP site	27	27	1	27
Uploading student transcripts to secure FTP site, including phone call regarding procedures	27	27	8	216
Total	27	135		864

¹ Should not be necessary for most TUDAs since they are covered by the NAEP data sharing agreement.

Estimate of Costs for Recruiting and Paying Respondents

No incentives will be provided for participants in the study. However, participating TUDA districts that express concern about the burden of providing the requested data will be reimbursed for their time, up to a maximum amount of $250.

Costs to Federal Government

The estimated cost to the federal government for recruiting the TUDAs, contacting the participating TUDAs to arrange electronic transfers of the catalogs and transcripts, setting up the secure FTP sites for the transfers, coding the catalogs and title matching to the transcripts, preparing the data files, analyzing the transcript data, and preparing the R&D report and TUDA data summaries is $1,084,602. Of this amount:

Approximately $216,920 will be spent on recruiting the TUDAs to participate in the study and negotiating any necessary data sharing agreements;
Approximately $379,611 will be spent on the collection and coding of the electronic course catalogs and student transcript records;
Approximately $162,690 will be spent on producing the weights and reconditioned NAEP assessment scores for the study; and
Approximately $325,381 will be spent conducting the analyses and producing the R&D report and TUDA data summaries.

Schedule

Table 2 depicts the timeline for the major tasks associated with the feasibility study.

Table 2. Timeline for the 2017 MSTS Feasibility Study with associated major tasks

Description	Dates
Contact TUDAs to recruit them for MSTS Feasibility Study	November 2016 – February 2017
Negotiate data sharing agreements (if needed)	November 2016 – June 2017
Collect course catalogs	March – June 2017
Collect student transcripts	June – August 2017
Analysis	September 2017 – January 2018
Prepare the R&D report and TUDA data summaries	November 2017 – May 2018

1 For the purpose of this study, electronic records are defined as school course listings and student transcripts stored in electronic database formats, such as Microsoft Excel or Access files, statistical software databases (e.g., SAS, SPSS, STATA), or file formats that can be easily converted into databases (e.g., CSV or XML files). PDF transcripts are considered analogous to paper copies because they require the NCES data collection contractors to manually key information from the transcripts into a database. As a result, such transcripts will be avoided if at all possible.

2 The R&D series of reports at NCES has been initiated to:

Share studies and research that are developmental in nature. The results of such studies may be revised as the work continues and additional data become available;
Share the results of studies that are, to some extent, the “cutting edge” of methodological developments. Emerging analytical approaches and new computer software development often permit new and sometimes controversial analyses to be done. By participating in “frontier research,” we hope to contribute to the resolution of issues and improved analysis; and
Participate in discussions of emerging issues of interest to education researchers, statisticians, and the federal statistical community in general. Such reports may document workshops and symposia sponsored by NCES that address methodological and analytical issues or may share and discuss issues regarding NCES practices, procedures, and standards.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-23