NPSAS 2012 CADE Usability Interviews 2013 Memo

April 10, 2013

M E M O R AN D U M

TO: Shelly Martinez, OMB

FROM: Matthew Soldner, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: NPSAS:12 Computer-Aided Data Extraction Usability Interviews 2013

(OMB No. 1850-0803 v.77)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.

The request for approval described in this memorandum includes interviews regarding institutional representatives’ experiences with the computer-aided data extraction (CADE) of student records requested as part of the 2011-12 National Postsecondary Student Aid Study (NPSAS:12), as well as related recruiting and screening procedures.

This package provides information about plans and procedures for conducting the interviews described above, planned for summer 2013.

Background

NPSAS:12 is a comprehensive study of how students and their families pay for postsecondary education. It combines information gathered from students via a web-based interview with data extracted from several sources, including those managed by the Department (e.g., the National Student Loan Data System, FAFSA Central Processing System) as well as campus-based student information systems. It is the latter that is the focus of this work.

Institutions have provided student-level data to the Department as part of the NPSAS family of studies since its earliest days. In preparation for NPSAS:12, NCES and its primary data collection contractor, RTI International, invested significant resources in the redesign of the web-based portal through which institutions would submit their data. This included improvements to methods of entry, including: a) case mode entry, where respondents entered data one student at a time via a web form; b) grid mode entry, where respondents entered data in a spreadsheet-like format; c) Excel mode, where respondents received a custom Excel file, and d) file upload entry, where respondents uploaded all requested data via CSV files.

Research goal. Although NCES and RTI received anecdotal feedback from portal users during the NPSAS:12 data collection, no systematic effort was made to gather data about the user experience, including usability, burden, and potential improvements for future collection cycles. This package begins to gather that data, informing not only plans for designing the NPSAS:16 portal but also how NCES can improve its efforts to collect user experience data in the future.

Design and Context

To meet the research goal identified above, NCES proposes to undertake the following steps with the assistance of its contractor, Westat:

1. NCES will provide Westat with contact information for 50 institutional representatives who participated in the NPSAS:12 computer-aided data extraction process;

2. Westat will recruit/screen potential interviewees for their participation in an hour-long usability interview using the protocol detailed in Attachment 1 until such point as recruitment quotas are met (up to 30 respondents, across a mix of respondent types);

3. After identifying eligible respondents, Westat will send recruited participants the Informed Consent/Assurance of Confidentiality Statement detailed in Attachment 2 via electronic mail, as well as a series of portal screen captures to help refresh respondents’ memories during the interview itself (Attachment 3);

4. After verifying that the respondent has read and understood the Informed Consent/Assurance of Confidentiality Statement emailed prior to their interview, a survey methodologist from Westat will conduct and record a 60 minute telephone interview with respondents (at the time and location of their choosing) using the protocol detailed in Attachment 4 as a guiding framework;

5. Westat will archive the digital recordings of each completed interview for the purpose of preparing a qualitative analysis and for NCES quality assurance and review;

6. NCES will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to NCES and shared with the NPSAS:16 data collection contractor.

Assurance of Confidentiality

Interview participants will be informed that their participation is voluntary and their responses are protected by federal statute (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Attachment 2). Respondents will be assigned a unique identifier (ID), which will be created solely for file management and used to keep all materials together. The ID will not be linked to the respondent name in any way or form. Acknowledgements of consent will be kept separately from interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule

Recruitment and screening for this project is planned for summer, 2013 and NCES anticipates Westat will complete interviews during fall, 2013.

Estimate of Hour Burden

NCES anticiaptes screening, at most, 50 potential interviewees at no more than 15 minutes per contact. Actual burden should be somewhat lower, as Westat will screen within a respondent type only until that type’s quota is reached. In total, 30 individual cognitive interviews are planned. Interview sessions are expected to average 60 minutes. In total, this collection represents 42.5 burden hours.

Estimated respondent burden

Respondent Type	Number of Respondents	Number of Responses	Hours per Respondent	Total Burden Hours
Recruitment and Screening of Potential Interviewees
CADE via File Upload	10	10	.25	2.5
Grid Mode, 50 or More Sampled Students	10	10	.25	2.5
Grid Mode, 49 or Fewer Sampled Students	10	10	.25	2.5
Case Mode, 20 or More Sampled Students	10	10	.25	2.5
Case Mode, 19 or Fewer Sampled Students	10	10	.25	2.5
Interviews
CADE via File Upload	6	6	1	6
Grid Mode, 50 or More Sampled Students	6	6	1	6
Grid Mode, 49 or Fewer Sampled Students	6	6	1	6
Case Mode, 20 or More Sampled Students	6	6	1	6
Case Mode, 19 or Fewer Sampled Students	6	6	1	6
Totals	50	80		43

Estimate of Costs for Recruiting and Paying Respondents

Respondents will not be offered incentives for their participation in this data collection.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to Federal Government

The estimated cost to the federal government for conducting the interviews described here is $31,500, which includes all contractor costs.

WASHINGTON, DC 20006