Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: July 14, 2011
TO: Shelly Martinez, OMB
FROM: Tracy Hunt-White, NCES
THROUGH: Kashka Kubzdela, NCES
SUBJECT: NPSAS:12 Full Scale Cognitive Testing 2011 (OMB No. 1850-0803 v.53)
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.
The request for approval described in this memorandum includes recruiting and screening activities and cognitive testing of questionnaire items for the 2011-12 full scale National Postsecondary Student Aid Study (NPSAS:12). This package provides information about plans and procedures for conducting the cognitive testing in summer 2011.
Housed in NCES’s Postsecondary, Adult, and Career Education Division, NPSAS:12 is a comprehensive study of how students and their families pay for postsecondary education. In addition to providing cross-sectional information about college costs and financing, NPSAS:12 will serve as the base year data collection for the next Beginning Postsecondary Students Longitudinal Study (BPS:12/14, BPS:12/17), provided the BPS options will be exercised by NCES. As a longitudinal study, BPS is able to investigate persistence and enrollment of first-time beginning (FTB) students in less-than-2-year, 2-year, and 4-year institutions, transfer and graduation rates, employment, and student loan debt. BPS, which samples both traditionally- and nontraditionally-aged postsecondary students, is the only nationally-representative, longitudinal survey of first-time beginners in postsecondary education.
examine the thought processes affecting the quality of answers provided to survey questions,
understand the extent to which terms in questions are comprehended,
evaluate the memory demands of the questions,
evaluate the ability of respondents to make calculations and judgments,
determine appropriate presentations of response categories,
assess the time it takes to complete the interview,
assess the navigational problems users face, and
identify sources of burden and respondent stress.
Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and NPSAS cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes for selected items will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session). In addition, cognitive interviewers will employ spontaneous probes as needed. Examples of probes are provided in Attachment V.
RTI is drawing cognitive interview participants from the Research Triangle Park, NC area where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located. Recruitment will also be conducted in the Washington, D.C. area where RTI also has offices. Both locations are home to a large number of postsecondary institutions, making them particularly well-suited for recruiting the target population. In the event that we are unable to recruit adequate numbers of students to come to the RTP or DC offices, we will make arrangements with institutions in DC and/or RTP to conduct the cognitive interviews on campus. We will ensure that the space is private so the interviews cannot be overheard (such as a study room in a library or classroom). Attachment I provides additional detail about recruitment procedures.
A total of 30 cognitive interviews are planned. Interviews will be conducted in person and, if necessary, by telephone. Students will be selected from all levels of postsecondary institutions (4-year, 2-year, and less than 2-year) and will be selected to provide broad representation of the FTB population based on age, sex, and other key characteristics. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality.
The cognitive interviews will be held in facilities that are centrally located, easily accessible by car and public transportation, and allow for audio recording. Interviews will be conducted at times convenient for student schedules. The cognitive interview process is expected to average 90 minutes for each participant but could take as long as 2 hours. Interviews will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.
The NPSAS:12 field test was completed on June 30, 2011. Based upon review of field test data and monitored telephone interviews, Attachment IV identifies the interview items selected to be the focus of full scale cognitive interviews. Probing on these items and feedback from cognitive interviews will help to determine more efficient and effective ways to ask questions of respondents. Cognitive interview participants will complete the entire questionnaire to provide respondents with overall context for the interview items of focus identified in Attachment IV. While cognitive interview techniques, including think-alound questions and probes, will be focused primarily on these items, feedback from respondents on all interview items will be accepted and considered for the improvement of the full scale instrument. Example probes are identified in Attachment V.
Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.
Cognitive interview participants will be informed that their participation is voluntary and their responses are protected by federal statute (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Attachment III). Students will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form. The signed student consent forms will be kept separately from student interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.
Schedule
The NPSAS:12 technical review panel will meet on August 16-17, 2011, and the cognitive interviews will take place August through September, 2011.
In total, 30 individual cognitive interviews are planned. Interview sessions are expected to average 90 minutes but could last up to 2 hours to allow cognitive interview participants to review the entire survey and items of focus.
Estimated respondent burden
Institution Level |
Number of Respondents |
Number of Responses |
Hours per Respondent |
Total Burden Hours |
4‑year Postsecondary Institution Student Recruitment & Screening |
16 |
16 |
.066 hr (4 minutes) |
1 |
2-Year Postsecondary Institution Student Recruitment & Screening |
16 |
16 |
.066 hr (4 minutes) |
1 |
Less that 2-year Postsecondary Institution Student Recruitment & Screening |
16 |
16 |
.066 hr (4 minutes) |
1 |
4‑year Postsecondary Institution Student Cognitive Interview |
10 (5 Web/5 CATI) |
10 |
2.0 |
20 |
2-Year Postsecondary Institution Student Cognitive Interview |
10 (5 Web/5 CATI) |
10 |
2.0 |
20 |
Less that 2-year Postsecondary Institution Student Cognitive Interview |
10 (5 Web/5 CATI) |
10 |
2.0 |
20 |
Totals |
48* |
78 |
- |
63 |
Given the importance of recruiting a representative range of respondents and the difficulty typically experienced recruiting portions of the target population, especially students who are both working and enrolled in an institution, we will offer prospective respondents $50 for completing the 90-120-minutes cognitive interview. This incentive amount was successfully used in the NPSAS:12 field test cognitive interviews (OMB# 1850-0803 v.32, approved on 7/15/2010).
There are no direct costs to participants.
The estimated cost to the federal government for conducting cognitive interviews for NPSAS:12 is $50,629. Included in the contract estimates are contractor staff time, incentives, and project materials such as advertisements for recruiting.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Memorandum |
| Author | mcominole |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |