Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: May 6, 2013
TO: Shelly Martinez, OMB
FROM: Sarah Crissey, NCES
THROUGH: Kashka Kubzdela, NCES
SUBJECT: BPS:12/14 Cognitive Testing (OMB# 1850-0803 v.79)
The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides NCES the capability to test and improve data collection methodology, question/response wording, and delivery methods of its surveys and assessment instruments via cognitive interviews.
We request approval to conduct cognitive testing, beginning in June 2013, in preparation for the 2012-14 Beginning Postsecondary Students Longitudinal Study (BPS:12/14) full-scale data collection (OMB# 1850-0631). Described in this submission are recruiting, screening, and testing procedures for the questionnaire items to be used in the full-scale data collection.
The results of this round of cognitive testing will be supplemented with results of the BPS:12/14 field test data collection currently underway, to be presented at the technical review panel (TRP) meeting on August 1-2, 2013. Based on feedback from the TRP, a revised interview protocol will be submitted for clearance prior to subsequent cognitive testing in August and September 2013. Results from the cognitive testing will be used to refine the full-scale questionnaire, which will be submitted for OMB’s review in October 2013, as part of request for the full-scale data collection clearance.
The BPS:12/14 full-scale data collection will be conducted by NCES. The sample for the study was drawn from the 2011-12 National Postsecondary Student Aid Study (NPSAS:12) and is the first follow-up interview with students who were first-time beginning college students (FTBs) during the 2011-12 academic year. The purpose of BPS is to track FTBs’:
Experiences and continued enrollment in postsecondary education;
Degree and certificate completion;
Employment during and following postsecondary education;
Debt and finances;
Career goals and expectations; and
Family formation.
As the first follow-up, BPS:12/14 will be the second time that FTBs will be contacted. The first contact was made in 2012 as part of the NPSAS:12 data collection. FTBs in the BPS:12 cohort will be contacted again in 2017, and a postsecondary transcript collection is also planned for 2017.
The BPS:12/14 field test data collection is currently underway. The results from this first round of cognitive testing for which clearance is requested will be combined with evaluation of the field test data, observations from interview monitoring, and meetings with interviewing staff. The results will be presented for discussion with the TRP in August 2013. The cognitive testing process will enable the instrumentation team to:
Examine the thought processes affecting the quality of answers provided to survey questions;
Understand the extent to which terms in questions are comprehended;
Evaluate the memory demands of the questions;
Evaluate the ability of respondents to make calculations and judgments;
Determine appropriate presentations of response categories;
Assess the time it takes to complete the interview;
Assess the navigation process for potential problems a user might face; and
Identify sources of burden and respondent stress.
Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and BPS:12/14 cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes prepared in advance of testing will be both concurrent, asked at the same time the subject answers the questions, and retrospective, asked during a debriefing session. Methodologists will use spontaneous probes as needed.
RTI will recruit cognitive interview participants from the greater Research Triangle Park (RTP) area in North Carolina who enrolled at public, private nonprofit, and private, for-profit 4-year, 2-year, and less-than-2-year institutions during their first year of postsecondary education in 2011-12; continued enrollment is not required. If required to ensure representation of the sectors, RTI will also recruit and conduct cognitive testing out of their Washington, DC and Chicago, Illinois offices, and conduct telephone cognitive interviews with recruits in the area surrounding RTI’s Berkeley, California office. Approximately 30 cognitive interviews are planned, half before the August TRP meeting and half after the meeting. Interviews will be conducted in person and by telephone in order to evaluate both self-administered and interviewer-administered modes of the instrument. Special flyers and online announcements will be used to attempt to recruit participants who have attended a beauty/trade school at any time since first starting college in 2011-2012, in an effort to adequately represent this group in the cognitive interviews. Participants will be offered $40 to thank them for their time and participation. Interviews will be conducted in Research Triangle Park and possibly Washington, DC, and Chicago, Illinois, at sites accessible by public transportation, as well as by telephone at times that are convenient for participants. Each interview will take about 90 minutes and will be led by an RTI staff member with expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.
Attachment I submitted with this memorandum presents the materials that will be used for recruitment of cognitive interview participants. Attachment II presents the screening questions that will be used (via telephone and website) to determine eligibility for cognitive interview participation and attachment III contains the Assurance of Confidentiality. The interview questions are presented in attachment IV.
Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s BPS:12/14 project staff and to NCES.
Cognitive interview participants will be informed that their participation is voluntary and their responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573) (Attachment III). Participants will be assigned a unique student identifier (ID) which will be created solely for file management and used to keep all materials together. The participant ID will not be linked to student name in any way. The signed consent forms will be kept separately from interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.
Schedule for BPS:12/14 Cognitive Testing
Recruit participants and conduct cognitive testing |
Immediately upon OMB clearance |
TRP meeting |
August 1 and 2, 2013 |
Submit change memo |
August 12, 2013 |
Recruit participants and conduct cognitive testing (pending approval of change memo) |
August 12-September 6, 2013 |
Up to 30 individual cognitive interviews are planned, depending on scheduling and participant availability. Interview sessions are expected to average approximately 90 minutes. This will allow participants to review the entire survey, provide think aloud responses, and respond to interviewer probes.
|
Number of Respondent |
Number of Responses |
Hours per Respondent |
Total Burden Hours |
Recruitment |
60 |
60 |
0.0667 |
4 |
Cognitive Interview |
30 |
30 |
1.5 |
45 |
Total Burden |
60 |
90 |
- |
49 |
We will offer prospective respondents $40 for completing the 90-minute cognitive interview. No compensation will be offered for the screening process as the response burden is minimal.
Other than transportation to the testing facility for in-person interviews, there are no direct costs to participants.
The estimated cost of conducting the cognitive interviews will be about $51,700. This estimate includes costs for contractor staff time, compensation to participants, and project materials such as advertisements for recruiting.
Attachment I
Recruitment Materials
Sample Recruitment Flyer
Did you start college for the first time in 2011-2012?
You may be eligible to help us review a 90-minute EDUCATION SURVEY and RECEIVE $40 for your time!
General
We are interested in talking to individuals who first started college, university, or vocational/trade school at any time between July 1, 2011 and June 30, 2012, even if you are no longer enrolled in school.
Beauty and Trade School Students
We are interested in talking to individuals who first started college, university, or vocational/trade school at any time between July 1, 2011 and June 30, 2012, or who attended a beauty or trade school at any time since 2011.
RTI International is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.
Flexible appointment openings, including weekends and evenings to meet your scheduling needs. Email us: [email protected] or Call: 1-8XX-XXX-XXXX
RTI International is a non-profit research firm located in Research Triangle Park near Cornwallis Road and Davis Drive, with easy access to and from I-40./DC FLYER: RTI International is a non-profit research firm located in downtown Washington, D.C./CHICAGO FLYER: RTI International is a non-profit research firm located in the Loop in Downtown Chicago.
Sample Recruitment Online Ad
FIRST STARTED COLLEGE IN 2011-2012: RECEIVE $40 FOR A SURVEY
IF YOU ...
First started college, university, or trade school at any time between July 1, 2011 and June 30, 2012, even if you are no longer enrolled in college…
FOR THE FOLLOWING SPECIAL INTEREST GROUP– WE WILL HAVE A SEPARATE AD TO RECRUIT ESPECIALLY FOR THIS GROUP
or attended a beauty or trade school at any time since 2011…
WE NEED YOU TO...
Answer a few questions about your experiences in college and at work
You may be eligible to help us pretest a college survey and RECEIVE $40 for your time!
Email us: [email protected] or Call: 1-8XX-XXX-XXXX
Our appointment schedule is flexible, with weekend and evening sessions.
Your responses will be used to refine survey questions.
RTI International is conducting this survey for the National Center for Education Statistics located within the U.S. Department of Education.
RTI is a non-profit research firm located in Research Triangle Park near Cornwallis Road and Davis Drive, with easy access to and from 1-40./ DC FLYER: RTI International is a non-profit research firm located in downtown Washington, D.C./CHICAGO FLYER: RTI International is a non-profit research firm located in the Loop in Downtown Chicago.
On average, these sessions take about 90 minutes.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Memorandum |
| Author | mcominole |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |