National Center for Education Statistics
Volume I
Supporting Statement
2012/17 Beginning Postsecondary Students Longitudinal Study (BPS:12/17) Cognitive and Usability Testing – Second Round
Attachments:
Attachment I – Respondent Recruitment Procedures and Materials
Attachment II – Consent to Participate in Research
Attachment III – Eligibility Screening Questions
Attachment IV – Post Eligibility Script for Device Determination
Attachment V – Cognitive Interview Protocol
Attachment VI – Cognitive Interview Survey Items
September 8, 2015
Revised September 23, 2015
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB # 1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pre-tests, focus groups, and cognitive interviews.
This request is to conduct a second round of cognitive and usability interviews in preparation for the 2012/17 Beginning Postsecondary Students Longitudinal Study (BPS:12/17) with recruitment starting in September of 2015. The data collection for this study is being carried out for NCES by RTI International – with Research Support Services (RSS) as a subcontractor – under contract to the U.S. Department of Education (Contract # ED-IES-09-C-0039). The cognitive research report summarizing the results will be available by January 20th, 2016. The results from this second round, combined with feedback from the previous first round and the recent technical review panel (TRP), will be used to prepare a survey instrument for a pilot test in February of 2016. This submission includes recruitment and screening materials, the informed consent form, the questions to be tested, the scheduling and device determination, and the protocol for conducting the cognitive interviews.
The BPS longitudinal study is designed to follow a cohort of students who enroll in postsecondary education for the first time during the same academic year, irrespective of the date of high school completion. The study collects data on student persistence in and completion of postsecondary education programs; their transition to employment; demographic characteristics; and changes over time in their goals, marital status, income, and debt, among other indicators. Data from BPS are used to help researchers and policymakers better understand how financial aid influences persistence and completion, what percentages of students complete various degree programs, what are the early employment and wage outcomes for certificate and degree attainers, and why students leave school.
Sampled from the 2011-12 National Postsecondary Student Aid Study (NPSAS:12), BPS:12/17 is the second follow-up interview with sample members who were first-time beginning college students (FTBs) during the 2011-12 academic year. The first contact was made in 2012 as part of the NPSAS:12 data collection, and the second in 2014 as part of BPS:12/14.
Topics for cognitive testing were considered against the overarching purpose of BPS to collect data on persistence, attainment, and labor market outcomes, as well as the human capital framework that informs data collection for this cohort.The items listed in this package will be included for the second round of cognitive testing because they are new to the BPS study, new for this BPS cohort, or present opportunities to improve existing items. The topics include online course taking; remedial course taking; familiarity with and participation in competency-based education and prior learning assessements; financial literacy; loan repayment and the impact of loans on employment; and the time it takes to earn a degree.
The requested cognitive and usability testing will be used to refine the survey questions to maximize the quality of data collected, and provide information on issues with important implications for the survey design, such as the following:
The manner and degree to which terms in the questions and instructions are comprehended;
Memory search strategies commonly used, and appropriate cues to facilitate recall;
Memory demands of the questions and the answering strategies used;
The appropriate levels at which conceptual categories should be presented;
Respondents’ ability to make calculations and judgments;
Sources of burden and respondent stress; and
A previous round of cognitive testing of a subset of draft questions (OMB# 1850-0803 v.134) was conducted from May 2015 through July 2015. It did not test the programmed instrument due to timing constraints. This request is to conduct a second round of testing focusing on questionnaire content and understanding and usability of the self-administered instrument (web and mobile-friendly versions).
The second round of cognitive testing of BPS:12/17 will be conducted for NCES by its contractor, RTI International, with RSS as a subcontractor. RSS, located in Evanston, IL, anticipates screening during recruitment up to 80 individals in the Chicago metro area in order to ensure 20 interviews are completed during the second round of testing. Cognitive interview participants will be selected to provide representation of the specific populations of the BPS:12/17 study. This distribution includes individals who were first enrolled between July 1, 2010 and June 20, 2012 at institutions with particular characteristics, including level of degree offered (less-than 2-year, 2-year, and 4-year) and institution control (public, private not-for-profit, private for-profit). The sample will include both program/degree completers, those still enrolled, those who left prior to completing a degree, as well as sample members with financial aid, and who are currently married.
To address multiple components of the respondents’ experiences participating in the BPS survey, usability testing will be conducted on tablets, Android phones, and iPhones in addition to laptop computers. Those respondents to the eligibility screener who meet the criteria for and are interested in participating in the study (approximately half of those who are screened), will then be asked if they have and are willing to use their own smartphone (Android or iPhone) or tablet for the study. Respondents who do not own or wish to use those devices, but who do still wish to participate in the study, will be provided a tablet or laptop device to use during the survey. To maximize efficiency and minimize burden, the cognitive testing and usability testing will be conducted simultaneously.
To recruit respondents, RSS will advertise on a variety of websites, such as Facebook and Craigslist, as well as use direct outreach, such as postings at area businesses. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, one of the designated recruiters will use a set of scripted screening questions (Attachments III and IV) to determine his/her eligibility for the cognitive interview.
Table 1, below, provides a distribution of the desired minimum number of respondents by type, although a respondent may fit more than one category.
Trained RSS staff will conduct the cognitive interviews in rented conference rooms, with the goal of completing an average of five cases per day. The cognitive testing will involve intensive one-on-one interviews. The survey will be developed and tested as a web instrument; the survey items to be tested are provided in Attachment VI. Each interview will last a maximum of 90 minutes, and all participants will be offered an incentive of $40 to thank them for their time, effort, and participation. Audio recording of each interview will be available to NCES for review.
The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions, and to pinpoint potential sources of question misinterpretation in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant responses, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions, to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes – e.g., “How certain are you of your answer?” or “How easy or difficult was it to answer this question?” – seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or elicit notions that the respondent thought were critically relevant to but absent from the question. Usability testing will evaluate the design of the survey’s user interface for ease of navigation, response, and use of of supporting features (e.g., help text, progress bar). Respondents will be asked about the visual display, layout of the questions, ease of navigation, and other aspects important for completing the interview.
Table 1: Desired minimum number of respondents, by characteristics
Characteristic |
Desired number of respondents |
Institution control |
|
Public |
6-8 |
Private, for-profit |
6-8 |
Private, not-for-profit |
3-4 |
Institution level |
|
4-year |
6-8 |
2-year |
6-8 |
Less-than-2-year |
3-4 |
Student enrollment type |
|
Completer |
10-12 |
Leaver |
3-4 |
Still enrolled |
3-4 |
Other characteristic |
|
Had financial aid |
At least 5 |
Currently married |
At least 5 |
A cognitive interviewing protocol (Attachment V) will be used to guide the flow and content of the cognitive interviews. Instructions for cognitive interview respondents and probes inquiring about the meaning of key terms or constructs will be included in the protocol. Interviewers will refer to the protocol to guide the content of the interviews, but will also be free to deviate from the guide should participants have difficulty answering questions that do not have scripted probes or prompts.
Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RSS will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report.
Cognitive and usability interview participants will be informed that their participation is voluntary and that:
“Research Support Services (RSS) and RTI International are carrying out this research for the National Center for Education Statistics (NCES) of the U.S. Department of Education. NCES is authorized to conduct this study under the Education Sciences Reform Act (20 U.S.C., § 9543). Your participation is voluntary. Your responses are protected from disclosure by federal statute (20 U.S.C., § 9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.”
Participants will be assigned a unique identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to their name in any way. Participants will sign an informed consent form (Attachment II) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is completed.
RSS will begin recruiting for the second round of cognitive and usability testing upon receiving OMB clearance, and the testing is scheduled to begin by early October 2015. With feedback from the these two rounds of cognitive and usability testing, a final draft of the survey will be used in a pilot test with approximately 3,120 sample members, beginning in February 2016. The generic clearance request for the pilot test will be submitted to OMB for review and approval in winter of 2015-16.
To yield 20 completed interviews, we anticipate screening up to 80 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 5 minutes per person. This burden includes the time it would take for the screened respondents who are determined to be eligible (approximately half) to be administered the follow-up device questions via the Post Eligibility Script for Device Determination (Attachment IV).
Table 2: Estimate respondent burden
Activity |
Number of respondents |
Number of responses |
Minutes per respondent |
Maximum total burden hours |
Screening |
80 |
80 |
5 |
7 |
Cognitive and usability interview |
20* |
20 |
90 |
30 |
Study Total |
80 |
100 |
|
37 |
In order to encourage participation in this study, and to thank respondents for their time and effort spent traveling to the interview facility and completing the interview, each respondent will be offered $40 for their participation.
Participants in the cognitive and usability interviews must bear direct costs associated with travel by car or use of Chicago Transit Authority services to RSS’s rented conference rooms in Chicago.
The cost to federal government for conducting these cognitive and usability interviews will be $35,357, under the RSS subcontract to RTI, including recruitment, interviewing, analysis, report writing, and participant incentives.
Attachment I
Respondent Recruitment Procedures and Materials
Identification of Participants
RSS will focus recruitment efforts in the Chicago metropolitan area, particularly among postsecondary institutions. RSS uses a variety of recruiting methods to generate sufficient numbers of callers from potential respondents, including:
Outreach to educational institutions, organizations, community leaders, and affinity groups that can distribute flyers among targeted groups. Many of these organizations and venues have worked with RSS before. Because they are familiar with the populations they serve, these organizations are often able to help RSS target specific populations;
Flyers, print and online advertisements; and
Word of mouth.
Advertisements will be targeted toward individuals qualified for this study, describing briefly the purpose of the cognitive interviews and the opportunity for respondents to contribute to the development of a survey. The advertisements will identify the requirement that participants began college between 2010 and 2012, as well as the time commitment of the interview, and the incentive amount for participation. Sample print and online advertisements are presented below. The exact months of eligibility will not be included on the flyers in order to prevent potential participants from giving time frames that match eligibility rather than their own experience.
Sample Recruitment Flyer
DID YOU START COLLEGE FOR THE FIRST TIME between 2010 and 2012?
You may be eligible to help us review a 90-minute education survey and receive $40 for your time!
We are interested in talking to individuals who first started college, university, or vocational/trade school at any time between 2010 and 2012, even if you are no longer enrolled in school.
Interviews will be scheduled at a time convenient to you.
Research Support Services is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.
Please call Barbara at 1-8XX-XXX-XXXX to complete a brief screening to see if you are eligible to participate.
Sample Online Ad
Title:
First started college between 2010 and 2012: receive $40 for a survey!
Ad text:
Did you start college for the first time between 2010 and 2012?
You may be eligible to help us review a 90-minute education survey and receive $40 for your time!
We are interested in talking to individuals who first started [college, university, vocational/trade school/certification program] at any time between 2010 and 2012, even if you are no longer enrolled in school.
Interviews will be scheduled at a time convenient to you.
Research Support Services is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.
Email us:
[email protected]
Call: 1-8XX-XXX-XXXX
Alternative flyer and ad text:
Flyers and ads will have headlines matched to the specific criteria required to test the instrument, sample headlines, which may be combined, include:
Married?
Started college between 2010 and 2012?
Did you graduate from a for-profit [college, university, vocational/trade school/certification program]?
[Have you attended/are you enrolled at] a for-profit [college, university, vocational/trade school/certification program]?
[Have you attended/are you enrolled at] a not for profit [college, university, vocational/trade school/certification program]
Left [college, university, vocational/trade school/certification program] without a degree?
Student loans?
Do you and your spouse both have student loans?
Recruitment
The recruitment screener, a 5 minute phone script (Attachment III), will be used to elicit both the educational status and eligibility criteria needed to test the items identified for cognitive testing. Potential respondents will call a toll-free number and an experienced recruiter will screen the caller for eligibility for the project. Screening includes an opportunity to provide the potential respondent with a brief description of the project, the purpose of the interview, and questions about the specific characteristics needed. RSS’s experience on past projects with recruitment suggests that it is important to elicit respondent characteristics without asking the same questions which are intended to test and thus priming the individual. RSS’s screening methods are designed to minimize the otherwise negative impact of the “professional respondents” on the quality of the study results.
Attachment II
Consent to Participate in Research
Title of Research: Beginning Postsecondary Students Longitudinal Study – Cognitive Interview
You, along with other students who began college between 2010 and 2012, are being asked to participate in in-depth interviews being conducted by RTI International and Research Support Services (RSS) for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The purpose of the interviews is to obtain your feedback on some of the questions that will be used in the Beginning Postsecondary Students Longitudinal Study.
You are one of 20 individuals participating in these cognitive interviews.
The interviews will be audio recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recordings will be destroyed. Your name will never be used in the report that we write.
Your participation in an interview will take no more than 90 minutes.
We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.
You will receive $40 to thank you for your participation in this study.
These interviews will help us develop and improve the questionnaires used in the national study so that they are clear and can be easily understood and answered.
“Research Support Services (RSS) and RTI International are carrying out this research for the National Center for Education Statistics (NCES) of the U.S. Department of Education. NCES is authorized to conduct this study under the Education Sciences Reform Act (20 U.S.C., § 9543). Your participation is voluntary. Your responses are protected from disclosure by federal statute (20 U.S.C., § 9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.”
We will not contact you in the future as part of this study.
Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.
If you have any questions about the study, you may call Dr. Alisú Schoua-Glusberg at Research Support Services (847-864-5677) or Jason Hill at RTI International (919-541-6425). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).
YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.
Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.
|
|
|
Signature of Participant |
|
Date |
|
|
|
Printed Name of Participant |
|
|
I certify that the nature and purpose, the potential benefits, and possible risks associated with participating in this research have been explained to the above-named individual.
|
|
|
Signature of Person Obtaining Consent |
|
Date |
|
|
|
Printed Name of Person Obtaining Consent
|
|
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Wine, Jennifer S. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |