B and B 2008-12 Full Scale Items Cog Labs Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: September 1, 2011

TO: Shelly Martinez, OMB

FROM: Ted Socha, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: B&B:08/12 Cognitive Testing (OMB# 1850-0803 v.56)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides NCES the capability to test and improve data collection methodologies, question/response wording; and delivery methods of its surveys and assessment instruments via cognitive interviews. This submission requests approval to conduct cognitive testing in preparation for the 2008-12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12) full-scale data collection (OMB# 1850-0729).

Described in this memorandum are recruiting and screening procedures and cognitive testing of suggested questionnaire items to be used in the full-scale data collection. All of these activities are informed by the results of the B&B:08/12 field-test cognitive testing (see cog lab report in Attachment V) as well as field test phone-/web-based interviewing (currently underway).

Background

The B&B:08/12 study is to be conducted by NCES’ Postsecondary, Adult, and Career Education Division (PACE). The sample was drawn from the 2007/08 National Postsecondary Student Aid Study (NPSAS:08) and is the second follow-up of 2007-08 baccalaureate degree recipients, in the 4^th year after undergraduate completion. Its intention is to track sample members’ experiences with:

• further postsecondary education enrollment and completion;

• employment outcomes;

• debt and finances;

• family formation

• civic engagement and volunteerism; and

• K-12 teaching.

In addition to providing the public with a general study of labor market outcomes for the cohort as a whole, B&B allows for the comparison between teachers and other occupations. Of particular relevance to B&B:08/12 are questions concerning whether American colleges and universities are preparing enough graduates in the fields of science, technology, engineering, and mathematics (STEM) and whether college graduates with training in these fields are using that training in the workplace or pursuing graduate education in STEM fields.

As the second follow-up, B&B:08/12 represents the third time that the B&B:08 cohort will be contacted. The first contact was made in 2008 as part of NPSAS:08 base year. NPSAS:08 sample members that completed a baccalaureate degree that year were contacted again in 2009, one year after completing their degree, as part of B&B:08/09. The B&B:08/12 field test data collection is currently underway. Evaluations conducted in the field test, including a thorough review of the data, observations from interview monitoring, and meetings with interviewing staff, have informed the changes to be tested in the first round of cognitive testing.

The cognitive testing process will enable the instrumentation team to:

• examine the thought processes affecting the quality of answers provided to survey questions;

• understand the extent to which terms in questions are comprehended;

• evaluate the memory demands of the questions;

• evaluate the ability of respondents to make calculations and judgments;

• determine appropriate presentations of response categories;

• assess the time it takes to complete the interview;

• assess the navigation process for potential problems a user might face; and

• identify sources of burden and respondent stress.

Once the cognitive testing is complete, RTI will submit the items and test results from cognitive testing to the Q Bank¹ -- a federal repository of questions that have been tested along with the results.

Design and Context

Cognitive Testing. Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and B&B:08/12 cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session) and will be prepared ahead of time. In addition, methodologists will use spontaneous probes as needed.

RTI will recruit cognitive interview participants from two locations: the greater Research Triangle Park (RTP) area in North Carolina and Chicago. While RTI’s North Carolina headquarters house the Laboratory for Survey Methods and Measurement, both locations are equipped with the trained staff and the facilities needed to conduct cognitive interviews. Both RTP and Chicago are particularly well-suited for recruiting and conducting cognitive testing as both locations have a large population of recent baccalaureate degree recipients.

In this round, up to 30 cognitive interviews will be conducted in person and by telephone in order to evaluate both self-administered and interviewer-administered modes of the instrument. Participants will be selected to provide representation of key analytic areas of interest (e.g., teachers, those who majored in STEM fields, graduate students, etc.). Attachment I presents the materials that will be used for recruitiment. Attachment II presents the screening questions that will be used (via telephone and website) to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality. The interview questions are presented in Attachment IV, where any substantive revisions made since the field test have been highlighted and listed in the table of contents.

The cognitive interviews will be held in RTP and Chicago in centrally located facilities that are easily accessible by car or public transportation, and that allow for professional audio recording. Interviews will be conducted at times that are convenient for participants. Each interview will take about 90 minutes (to allow participants to review the entire survey, provide think aloud responses and respond to interviewer probes) and will be led by an RTI staff member with expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.

Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573) (Attachment III). Participants will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student name in any way. The signed consent forms will be kept separately from interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule for B&B:08/12 OMB requests and related activities

We plan to conduct the cognitive testing as soon as possible this fall, so that the results from the first round of cognitive testing can be presented at the technical review panel (TRP) meeting on November 14, 2011. Based on feedback from the TRP, a subsequent clearance request with a revised questionnaire will be submitted for approval in November, and the second round of cognitive tests will be conducted in November and December 2011. Results from the cognitive testing will be used to refine the full-scale interview, and final wording will be submitted for OMB’s review in late January, 2012. Full-scale data collection will begin in July 2012.

Estimated Respondent Burden

	Number of Respondents	Number of Responses	Hours per Respondent	Total Burden Hours
Recruitment /Screener	45	45	.066 hr (4 minutes)	3
Cognitive Interview	30	30	1.5 hr	45
Total	45	75		48

Estimate of Costs for Recruiting and Paying Respondents

To thank them for their time and effort, respondents will be offered $40 for completing the 90-minute cognitive interview.

Cost to Federal Government

The estimated cost of conducting the cognitive interviews will be about $51,700. This estimate includes costs for contractor staff time, incentives, and project materials such as advertisements for recruiting.

1 http://wwwn.cdc.gov/QBANK/Report/Miller_2005.pdf

B&B12 Cognitive Testing Summary Report C-i