Justification for NCES ECLS-K cog lab

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: January 21, 2011

TO: Shelly Martinez, OMB

FROM: Ted Socha, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: B&B:08/12 Field Test Items Cognitive Testing (OMB No. 1850-0803 v.42)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB No. 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.

The request for approval described in this memorandum includes recruiting and screening activities and cognitive testing of the questionnaire items to be used in the 2008-12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12). This package provides information about plans and procedures for conducting the cognitive testing, and presents the question wording to be tested in the March 2011 cognitive interviews.

Background

The B&B:08/12 study is housed in NCES’s Postsecondary, Adult, and Career Education Division. The B&B:08/12 sample is drawn from the 2007/08 National Postsecondary Student Aid Study (NPSAS:08) and is the second follow up of 2007-08 baccalaureate degree recipients four years after completing requirements for a baccalaureate degree. B&B:08/12 tracks sample member’s experiences with continuing education, degree attainment, graduate education, employment, debt and finances, family formation, civic engagement and volunteerism, and interest in and preparation for K-12 teaching. There is a strong focus on entry into and persistence in the K-12 teacher workforce. Information on teacher’s professional experiences will be collected, which will allow a comparison of the teaching profession to other occupations as well as a general study of labor market outcomes for the cohort as a whole. Of particular relevance to B&B:08/12 are questions concerning whether American colleges and universities are preparing enough graduates in the fields of science, technology, engineering, and mathematics (STEM ) and whether college graduates with training in these fields are using that training in the workplace or pursuing graduate education in STEM fields.

B&B:08/12 represents the third time that the B&B:08 cohort will be contacted. The first contact was made in 2008 as part of NPSAS:08 base year. NPSAS:08 sample members that completed a baccalaureate degree that year were contacted again in 2009, one year after completing their degree, as part of B&B:08/09 follow up. B&B:08/12 is the four year follow-up of the 2008 baccalaureate degree recipients.

Instrument development is currently underway. The cognitive testing process will enable the instrumentation team to

• examine the thought processes affecting the quality of answers provided to survey questions;

• understand the extent to which terms in questions are comprehended;

• evaluate the memory demands of the questions;

• evaluate the ability of respondents to make calculations and judgments;

• determine appropriate presentations of response categories;

• assess the time it takes to complete the interview;

• assess the navigational problems users face; and

• Identify sources of burden and respondent stress.

Once the cognitive testing is complete, RTI will submit the items and test results from both evaluations (focus groups and cognitive testing) to the Q Bank¹ -- a federal repository of questions that have been tested along with the results.

Design and Context

Cognitive Testing. Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and B&B:08/12 cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session) and will be prepared ahead of time. In addition, methodologists will employ spontaneous probes as needed.

RTI will draw cognitive interview participants from the greater Research Triangle Park, where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located and where a large number of recent baccalaureate degree recipients are located, making it particularly well-suited for recruiting the target population. Approximately 45 cognitive interviews are planned, with up to three rounds of 15 interviews each. Interviews will be conducted in person and by telephone in order to evaluate both a self-administered web and interviewer-administered telephone version of the instrument. Participants will be selected to provide representation of key analytic areas of interest (e.g., teachers, those who majored in STEM fields, graduate students, etc.). Participants will be compensated for their time. Attachment I provides additional detail about recruitment procedures. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality. The interview questions are presented in Attachment IV².

The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Interviews will be conducted at times that are convenient for participants. Each interview will take no more than 90 minutes and will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.

Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573) (Attachment III). Participants will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student name in any way. The signed consent forms will be kept separately from interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule for B&B:08/12 OMB requests and related activities

Recruit Participants	February 2011
Conduct cognitive testing	February-March 2011
Finalize revisions to item wording	15 March 2011

Estimate of Hour Burden

Up to 45 individual cognitive interviews are planned, depending on scheduling, participant availability, etc. Each interview session is expected to last approximately 90 minutes. This will allow participants to review the entire survey, provide think aloud responses and respond to interviewer probes.

Estimated respondent burden

Maximum Number of Recruits	Hours per recruit	Hours for recruiting	Number Selected	Hours per respondent	Hours for Cognitive Testing	Maximum Total Burden Hours
60	.066 hr (4 minutes)	4.0	45	1.5	67.5	71.5

Estimate of Costs for Recruiting and Paying Respondents

Given the importance of recruiting a representative range of respondents and the difficulty typically experienced recruiting portions of the target population; we will offer prospective respondents $40 for completing the 90-minute cognitive interview. No compensation will be offered for the screening process as the response burden is minimal.

Estimate of Cost Burden

Other than transportation to the testing facility for in-person interviews, there are no direct costs to participants.

Cost to Federal Government

The cost of conducting the cognitive interviews will be $62,412. The following table provides the overall project cost estimates:

Cognitive Interviews, 2011
1 Staff Costs	$28,521
2 Other Project Materials	$5,061
Newspaper Advertisements for Recruiting	$2,678
Incentives for Student Participation	$1,800
Photocopy/Shipping	$53
Telecommunications	$80
Indirect Costs	$24,894
Total Cost of Task Without Fee	$58,026
Fixed Fee	$2,047
Award Fee	$2,339
Total Cost of Task	$62,412

Attachment I

Recruitment Procedures

RTI will draw cognitive interview participants from the greater Research Triangle Park, where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located and where a large number of recent baccalaureate degree recipients are located, making it particularly well-suited for recruiting the target population. Between 24 and 45 cognitive interviews are planned, depending on time and availability of participants. The interviews will be conducted in multiple rounds. Interviews will be conducted in person and by telephone in order to simulate both a self-administered web and interviewer-administered telephone versions of the instrument.

Participants will include baccalaureate degree recipients who completed the requirements for their degree between July 1, 2006, and June 30, 2007. Important sub-groups of interest include those who are currently enrolled in college or graduate school, elementary or secondary education teachers, and those who earned a bachelor’s degree in a science, technology, engineering or mathematics (STEM) field. Participants will be offered a $40 cash incentive for completing the interview.

RTI will place advertisements using online classifieds, including Craigslist (http://raleigh.craigslist.org/), the RTI area newspaper classifieds, and online teacher chat boards (www.teachers.net). RTI will also post flyers at a variety of public locations. Interested individuals will be asked to call the RTI recruitment coordinator who will screen them for eligibility using a scripted series of questions and set up an appointment for the interview. Attachment I shows the materials that will be used for recruitment. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality. Item wording for the interview is presented in Attachment IV.

The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Interviews will be conducted at times that are convenient for participants. Each interview will take approximately 90 minutes, and will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.

Sample Recruitment Flyer

Did you graduate from college in 2007?

You may be eligible to help us review a 90-minute EDUCATION SURVEY and RECEIVE $40 CASH for your time!

General

We are interested in talking to individuals who completed the requirements for a bachelor’s degree (B.A. or B.S. only) between July 1 2006 and June 30, 2007.

Teachers

We are interested in talking to individuals who completed the requirements for a bachelor’s degree (B.A. or B.S. only) between July 1 2006 and June 30, 2007 who are currently teaching in an elementary or secondary school.

STEM

We are interested in talking to individuals who completed the requirements for a bachelor’s degree (B.A. or B.S. only) in science, technology, engineering or mathematics between July 1 2006 and June 30, 2007.

Enrolled

We are interested in talking to individuals who completed the requirements for a bachelor’s degree (B.A. or B.S. only) between July 1 2006 and June 30, 2007, who are currently enrolled in college or graduate school.

RTI International is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.

Flexible appointment openings, including weekends and evenings to meet your scheduling needs.

CALL OR EMAIL TO LEARN IF YOU QUALIFY

1-800-334-8571, extension 28050

[email protected]

RTI International is a non-profit research firm located in Research Triangle Park near Cornwallis Road and Davis Drive, with easy access to and from I-40.

Sample Recruitment Online Ad

2007 COLLEGE GRADUATES: RECEIVE $40 CASH FOR A SURVEY

IF YOU ...

Completed the requirements for a bachelor’s degree (B.S., B.A.) between July 1, 2006 and June 30, 2007

FOR SPECIAL INTEREST GROUPS – WE WILL HAVE SEPARATE ADS FOR EACH GROUP

• and you have enrolled in college or graduate school since earning your bachelor’s degree…

• and you majored in science, technology, engineering or mathematics…

• and you are a teacher in an elementary or secondary school…

WE NEED YOU TO...

answer a few questions about your experiences since college, with a focus on additional education and employment after receiving your bachelor’s degree

You may be eligible to help us pretest a college survey and RECEIVE $40 CASH for your time!

Email us: [email protected]

Or Call: 1-866-214-2040

Our appointment schedule is flexible, with weekend and evening sessions.

Your responses will be used to refine survey questions.

RTI International is conducting this survey for the National Center for Education Statistics located within the U.S. Department of Education.

RTI is a non-profit research firm located in Research Triangle Park near Cornwallis Road and Davis Drive, with easy access to and from 1-40.

On average, these sessions take about 90 minutes.

Attachment II

Baccalaureate and Beyond

Eligibility Screening Questions

Thank you for inquiring about this study. I need to ask you a few questions to determine if you are eligible to participate in the study. The screening is expected to take approximately 3-5 minutes.

1. Were you enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2006 and June 30, 2007?

• Yes

• No (Thank you for inquiring about this study, but you are not eligible to participate <end>)

2. When did you complete the requirements for a bachelor’s degree?

• Between July 1, 2006 and June 30, 2007

• Some other time (Thank you for inquiring about this study, but you are not eligible to participate <end>)

3. In what month and year were you awarded your bachelor’s degree?

• By June 30, 2008

• Some other time (Thank you for inquiring about this study, but you are not eligible to participate <end>)

4. What is your age?

______

(If under 18 Thank you for inquiring about this study, but you are not eligible to participate <end>)

5. What is your sex?

• Male

• Female

6. In what field did you earn your bachelor’s degree?

• STEM degree (see STEM reference on page 3)

• Non-STEM degree

7. Have you enrolled in any school for an additional degree or certificate (including enrollment for any graduate or additional undergraduate degree or certificate programs) since earning your bachelor's degree?

• Yes

• No

8. Since graduating, have you worked as a teacher at the K-12 level?

• Yes (skip to question 9)

• No

9. Have you done anything to prepare for a teaching career at the K-12 level?

• Yes (skip to question 9)

• No

10. Are you currently considering a career in teaching at the K-12 level?

• Yes

• No

What is your name?

What is the best telephone number to reach you?

What is your email address?

14. What days of the week and times of day would you be available to attend an interview?

Thank you for answering our questions. You are eligible to participate in this study. Participation will involve answering questions and reviewing a survey for an upcoming study that will be used to learn about people’s experiences after graduation. During the interview, we will ask you to complete a survey and give us your thoughts on the questions you are being asked. In addition, we will ask you questions as you go in order to measure your understanding and interpretation of the questions in the survey.

Each full interview is expected to last no longer than 90 minutes. The interviews will be recorded so that we can review your ideas and comments. We will prepare a report that does not single out any one person, but will summarize what we heard from the group as a whole. We will keep the recording only until our report is finished. No names will be mentioned in the report.

Once the interviews have been scheduled, you will be contacted by phone and e-mail to notify you of the date, time, and location. At that point, you must RSVP to the invitation to guarantee an interview time. Upon completion of the interview you will receive $40.

STEM MAJORS FOR REFERENCE

• Computer and Information Sciences

• Artificial Intelligence and Robotics

• Information Technology

• Computer Programming/Programmer

• Computer Programming

• Data Processing and Data Processing Technology

• Information Science/Studies

• Computer Systems Analysis/Analyst

• Computer Science

• Web Page, Digital/Multimedia and Information Resources Design

• Data Modeling/Warehousing and Database Administration

• Computer Graphics

• Computer Systems Networking and Telecommunications

• System Administration/Administrator

• System, Networking, and LAN/WAN Management

• Computer and Information Systems Security

• Web/Multimedia Management and Webmaster

• Engineering

• Aerospace, Aeronautical and Astronautical Engineering

• Agricultural/Biological Engineering and Bioengineering

• Architectural Engineering

• Biomedical/Medical Engineering

• Ceramic Sciences and Engineering

• Chemical Engineering

• Civil Engineering

• Geotechnical Engineering

• Structural Engineering

• Transportation and Highway Engineering

• Water Resources Engineering

• Computer Engineering

• Computer Hardware Engineering

• Computer Software Engineering

• Electrical, Electronics and Communications Engineering

• Engineering Mechanics

• Engineering Physics

• Engineering Science

• Environmental/Environmental Health Engineering

• Materials Engineering

• Mechanical Engineering

• Metallurgical Engineering

• Mining and Mineral Engineering

• Naval Architecture and Marine Engineering

• Nuclear Engineering

• Ocean Engineering

• Petroleum Engineering

• Systems Engineering

• Textile Sciences and Engineering

• Materials Science

• Polymer/Plastics Engineering

• Construction Engineering

• Forest Engineering

• Industrial Engineering

• Manufacturing Engineering

• Operations Research

• Surveying Engineering

• Geological/Geophysical Engineering

• Engineering Technology

• Architectural Engineering Technology

• Civil Engineering Technology

• Electrical, Electronic and Communications Engineering Technology

• Laser and Optical Technology

• Telecommunications Technology

• Biomedical Technology

• Electromechanical Technology/Electromechanical Engineering Technology

• Instrumentation Technology

• Robotics Technology

• Heating, Air Conditioning and Refrigeration Technology

• Energy Management and Systems Technology

• Solar Energy Technology

• Water Quality and Wastewater Treatment Management and Recycling Technology

• Environmental Engineering Technology/Environmental Technology

• Hazardous Materials Management and Waste Technology

• Plastics Engineering Technology

• Metallurgical Technology

• Industrial Technology

• Manufacturing Technology

• Occupational Safety and Health Technology

• Quality Control Technology

• Industrial Safety Technology

• Hazardous Materials Information Systems Technology

• Aeronautical/Aerospace Engineering Technology

• Automotive Engineering Technology

• Mechanical Engineering/Mechanical Technology

• Mining Technology

• Petroleum Technology

• Construction Engineering Technology

• Surveying Technology/Surveying

• Hydraulics and Fluid Power Technology

• Computer Engineering Technology

• Computer Technology/Computer Systems Technology

• Computer Hardware Technology

• Computer Software Technology

• Drafting and Design Technology

• CAD/CADD Drafting and/or Design Technology

• Architectural Drafting and Architectural CAD/CADD

• Civil Drafting and Civil Engineering CAD/CADD

• Electrical/Electronics Drafting and Electrical/Electronics CAD/CADD

• Mechanical Drafting and Mechanical Drafting CAD/CADD

• Nuclear Engineering Technology

• Engineering/Industrial Management

• Biology/Biological Sciences

• Biomedical Sciences

• Biochemistry

• Biophysics

• Molecular Biology

• Molecular Biochemistry

• Molecular Biophysics

• Structural Biology

• Photobiology

• Radiation Biology/Radiobiology

• Biochemistry/Biophysics and Molecular Biology

• Botany/Plant Biology

• Plant Pathology/Phytopathology

• Plant Physiology

• Plant Molecular Biology

• Cellular Biology and Histology

• Anatomy

• Developmental Biology and Embryology

• Neuroanatomy

• Cellular and Molecular Biology

• Cell Biology and Anatomy

• Microbiology

• Medical Microbiology and Bacteriology

• Virology

• Parasitology

• Mycology

• Immunology

• Zoology/Animal Biology

• Entomology

• Animal Physiology

• Animal Behavior and Ethology

• Wildlife Biology

• Genetics

• Molecular Genetics

• Microbial and Eukaryotic Genetics

• Animal Genetics

• Plant Genetics

• Human/Medical Genetics

• Physiology

• Molecular Physiology

• Cell Physiology

• Endocrinology

• Reproductive Biology

• Neurobiology and Neurophysiology

• Cardiovascular Science

• Exercise Physiology

• Vision Science/Physiological Optics

• Pathology/Experimental Pathology

• Oncology and Cancer Biology

• Pharmacology

• Molecular Pharmacology

• Neuropharmacology

• Toxicology

• Molecular Toxicology

• Environmental Toxicology

• Pharmacology and Toxicology

• Biometry

• Biostatistics

• Bioinformatics

• Biotechnology

• Ecology

• Marine Biology and Biological Oceanography

• Evolutionary Biology

• Aquatic Biology/Limnology

• Environmental Biology

• Population Biology

• Conservation Biology

• Systematic Biology

• Epidemiology

• Mathematics

• Algebra and Number Theory

• Analysis and Functional Analysis

• Geometry

• Topology and Foundations

• Applied Mathematics

• Computational Mathematics

• Statistics

• Mathematical Statistics and Probability

• Military Technologies

• Physical Sciences

• Astronomy

• Astrophysics

• Planetary Astronomy and Science

• Atmospheric Sciences and Meteorology

• Atmospheric Chemistry and Climatology

• Atmospheric Physics and Dynamics

• Meteorology

• Chemistry

• Analytical Chemistry

• Inorganic Chemistry

• Organic Chemistry

• Physical and Theoretical Chemistry

• Polymer Chemistry

• Chemical Physics

• Geology

• Geochemistry

• Geophysics and Seismology

• Paleontology

• Hydrology and Water Resources Science

• Geochemistry and Petrology

• Oceanography, Chemical and Physical

• Physics

• Atomic/Molecular Physics

• Elementary Particle Physics

• Plasma and High-Temperature Physics

• Nuclear Physics

• Optical Sciences

• Solid State and Low-Temperature Physics

• Acoustics

• Theoretical and Mathematical Physics

• Biology Technician

• Industrial Radiologic Technology

• Nuclear Power Technology

• Chemical Technology

• Medical Scientist (MS, PhD)

• Actuarial Science

Attachment III

Sample Consent Form

Consent to Participate in Research

Title of Research: 2008-12 Baccalaureate and Beyond Longitudinal Study

Introduction

You are being asked to participate in a cognitive interview which is part of a larger research study sponsored by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The study is designed to examine the decisions and experiences of individuals four years after completing a baccalaureate degree. Before you decide if you want to take part in this study, you need to read this Informed Consent form so that you understand what the interview is about and what you will be asked to do. This form also tells you who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and who you can call if you have questions. Please ask the researcher to explain anything you don’t understand before you make your decision.

Purpose

The interview is being conducted by RTI International, a research organization located in Research Triangle Park, North Carolina.

Participation will involve answering questions and reviewing a survey for an upcoming study that will be used to learn about individuals’ experiences after college graduation. During the interview, we will ask you to complete a web or telephone survey and give us your thoughts on the questions you are being asked. In addition, we will ask you questions as you go in order to measure your understanding and interpretation of the questions in the survey. The information from the interviews will help us finalize development of the questions for the larger study that will be conducted early next year.

Procedures

You are one of approximately 45 individuals participating in these cognitive interviews.

The interview will be audio recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recording will be destroyed. Your name will never be used in the report that we write.

Study Duration

Your participation in the cognitive interview will take about 90 minutes.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits

Your Benefits

You will be paid $40.00 for your participation in this study.

Benefits for Other People

We hope that these interviews will help us develop and improve questions for a national survey that will help us understand individuals’ experiences after completing their baccalaureate degree.

Confidentiality

RTI International is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

Future Contacts

We will not contact you in the future.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.

Your Questions

If you have any questions about the study, you may call Melissa Cominole at RTI International (919-990-8456). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).

YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.

Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Signature of Participant		Date
Printed Name of Participant

I certify that the nature and purpose, the potential benefits, and possible risks associated with participating in this research have been explained to the above-named individual.

Signature of Person Obtaining Consent		Date
Printed Name of Person Obtaining Consent

1 http://wwwn.cdc.gov/QBANK/Report/Miller_2005.pdf

2 To facilitate the review process, new items (e.g. those that were not in the B&B:08/09 survey) have been highlighted and are listed in the table of contents.

14