Attachment V - B and B 2008-12 Field Test Items Cog Labs Report_May20_2011

This report focuses on the methodology, procedures, and general findings from cognitive testing conducted as part of the development for the field test instrument for the 2008/12 Baccalaureate and Beyond (B&B:08/12) survey instrument. The B&B:08/12 study is housed in NCES’s Postsecondary, Adult, and Career Education Division. B&B:08/12 tracks sample member’s experiences with continuing education, degree attainment, graduate education, employment, debt and finances, family formation, civic engagement and volunteerism, and interest in and preparation for K-12 teaching. There is a strong focus on entry into and persistence in the K-12 teacher workforce. Information on teacher’s professional experiences will be collected, which will allow a comparison of the teaching profession to other occupations as well as a general study of labor market outcomes for the cohort as a whole. Also of interest of importance for the B&B study are questions concerning whether American colleges and universities are preparing enough graduates in the fields of science, technology, engineering, and mathematics (STEM) and whether college graduates with training in these fields are using that training in the workplace or pursuing graduate education in STEM fields.

B&B:08/12 represents the third time that the B&B08 cohort will be contacted. The first contact was made in 2008 as part of NPSAS:08 base year. NPSAS:08 sample members that completed a baccalaureate degree that year were contacted again in 2009, one year after completing their degree, as part of B&B08 follow up. B&B:08/12 is the four year follow-up of the 2008 baccalaureate degree recipients.

Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) conducted cognitive interviews with recruited participants. The cognitive testing process was designed in order to enable the instrument team to:

examine the thought processes affecting the quality of answers provided to survey questions,
understand the extent to which terms in questions are comprehended,
evaluate the memory demands of the questions,
evaluate the ability of respondents to make calculations and judgments,
determine appropriate content and layout for of response categories,
assess the time it takes to complete the interview,
assess the navigational problems users face, and
identify sources of burden and respondent stress.

2 Methods and Procedures

2.1 Recruiting

Individuals in North Carolina’s Research Triangle Park and surrounding areas were recruited via flyers and advertisements that described the purpose of the cognitive interviews and details of participation, including the time commitment, incentive for participation, and contact information. RTI International staff recruited participants in the following ways:

Posting online ads on Craigslist and teacher websites
Placing advertisements in a local newspaper (The Independent Weekly)
Recruiting friends and family members via word of mouth
Placing advertisements on the RTI Intranet
Posting flyers in local high schools
Posting flyers in local universities, particularly in areas where STEM graduates might see them

Interested individuals were asked to call RTI to complete a brief screening interview to determine their eligibility. Individuals were eligible for the study if they met all of the following four criteria: (1) enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2006 and June 30, 2007; (2) completed the requirements for a bachelor’s degree between July 1, 2006 and June 30, 2007; (3) were awarded a bachelor’s degree by June 30, 2008; (4) were 18 years or older. Table 2-1 contains the number of individuals at different stages of the recruitment process.

Table 2-1. Participant Recruitment

Recruitment Stage	Number of Individuals
Contacted RTI but not screened*	27
Screened but not eligible	28
Eligible but not scheduled	5
Scheduled but not successfully interviewed	9
Completed interview	24
Total	93
Notes: Categories are mutually exclusive. *Individuals who did not return RTI’s follow-up contact attempts.

During the screening interview, individuals were also asked questions to determine whether they belonged to the three sub-groups of interest. The first sub-group was “enrolled” students, those who had enrolled in any school for an additional degree or certificate since earning their bachelor’s degree. The second sub-group of interest was “teachers.” Individuals who had worked as a K-12 teacher since earning their bachelor’s degree, done anything to prepare for K-12 teaching, or were currently considering a K-12 teaching career were classified as “teachers.” The final sub-group of interest was “STEM,” consisting of individuals who earned a bachelor’s degree in a STEM field. These sub-groups are not mutually exclusive: an individual could be classified as both a teacher and STEM, for example.

2.2 Interview Procedures

Survey methodologists from RTI International conducted one-on-one interviews with twenty-four participants. The interviews were divided up into three sets. Between each set of interviewing, the instrument was revised based on the results from the previous set of cognitive interviews. Table 2-2 contains the dates and number of interviews in each set.

Table 2-2. Dates and Number of Interviews, by Set

	Set 1	Set 2	Set 3
Start date	3/16/2011	3/28/2011	4/6/2011
End date	3/19/2011	4/1/2011	4/11/2011
Number of interviews	3	10	11

Out of the twenty-four interviews, nineteen were conducted in-person at RTI’s cognitive testing facilities in Research Triangle Park, NC. The other five interviews were conducted by telephone. Participants provided informed consent before beginning the study, and were given $40 after the completing the study. On average, interviews lasted about one and a half hours.

During the interviews, participants completed the survey and interviewers probed participants about the questions and participants’ responses. Interviewers asked a series of “scripted probes,” prepared in advance, that targeted specific questions (see appendix D) . Examples of scripted probes include “How did you come up with your answer to this question,” “How easy or difficult was it for you to find your major [from the list provided],” and “In your own words, what is this question asking?” These scripted probes were revised after each set of interviewing to reflect changes to the instrument and new priorities for the interviews. In addition, interviewers asked spontaneous probes about questions that participants appeared to have difficulty answering or when participants seemed hesitant or unsure.

Staff from RTI’s Educational Studies Division (ESD) and researchers from the U.S. Department of Education were given the opportunity to observe the interviews. These individuals could listen to the interview on a shared telephone line and watch the participants’ computer screen via remote screen-sharing software called GoToMeeting. The interviews were recorded using a software called Morae that captures both the audio and video of the participants’ computer screen.

Interviews were conducted in both web and telephone modes. For the web mode, participants used a laptop computer to enter their responses. For the telephone mode, five interviews were conducted over the telephone and six were conducted in-person. The latter simulated telephone interviewing by having the interviewer sit across from the participant, read the questions from the laptop, and enter the participants’ responses. The distribution of interviews by mode (telephone, web) and type of interview (enrolled, STEM, teacher, general) is shown in Table 2-3. Note that the categories for “type of interview” are not mutually exclusive. An individual might belong in the “enrolled” and “STEM” categories, for example. The “general” category consists of respondents who did not belong to any of the three sub-groups of interest.

Table 2-3. Number of Interviews, by Mode and Type of Interview

Interview Type	Telephone	Web	Total
General	2	3	5
Enrolled	2	3	5
STEM	0	3	3
Teacher	3	3	6
STEM/Enrolled	0	2	2
STEM/Teacher	1	0	1
Teacher/Enrolled	1	1	2
Total	9	15	24

3 Findings and Recommendations

This section presents the findings from the field test cognitive interviews and recommendations for revisions to the survey questions. Throughout the development process, the interview was revised based on findings from cognitive testing and from thorough review from the instrumentation team and survey methodologists. For example, one change that was not a direct result of cognitive testing and applied globally to a number of items was to revise a number of questions from “check all that apply” to forced-choice yes or no questions. Research has shown that forced-choice questions generate more positive responses than “check all that apply” questions and these types of questions are easier to administer over the telephone.

The sections below present both the results of the cognitive testing and the project team’s evaluation and review of the questions.

3.1 Eligibility

This section primarily collects information on the dates of enrollment and of degree completion at the school where the respondent earned his/her bachelor’s degree. It also asks about marital status and household composition. Wording changes were made to two items in this section.

B12AFINCON. The original wording of this question was “Is there an adult member of your household with whom you share financial burdens and concerns?” Most participants seemed to understand this question as it was intended, but the wording was revised after sets 1 and 3 to make it clearer. After set 3, the question was changed to “Is there another adult in your household with whom you are sharing financial responsibilities and decisions, such as income, bills, and budgeting?”
B12AFINWHO. If respondents identify in B12AFINCON that they share financial responsibilities and decisions with another adult in their household, in B12AFINWHO they are asked to specify who they share these responsibilities with. Originally, the question asked if that person is their partner or someone else. It was revised to ask, “Which best describes this person?” with the response options “Domestic partner or spouse, Boyfriend or girlfriend, Parent, Sibling, Friend, Other.”

3.2 Undergraduate Education

Overall, the questions worked well and participants were able to recall the information they were asked about. One participant commented during the undergraduate schools loop that she thought it should ask about her degrees in chronological order, rather than asking about her doctoral degree first and master’s degree second.

B12BOTHSCH. This item asks respondents if they attended any other colleges, universities, or trade schools after graduating from high school and before graduating with their bachelor’s degree. Most participants understood what it was asking, but some had to reread it to be certain of the time frame and requirements.
B12BTRNC01. This item asks respondents who attempted to transfer credits from one school to another if all, some, or none of them were accepted. Participants did not seem to have any problems recalling this information, and the response format is appropriate. When probed about the specific number of credits they attempted to transfer, some participants were unsure because they did not recall how many credits the classes were worth.
B12BINCHO and B12BAJCHO. These items ask respondents if they are satisfied with the quality of the undergraduate education they received (B12BINCHO) and their choice of undergraduate major (B12BMAJCHO). Of the participants asked, five thought the yes/no response options were adequate, but nine participants indicated that they would have preferred a range. As one participant put it, “yes” implies satisfaction with every aspect of the experience. Many participants identified that they would prefer response options covering a range of satisfaction. We recommend revising the question text to “How satisfied are you with… ,” and changing the response options to “very satisfied, generally satisfied, somewhat dissatisfied, very dissatisfied.”

3.3 Post-Baccalaureate Education/Training

In this section, participants were asked questions about any education and training they may have received after completing their bachelor’s degree. Overall, participants found it easy to understand the questions and provide a response in this section. There were a few minor problems with wording and question instructions. These problems and their solutions are described below.

B12CFINAID01. In this question, participants are asked to report the types of financial aid they received to support their additional education. Out of ten respondents probed, seven said it was easy to understand the response options and select their answer. One other respondent had to think about it, but then had no difficulty answering. Two other respondents had more difficulty. One of these respondents said that financial gifts or loans from family members did not count as “financial aid” in his mind. Another respondent initially did not select “employer,” even though she receives tuition remission from the university where she works—but then later selected “employer.” In the course of revisions, the response options were re-formatted to make it easier for respondents to find the relevant responses in the list.
B12CAIDCST01. This item asks participants to report the percent of their educational costs that were covered by financial aid. To provide their answer, participants moved a vertical slider to indicate a percentage. As participants moved the slider up (or down), the box “filled up” (or down) with a blue color, and a percentage for financial aid and a percentage for out-of-pocket was displayed. Initially, the label for financial aid was at the bottom and the label for out-of-pocket costs was at the top of the box. The placement of the out-of-pocket label caused some participants to be confused about which direction the slider should be moved in. To address this problem, the label for out-of-pocket was later moved underneath the label for financial aid and placed in parentheses. Participants in the final set of testing did not have any difficulty understanding which way to move the slider.

Additional changes based on cognitive testing included removing the boxes next to the labels (while intended to display the response chosen, participants thought they were supposed to enter a value in the box) and changing the word “proportion” to “percent.” Another issue was the definition of “educational costs.” Although there was consensus that “educational costs” included tuition, fees, books, and supplies, participants were inconsistent about whether living expenses should be included in “educational costs.” We recommend that the question explicitly state whether living expenses are considered “educational costs.”

B12CEFUTJB01. In this question, participants were asked to report the benefits they expect to receive after completing their additional education. One of the options is “Increased professional autonomy.” One participant had difficulty answering this question because he has not worked outside of school and does not have a current amount of “professional autonomy” to compare with his future job.
B12CEFUTN01. This question asks participants to report the amount they thought they would earn after finishing their degree. One participant said she never thought about this issue when she was enrolled and does not know how to answer the question.
B12CELNSTAT. Participants were asked about the status of their federal student loans in this question. One participant said she consolidated her loans, but could not decide between the first option, “Repaying in original payment amount” and the second option, “Repaying a different amount due to loan modification, consolidation or extension.” We recommend that “the original amount” be defined. For example, does this mean the original total loan amount, the monthly payment amount, or something else?
B12CENEMP01. This question asks participants about the number of hours they worked while enrolled in additional education or training. Initially, two participants were unsure about whether “working” included unpaid coursework for school. To address this confusion, “work” was changed to “work for pay.” This change reduced confusion for most of the subsequent participants. However, two other participants still reported ambiguities between their school work and work for their “career.” For example, one participant did not know whether to count his paid teaching assistantship and research assistantship as working for pay. We recommend that this question include an instruction to include all paid positions.
B12CESTCST. This item asks participants who said they are likely to enroll for an additional degree or certificate about the total costs and out-of-pocket costs for this additional education. Two participants expressed confusion about the definitions of “total costs” and “out of pocket” costs. A subsequent version of this question included more examples of “total costs” and “out of pocket” costs. Three participants reported being very unsure of their estimates. We recommend that a “Don’t Know” option be included.
B12CFENR01. This question asks participants when they first enrolled for their post-bachelor’s degree education. One participant enrolled in a Ph.D. program said she provided the date of entering her doctoral degree program. However, interviewer probing revealed that this date was when she enrolled in her master’s degree, not her doctoral program. To clarify the dates of entering doctoral versus master’s programs, another question was added (B12CENRTDG01) that asks about earning a master’s degree on the way to earning a doctoral degree.
B12CFIRST01. This question inquired about whether the participant attended his or her first choice of schools. Participants voiced some confusion about whether the question was asking about all the schools they applied to, or about the schools to which they were accepted. This question was revised to refer to the schools to which the participant applied.
B12CMAJ01. This question asks participants to report their major. Almost all participants said it was easy to find their major, although two participants did not find a satisfactory major in the results. Several participants who received one result from their search were not sure whether to press “select” or “next.” To address this problem, the word “Select” was bolded in the instruction “Please click the ‘Select’ button …”
B12CNDGCWK. This question asks about additional non-degree coursework. One participant initially said “no,” but when probed, changed her answer to “yes.” The participant said she missed the “non-degree” words because this question appears to be similar to other questions about degree programs. We recommend that “non-degree” be bolded.
B12COTH01. This question asks if the participants who have enrolled in post-bachelor’s education if they have enrolled, are enrolled, or will enroll at the same institution for an additional degree. One participant said this question was hard to understand. The text was later revised to improve the question wording.
B12CPBINFO01. This question asks participants to describe the sources of information they used to learn about the school. Initially, one response option read “Faculty/staff from [FILL: BACHELOR’S DEGREE INSTITUTION].” One participant wanted to select faculty/staff from her graduate institution. After set 2, this option was changed to the more general “Faculty/staff.”
B12CPRIVSTAT. This question inquires about the status of the participants’ private loans. Although two participants reported no difficulty answering this question, one participant said she would have liked to select more than one answer. As the response options are not mutually exclusive, we recommend splitting this question up into multiple questions that inquire about whether (1) any loans have been paid off, (2) any loans are in deferment, and (3) the status of loans that have not been paid off or deferred.
B12CSCH01. This question asks participants to enter the name of their post-bachelor’s educational institution. None of the participants had difficulty entering the name of the school. However, three individuals who received only one result from their search were unsure whether they had to select “next” or “select.” We recommend that this screen include a fifth step, which is to “Click ‘Select’ on your school.” Currently, the screen only includes four steps, but there are actually five steps.

One participant that had attended multiple graduate schools said he was unsure which school to report first. We also recommend that the instructions provide guidance about which school to report first.

3.4 Post-Baccalaureate Employment

This section focuses on employment experiences that participants had after receiving their bachelor’s degree. For the purpose of cognitive testing, most participants were asked follow-up questions about just one job—typically his or her most recent job. Participants had a bit of difficulty deciding whether part-time jobs, temporary jobs, or internships should be considered, as discussed in more detail below. For the most part, participants did not have much difficulty with these questions. However, there were a few questions that were slightly problematic during testing. These items are detailed below:

B12DANYJOBS. This question asked if participants worked for pay since earning their bachelor’s degree. Two participants answered no even when they were working for pay. They answered no because they had research jobs related to their graduate education, and considered these as part of their schooling rather than “employment.” To address this, the question was revised after set 2, instructing participants to include graduate school jobs such as teaching and research assistantships and fellowships.
B12DEMPLOY01. Participants who indicated that they worked for pay were asked to enter information about all the jobs they had into a job grid. The cognitive interviewers asked participants if there were any jobs that they were unsure whether they should report. Participants tended to not report jobs that were not career-related (e.g. working a record store, waitressing) and jobs that were career-related but were part-time or of limited duration (e.g. internships). Out of 11 participants who indicated that they had a part-time job, internship or similar, 8 of them said they would not have listed those jobs for this question. Since it is important to capture career-related jobs, even if part-time or temporary, the following text was added, “including part-time jobs, graduate school jobs, and paid internships”. One participant was self-employed and did not think that counted for this question. To address, this, a check box for “self-employed” was added to this question.
B12COMDAY01 and B12DCOMTIM01. In the first set of testing these two questions asked how often one traveled to work and how long it took to travel to work, respectively. Some participants were confused by this question and thought it meant travel “for work”. One participant noted that she walks to her job, so she would not consider that travel either. For the second set, “travel” was changed to “commute”, but this also proved problematic. For the third set, the question B12COMDAY01 about how often one “travels ” or “commutes ” to work was eliminated as it was not necessary. B12DCOMTIM01 was revised to simply read, “On an average day, how much time did it take you to get to work.”
B12DEDREL01. This question asks how relevant the skills obtained in the participant’s bachelor’s degree program and any additional education were to their current job. One participant for the “and any additional education” clause to be confusing. Another participant indicated that this question was double-barreled because her bachelor’s degree was very irrelevant while her graduate degree was very relevant. We recommend separating this question into two separate questions – one about the participant’s bachelor’s degree and one about graduate education, if applicable.
B12DIMPBEN. This item was a check all that apply question asking participants to indicate which benefits were important when considering a job. Many participants selected all or almost all of the benefits, making it hard to discern what were the most important benefits. This question was revised after set 3 to allow participants to indicate how important each benefit was to them on a scale of 1 to 5. This will allow researchers to better distinguish the perceived value of the different benefits.
B12DINDCD01. This question asks participants to enter their company’s primary business or industry in an open-ended text box and then to select the category that fits, and that an example of that category will be provided for the participant to review. In the initial sets, participants had a little trouble with this item because there was a lot of information on the page and the visual layout did not match the text. The question was revised into steps to make it clear what information should be provided where.
B12DLKWRK. This question asks participants to indicate which months they were looking for work since graduating from college — requiring recall over a period of four years. Participants indicated some difficulty remembering the months that they were looking for work. The survey screen was also slightly complicated by the check boxes for the various months. As a result, the question was revised so that the answer categories, the months, were grouped according to calendar year and season in a grid. This provided a neater, cleaner format and it is possible that the seasonal grouping of the months will facilitate recall.
B12DSPPSUP01. This question asks if the participant’s spouse was supportive of the particular job being asked about in this section. The logic is based on whether the participant is currently married, but the job being asked about could have been up to four years ago. At least two participants indicated that they were not married when they had this job. This question was revised to include a checkbox to indicate that the participant was not married when they had this job.
B12DTNGTYP. This question asks participants what types of formal employee training (e.g. safety and compliance, remedial, position-specific) they have had in the past 12 months. One of the job types was initially listed as “Basic Skills training,” which included training in elementary reading, writing, and arithmetic. However, four participants who were teachers selected this category because they thought it meant courses on how to TEACH elementary reading, writing and arithmetic. The category type was changed to “Remedial training, which includes training to improve your basic reading, writing, arithmetic and English language skills (including English as a second language.)”
B12DTNGINST. This question asks who taught their last job training. The response options included, an employee of your company, a private vendor, a professional association, etc. Two participants indicated that they did not know how to answer the question because they took an online training. The question was revised for set 3 to include the option “Virtual instruction (online, DVD, CD)”.
B12DTNGNUM. This question asks participants if they had one or more than one formal training in the past 12 months. Several participants were uncertain what was meant by “formal” training and if that included the trainings they selected on the previous screen. To address this, a definition of “formal training” was provided on a separate screen before the start of this section. Several participants indicated that it might be more helpful if the definition of a formal training was on the same screen as B12DTNGYP which asks about the specific types of formal training that a participant has had. However, that screen is quite wordy and we hope that including the definition on a separate page will call more attention to it.
B12DTNGREC. This question asks participants which type of training they had taken most recently. One participant indicated that he just had one training that covered several different topics, therefore he wanted to select all options instead of just one. We recommend adding a response option to indicate that multiple training types occurred at the same time.

3.5 Teaching

This section asked about participants’ interest and experience in teaching at the K-12 grade levels. Some participants who were teachers found this section to be somewhat repetitive to the job loop questions because they had to reenter information about their employee and when they started and stopped working. We recommend that instead of repeating the questions about the employee that participants can indicate if any of the jobs they listed in the job grid were teaching jobs.

Overall most of the questions in this section were well understood. Item-specific findings and recommendations are discussed below.

B12ECRTTYP. Most participants did not have difficulty with this question. However one participant could not answer the question because she indicated that she did not have a teaching “certificate” but had a teaching “license” which certified her to teach in the state of North Carolina. We recommend referring to “certificate” as “certificate/license” because states vary as to the terminology used.
B12EEVRTCH. This question asks participants if they have ever worked as a teacher at the K-12 level. Two participants answered yes to this question when they should not have. One participant answered yes because she led an ESL training session workshop that included individuals aged 16-22. Another participant said yes because she worked as a tutor with a private tutoring test prep company. We recommend adding the categories “other” or “tutor” to B12EJBST01, which asks what type of teacher they were, then not continuing the K-12 position loop with participants who select the “other” or “tutor” categories.
B12EPRP01. This question asks participants if they felt adequately prepared during their first teaching job. Three participants indicated that they would have preferred a range for this question rather than yes/no options However all participants were able to choose a yes or no response.
B12ETCHSAT. This question asks participants if they were satisfied with several different aspects of their career. Four participants indicated that they would have preferred a range for this question rather than yes/ no options.

3.6 Background

Several issues were identified with items in the background section. While most of the issues call for revisions, few call for substantial revisions. The most prominent issue is that the list of job descriptions for parents’ occupations is sometimes very long. Most of the other issues identified were problematic for only a small number of participants, but revisions are recommended.

B12FLNGCLS. A telephone interview participant thought that this was asking for a date, rather than the elapsed time. The question was revised to ask “How long has it been since you took a class to learn {LANG}?”
B12FLNGCOM. This item asks about proficiency in a foreign language compared to proficiency in English. The resulting response options are “More proficient than in English,” “Same as in English,” “Less proficient than in English,” and “Not proficient at all.”
B12FGUARDED. One participant interpreted “relationship to you” to refer to the quality of the relationship. We recommend replacing describe” with “provide” as follows, “For each of your parents or guardians, provide their relationship to you (for example, “mother”) and the highest level of education they have completed.”

B12FPAROCC1 and B12FPAROCC2. These items ask for the respondent’s parent’s job title and main duties. Participants whose parents are unemployed had difficulty determining how they should complete this question. Should they check the unemployed box or fill in their parent’s previous job information? If the parent is currently working, the question asks about their current job. However, the question asks about their “primary occupation when they were employed” if the parent is unemployed, retired, or deceased. This is inconsistent with what the question asks for parents currently employed.

After supplying the information, many participants received a lengthy list of job descriptions to choose from. Some participants appeared to be overwhelmed by the length of the list.

We recommend making the following changes to this question:

What is your [T_RELFILL]’s job title and main duties? If (s)he is not currently working, please respond with their most recent job.

FIRST, select all that apply: (checkboxes) Employed, Unemployed, Homemaker, Has never worked for pay, Deceased
SECOND, verify job title:
THIRD, type in job duties:
LAST, click ENTER to select occupation

B12FAMTRET. This item asks how much the respondent (and, if applicable, his/her spouse) have saved in retirement accounts. One participant gave an amount but said she was only “60% sure” it was accurate; another participant reported being fairly certain about the amount. Respondents who are less certain about the amount may be more inclined to skip this item. For respondents who leave this item blank, we recommend providing a follow-up question with response ranges (e.g. $0; $1-10,000; $10,000-20,000, etc.)
B12FMTGAMT. This item asks for the average amount paid for rent or mortgage. One participant put “$0” because her husband would know the amount but she didn’t. (If you do not know the exact amount, please provide your best guess.) was added after set 1.

After set 3, the text was revised to specify that respondents who both own and rent should report the amount paid for both. The text was changed to, “How much (on average) is your total monthly housing payment (including both rent and mortgage payments)? Please indicate only the amount that you are responsible for paying. If someone else pays your housing payment on your behalf, please indicate ’0’.”

B12FCARAMT. This item asks how much the respondent (and, if applicable, his/her spouse) pay each month for their vehicle lease(s)/loan(s). One participant pointed out that it is unclear whether it is asking about only the respondent, or also his/her spouse or partner.

The item was revised after set 1 to improve the flow of the question. The text in parentheses was moved to after the question. We recommend also specifying in the question text whether respondents should include their spouse’s/partner’s vehicle.

B12FINCOM. This item asks about the respondent’s income from all sources for calendar year 2010. One participant considered only her primary job until probed about her part-time job. We recommend adding “total” to the item text as follows, “…what was your total income from all sources…”
B12PSELLPO. This question asks, “Suppose you were to sell all your major possessions (including your home), turn all of your investments and other assets into cash, and pay off all your debts. Would you have something leftover, break even, or be in debt?” Some participants found this to be a strange question, but all were able to provide an answer. Participants seemed to consider mainly their large assets and debts, such as cars, student loans, and money in the bank.

3.7 Use of Sliders

In the first set of interviews, some of the questions used a 1 to 5 slider scale and other questions used a 0 to 10 slider scale. The cognitive interviewers asked participants if they had a preference for either type of scale. While some participants were indifferent, the majority of respondents preferred the smaller 1 to 5 scale frequently noting that 0 to 10 was too much to think about. For the subsequent sets, most questions were changed to the 1 to 5 scale where appropriate.

For the most part, participants liked the sliders and liked the ability to be able to answer using decimals. To minimize the potential for likely mode effects, we recommend explicitly stating in the instructions for telephone respondents that decimals are allowable in the response range. Most participants found the sliders easy to use. The one exception is that several participants did not move the slider when they wanted to select the middle option (i.e. a “3” on a 1 to 5 scale). Since the slider was already in the middle, they did not realize that it needed to be moved to make a selection. To reduce the amount of missing data caused by this, a soft-check was programmed to appear on the screen for any slider question that was left blank.

One participant thought that she could type in the number into the box that displayed the current selection for the slider. To prevent participants form trying to type in the display box, the box surrounding the number was removed so that it no longer looked like an editable text box.

3.8 Telephone Interviewing

The telephone interviews revealed instances in which the question wording and interviewer instructions need to be modified. We can provide several recommendations to accomplish this goal. First, the introductory text and transitions read by interviewers should be in first person singular (e.g., “I have some questions …”) rather than first person plural (e.g., “We have some questions” …). Second, the response options should be in second person singular (e.g., “You wanted”) instead of first person singular (e.g., “I wanted”).

Third, we recommend that most of the “check all that apply” questions, in which the response options are not read, be converted into a yes/no matrices in which the interviewer reads all the response options and selects “yes” or “no.” Methodological studies show that yes/no matrix questions produce more affirmative responses than do “check all” questions. There were several examples during the telephone interviews in which a participant did not mention a response option, but probing revealed that the participant would have selected that option if it had been read to the participant. This occurred in items such as B12CEFUTJB01 and B12CFINAID01. Leaving these questions with the “check all” structure may lead to a mode effect, whereby “check all” items are underestimated in the telephone relative to web model.

Finally, the interviewers observed several specific instances in which the question was difficult to read in the telephone mode. These instances include the following:

B12CALTCSR01: One participant took only one class, which was online. It is awkward to ask whether the online class was taught "at night" or "on the weekend." To address this issue, we recommend placing "online" after "at night" or "on the weekend."
B12CELNSTAT: One participant waited to hear response options for this question because the question is broad. We recommend reading the response options in this question.
B12CESATS201: The introductory text does not need to be read again. This text was read in the previous question (B12CESATS101.)

Appendix A: Incentive Receipt

Receipt for Participation

The participant has voluntarily taken part in a study to improve the materials used in a national survey of college graduates. This study is being conducted by RTI International for the National Center for Education Statistics. By my signature below, I confirm that I have given the participant $40.00 in cash in appreciation for his/her participation in this project.

RTI Staff Signature

Staff Signature signifies that respondent has received incentive payment.

Interview Date

Interview Time

Appendix B: Consent Form for In-Person Interviews

Consent to Participate in Research

Title of Research: 2008-12 Baccalaureate and Beyond Longitudinal Study

Introduction

You are being asked to participate in a cognitive interview which is part of a larger research study sponsored by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The study is designed to examine the decisions and experiences of individuals four years after completing a baccalaureate degree. Before you decide if you want to take part in this study, you need to read this Informed Consent form so that you understand what the interview is about and what you will be asked to do. This form also tells you who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and who you can call if you have questions. Please ask the researcher to explain anything you don’t understand before you make your decision.

Purpose

The interview is being conducted by RTI International, a research organization located in Research Triangle Park, North Carolina.

Participation will involve answering questions and reviewing a survey for an upcoming study that will be used to learn about individuals’ experiences after college graduation. During the interview, we will ask you to complete a web or telephone survey and give us your thoughts on the questions you are being asked. In addition, we will ask you questions as you go in order to measure your understanding and interpretation of the questions in the survey. The information from the interviews will help us finalize development of the questions for the larger study that will be conducted early next year.

Procedures

You are one of approximately 45 individuals participating in these cognitive interviews.

The session will be recorded and may be observed by other RTI and or NCES staff members. The purpose of recording and observation is to provide NCES with feedback on the interview process so it can be improved, if necessary. The recordings will also be used to verify that the information from the interview is entered into the computer correctly. All recordings will be kept confidential and will be destroyed at the completion of the study. Your name will never be used in the report that we write. If you do not wish to be recorded or observed, you still may participate in this research.

You will be paid $40.00 for your participation.

Study Duration

Your participation in the cognitive interview will take about 90 minutes.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits

We hope that these interviews will help us develop and improve questions for a national survey that will help us understand individual’s experiences after completing their baccalaureate degree.

Confidentiality

RTI International is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

Future Contacts

We will not contact you in the future.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.

Your Questions

If you have any questions about the study, you may call Melissa Cominole at RTI International (919-990-8456). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).

YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.

Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Audio-taped: Check one.

I Shape1 agree to be audio-taped during the interview.

I Shape2 do not agree to be audio-taped during the interview.

Observed: Check one.

I Shape3 agree to be observed during the interview.

I Shape4 do not agree to be observed during the interview.


Signature of Participant		Date

Printed Name of Participant

I certify that the nature and purpose, the potential benefits, and possible risks associated with participating in this research have been explained to the above-named individual.


Signature of Person Obtaining Consent		Date

Printed Name of Person Obtaining Consent

Appendix C: Consent Form for Telephone Interviews

Consent to Participate in Research – Telephone Script

Title of Research: 2008-12 Baccalaureate and Beyond Longitudinal Study

Introduction

Hello. My Name is _________________ and I am calling from RTI International on behalf of the U.S. Department of Education. You are being asked to participate in a cognitive interview which is part of a larger research study sponsored by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The study is designed to examine the decisions and experiences of individuals four years after completing a baccalaureate degree. Before you decide if you want to take part in this study, I need to read to you this Informed consent material so that you understand what the interview is about and what you will be asked to do. This will explain who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and who you can call if you have questions. Please ask us to explain anything you don’t understand before you make your decision.

Purpose

The interview is being conducted by RTI International, a research organization located in Research Triangle Park, North Carolina.

Participation will involve answering questions for an upcoming study that will be used to learn about individuals’ experiences after college graduation. During the interview, we will ask you to give us your thoughts on the questions you are being asked. In addition, we will ask you questions as we move through the survey in order to measure your understanding and interpretation of the questions in the survey. The information from the interviews will help us finalize development of the questions for the larger study that will be conducted early next year.

Procedures

You are one of approximately 45 individuals participating in these cognitive interviews.

You will be paid $40.00 for your participation.

Study Duration

Your participation in the cognitive interview will take about 90 minutes.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits

We hope that these interviews will help us develop and improve questions for a national survey that will help us understand individual’s experiences after completing their baccalaureate degree.

Confidentiality

Future Contacts

We will not contact you in the future.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.

Your Questions

Audio-taped: Check one.

I Shape5 agree to be audio-taped during the interview.

I Shape6 do not agree to be audio-taped during the interview.

Observed: Check one.

I Shape7 agree to be observed during the interview.

I Shape8 do not agree to be observed during the interview.

May we begin the survey?

Appendix D: Cognitive Testing Protocol

Baccalaureate and Beyond

Cognitive Interview Protocol

Participant ID _____________

Interview Date _____________

Start Time _____________

BEFORE INTERVIEW

Send email to observers and programmers 1 day before and 1.5 hours before (Tamara)
Bring laptop, mouse, external microphone, $40 incentive; plug into network
Print informed consent forms
Print incentive receipt form
Set up GoTo Meeting (turn off beeps, turn off entry/exit, ensure “mic and speakers” selected, don’t show screen yet, mute self)
Close out of other applications (e.g., Skype, Outlook)
Initiate meet-me line
Set up (but do not start) recorder
Set up new case, including the following globals:
- Y_BB09RESP =3
- Y_NPSCHL = School
- T_COGSTEM = (default is 0) 0 if non-STEM, 1 if STEM
- Y_PARDEC=0
- Y_BB09AWRDM – month of graduation (MM)
- Y_BB09AWRDY – year of graduation (YYYY)
Website: https://edsurveysstage.rti.org/bb12ft_Student/testlaunch.aspx
Once observers join GoTo Meeting: share screen, mute all, hide control panel

SECTION 1: INTRODUCTION

READ OR PARAPHRASE:

We are currently planning a study to learn about people’s experiences after graduation. As part of that planning, we are testing a survey that asks about your undergraduate education, as well as educational and employment experiences after graduation. Today, I’m hoping to get your honest feedback on the survey questions.

[IF CATI: During the interview, I will read you survey questions and ask you to respond aloud.] [IF WEB: During the interview, you will read survey questions on the laptop and enter your responses in the laptop.] In this survey, I am less interested in your actual answers to the questions than I am in what you think about the questions and how you interpret them. As you go through the survey, I will also ask you some questions about your answers and the interpretation of the questions. Sometimes the questions I ask you may sound a little strange. For example, I might ask you what a certain word means to you. The reason for this is to see if people interpret things differently.

Even though some of the questions I may ask you might sound like I am giving you a test, there are no right or wrong answers. My main goal is to make sure that the questions are understandable to everyone who takes the survey. Please feel free to tell me anything that comes to mind or ask me anything you are unclear about. You can also help by telling me anything you find hard to remember or difficult to understand.

Your participation in this interview is very important because it will help us improve the questionnaire. Once we start, the interview will last about an hour and a half. When we are done, I will give you $40 cash as compensation for your time.

Do you have any questions before we begin? WRITE DOWN QUESTIONS:

SECTION 2: INFORMED CONSENT

Before we start, I’m going to give you a written consent form. This document explains the purpose of the study, who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and whom you can call if you have questions.

GIVE THE INFORMED CONSENT DOCUMENT TO THE PARTICIPANT. TELL PARTICIPANT ABOUT OBSERVERS ON PHONE AND BEHIND MIRROR IN HILL 117 (IF APPLICABLE).

ASK: Do you want me to read the consent document to you? Or do you want to read it yourself?

READ THE DOCUMENT TO PARTICIPANT OR LET PARTICIPANT READ IT

WHEN DONE:

ASK PARTICIPANT IF HE OR SHE HAS ANY QUESTIONS.
VERIFY PARTICIPANT AGREES TO BE AUDIO RECORDED AND OBSERVED.
IF PARTICIPANT AGREES, ASK HIM OR HER TO SIGN THE DOCUMENT. GIVE PARTICIPANT INFORMED CONSENT TEXT (NOT SIGNATURE PAGE).

SECTION 3: COGNITIVE INTERVIEWING

IF CONSENT WAS GIVEN, START TAPE RECORDER. COLLECT VERBAL CONSENT FOR AUDIO RECORDING.

CONDUCT INTERVIEW. START ON VARIABLE B12AMARR.

OVERALL

SLIDER QUESTIONS (as time allows and as respondents provide useful information). ASK ABOUT AT LEAST ONE 5 POINT SCALE AND AT LEAST ONE 10 POINT SCALE

[NO ENDPOINT] Why did you select [ANSWER] rather than [ANSWER-1] or [ANSWER+1]?

Why did you chose a decimal value? (If applicable).

You selected 1, which means [very little], is that accurate for you or would you have preferred to answer “not at all”?

PERSONAL/FAMILY REASONS (ask at least once if applicable):

What personal or family reasons are you thinking about? (Tell me more about that).

CHECK ALL THAT APPLY QUESTIONS (especially the ones that might work well as forced choice)

Can you tell me how you came up with your answers to this question? (What did you

think of the answer choices?)

Are there any options that are missing that you think we should include?

UG EDUCATION

B12BOTHSCH. Other than [school], did you attend any other colleges, universities or trade schools as an undergraduate between the time you graduated from high school and the time you graduated with your bachelor’s degree from BA school in?

Are there any schools that you were unsure whether you should include here or not? Why?

B12BSCH01. What is the name of that school?

How easy or difficult was it for you to enter the information and select the school name?

B12BTRNC01. Were all, some, or none of those credits accepted by [SCHOOL]?

[If all or some]: Do you recall how many credits were accepted? (How many?)

[If some or none]: Do you recall how many were not accepted? (How many?) Do you recall why your credits were not accepted?

How easy or difficult is it to remember how many credits were accepted and not accepted? How did you come up with your answer? (PROBE TO INVESTIGATE WHY IT IS EASY OR DIFFICULT TO REMEMBER THE ANSWER TO THIS QUESTION).

B12BINCHO/B12MAJCHO. Are you satisfied with the quality of the undergraduate education you received at BA school?/ Are you satisfied with your choice of undergraduate major(s) or course of study?

The last two questions had yes/no response options. Were those options okay or would you have preferred to answer differently such as mostly satisfied or somewhat satisfied?

POST-BA TRAINING

B12CPSTGRD. Have you enrolled in any school for an additional degree or certificate…

Were there any programs that you were unsure whether they should count for this question? Why?

B12CMAJ01. What is your primary major or field of study for your [DEGREE] at [NAME]?

How easy or difficult was it for you to find your major? (Were the instructions okay or could they be improved?)

B12CFINAID01. What types of financial aid did you receive to help pay for your [DEGREE] at [NAME]?

How easy or difficult was it to answer this question? Were there any types of aid that were confusing or that you were unsure about?(Were you thinking about financial aid for yourself or your parents? )

B12CAIDCST01. About what percent of your educational costs for your [DEGREE] at [SCHOOL] were covered by the financial aid or assistance (e.g. loans, grants, scholarships) you received, as opposed to what you paid out of your own pocket?

What do you think of the slider for this question? How easy or difficult was it to decide which way to push the slider?

What does “educational costs” mean to you? What were you thinking of?

What does “paid out of your own pocket” mean to you in this question? What did you include?

B12CHRDSHP01. To what extent do the costs for your [DEGREE] pose a hardship to you?

What does “hardship” mean to you, as it’s used in this question?

B12CPRIVSTA/B12CELNSTAT What is the status of your private/federal student loans?

Tell me why you selected {} instead of some other answer on the list? How well does that option fit your situation?

B12CEOUTLN. On a scale from 1 to 5, with 1 being “very low” and 5 being “very high”, please indicate your level of stress regarding your education-related debt.

[NO ENDPOINT] Why did you select [ANSWER] rather than [ANSWER-1] or [ANSWER+1]?

Why did you chose a decimal value? (If applicable).

You selected 1, which means [very little], is that accurate for you or would you have preferred to answer “none”?

POST-BA EMPLOYMENT

USE YOUR DISCRETION (CONSIDERING THE NUMBER OF JOBS AND DURATION OF INTERVIEW) TO DETERMINE WHETHER TO ASK ABOUT ALL JOBS.

B12DEMPLOY01. Please think about any job for pay you have had since earning your bachelor’s degree (including part-time and graduate school jobs). If you had more than one job for the same employer, please provide them separately. For each job, please provide the following …

Rather than entering the jobs on this page, please tell me all of the different jobs you have had since earning your bachelor’s degree:

Are there any jobs that you were unsure whether you should mention or include, such as internships, part-time jobs, or temporary jobs? Would you include those in this question? Why/Why not?

This question says “if you had more than one job or position for the same employer, please provide them separately.” What does “more than one job or position” mean to you?

ASK R TO ENTER MOST RECENT (NON-RTI) JOB IN THE GRID.

NOTE WHETHER R USEs THE “AUTOMATICALLY FILL CITY/STATE” BUTTON.

Starting salary ….How easy or difficult was it to remember you starting salary?

Did you include any bonuses, tips or commissions that you received or just your base salary?

B12DINDCD01. Please type in your employer's primary business or industry in the first box. Then, from the list below, select the category which best describes that business or industry. Examples of each industry will be displayed in the second box as you make a selection.

What is the question asking you to do? How easy or difficult was it to select a business and industry code?

B12DBEN01. For this job, please indicate which of the following benefits your employer offered you?

Were you thinking of benefits that you actually used or all benefits available to you?

B12DEDREL01. On a scale of 1 to 5 with 1 being “very irrelevant” and 5 being “very relevant,” how relevant were the skills you obtained in your bachelor’s degree program AND in any additional education to your job as economist?

What does the word “skills” mean to you, as it’s used in this question?

This question uses the scale “very irrelevant to very relevant,” is this scale okay or would you have preferred to answer differently?

B12DPARSUP01. How supportive were your parents of this job?

[NO ENDPOINT] Why did you select [ANSWER] rather than [ANSWER-1] or [ANSWER+1]?

This question uses a 10 point scale. Most of the previous questions used a 5 point scale. Do you have a preference?

B12DLKWRK. Since your graduation in [MONTH/YEAR] from [NAME], please indicate which months you were looking for work …

How easy or difficult was it for you to enter your answers on the screen?

How accurate do you think your responses are?

B12DIMPBEN. Which of the following benefits are most important to you when choosing a job?

This question asks about benefits that are MOST important to you. Are any of these benefits NOT important to you at all?

How did you decide which benefits were MOST important?

B12DTNGTYP. In the last 12 months, have you received any of the following types of formal training from an employer? (Mark all that apply. Include training from past as well as current employers.)

Would it help if the definition of formal training was on this page instead of the previous page?

Were there any trainings you had that you were unsure whether you should count or were unsure in which category to count them? Why?

B12DTNGINST. Was your most recent training taught or provided by…

How easy or difficult was it for you to find your answer from that list?

TEACHING.

B12EJBSL01. What is the name of the school?

How easy or difficult was it for you to enter and select your school?

B12EMOVE. On a scale of 0 to 10, where 0 is no chance…10 is absolutely certain, how likely do you think it is that you will move into or continue in a non-teaching job in elementary or secondary education (e.g. administrator or school counselor)?

In your own words, what is this question asking?

B12ETCHSAT. In your most recent teaching position, were you satisfied with each of the following...

This question uses yes/no responses. Is that okay or would have preferred to answer differently such as somewhat satisfied or very dissatisfied?

DEMOGRAPHICS.

B12FSELLPO. Suppose you and others in your household were to sell all your major possessions (including your home), turn all of your investments and other assets into cash, and pay off all your debts. Would you have something leftover, break even, or be in debt?

What were your first impressions when you read this question? How did you come up with your answer to this question? [DID THEY JUST THINK OF THE BIG THINGS – MORTGAGE, LOANS, ETC OR DID THEY THINK ABOUT EVERY SINGLE THING THEY OWNED]

END ON LAST DISABILITY QUESTION (B12FDISERND). DO NOT ASK CONTACT INFORMATION QUESTIONS.

SECTION 4: RETROSPECTIVE PROBES AND CONCLUSION

Do you have any overall opinions about the questions or the way the survey worked/functioned?

After hearing all of my questions, do you have any questions for me?

THANK PARTICIPANT.
TURN OFF THE TAPE RECORDER.
GIVE CASH INCENTIVE TO THE PARTICIPANT.
ASK PARTICIPANT TO SIGN RECEIPT.
WRITE END TIME.

End Time _____________

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Month 200X
Author	Charles Q. Strohm
File Modified	0000-00-00
File Created	2021-01-31