Results of Cognitive Testing

Appendix 4

Graduate Employment and Earnings Survey

Cognitive Interviews, Round 1 - Results and Recommendations

1. Recruitment and Participation

Cognitive testing was conducted from January 26 to February 11, 2015, in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago. To recruit graduates, RTI posted web advertisements on Craigslist and used direct outreach though advertisement postings at area businesses. A total of 42 individuals responded to the posted advertisements; of these, 19 were ineligible, 15 were eligible and participated, and 8 were eligible but did not participate due to cancellation, difficulty scheduling a time to participate, or not coming to their scheduled appointment. Participants were recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. Of those who participated, nine were female and six male. Table 1 provides a distribution of the final number of interview participants, by control and level of institution.

Table 1: Number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	3
	2-year	Vocational certificates and associates	2
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	1
Public	Less-than-2-year and 2-year	Vocational certificates and associates	7

Participants were given the short paper survey and asked to complete it on their own. Upon completion, they were asked to think aloud retrospectively in order to identify potential problems related to question comprehension, recall, information editing due to sensitivity or social desirability and response mapping. On average, the interviews took 32 minutes to complete.

2. Results and Recommendations

Introduction

The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

The information on this survey will be used to report average earnings for program graduates to the U.S. Department of Education. Your individual earnings will never be disclosed or used in any other manner unless required by law.

While most interviewees read the introduction, three appeared only to skim the introduction, and two indicated that they had not read it at all. Of those who read the introduction, respondents generally agreed that the introduction was an informative and adequately succinct summarization, clearly communicating the purpose of the survey. Two participants thought the survey applied to people who received federal financial aid and a few commented that the gainful employment legislation part of the introduction was confusing.

Recommendation:

Remove reference to gainful employment legislation. Revise first sentence of the introduction to read:

The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

Survey Questions

1. Please verify that you are _______________________________________. Is this correct? If you have legally changed your name, but once went by ______________________, please answer “Yes” below.

Yes

No  Please stop filling out the survey. It is important that you still return the survey in the envelope provided so that we will not contact you again.

Most interviewees noted that if there were a minor spelling error in their name, they would make the correction and continue filling out the survey. Several respondents noted that if there was an error in their name, they might return the survey if provided a postage-paid return envelope. However, six interviewees mentioned that, were the listed name completely incorrect, they would throw away the survey rather than return it.

Recommendation:

Shorten the instructions associated with the “No” response option to increase the chances of people reading them. Revise to read:

Thank you. Please return the survey in the attached envelope, so that we can remove you from our list.

2. According to our information, you completed the _______________________ program at _____________________________ between _______________ and ___________________. Is that correct?

Nearly all of the interviewees mentioned that they would have no difficulty recalling the year they began and ended their program; however, some reported that recalling the months might be more difficult. Several interviewees noted their ease in recalling the date was due to the recentness or significance of program completion.

Recommendation:

None.

3. Did you work at a job or business at any time during 2014?

Seven interviewees interpreted this question as asking about a job or business related to their degree program, despite the question not referencing this in particular. Their confusion may be due to the question’s proximity to the prior question which references the student’s degree program completion. The remainder of respondents generally saw no difference between “job” or “business,” and interpreted the question as “Did you have anywhere you earned income in 2014?” These respondents generally mentioned interpreting this question as excluding any unpaid work (e.g. unpaid internships or volunteer work).

Recommendation:

Revise to read: During 2014, did you work for pay at a job or business? (Source: modified Q29 of the American Community Survey—PAPI version).

4. Did you receive any wages or salary in 2014?

Respondents had no difficulty with “calendar year” and all interpreted as meaning January 1, 2014 – December 31, 2014. However, interpretations of the terms “salary” and “wages” were inconsistent. Seven interviewees interpreted “salary” as referring to a fixed yearly rate and wages as hourly pay, seven noted that they saw no difference between the two terms, and one interviewee interpreted salary as hourly pay and wages as a fixed yearly rate. In addition, some participants with more than one job (for example, a salaried job and part-time jobs, such as baby sitting, tutoring, or participating in studies) did not consider their part time jobs when answering this question.

Recommendation:

Start the series of questions on income with a common lead-in: “The next set of questions are about the different types of income you might have received in 2014.”

Additionally, include instructions to the question: “Please include both full-time jobs and part-time jobs.” Alternatively, revise the question to read: “In 2014, did you receive any wages or salary from full-time or part-time jobs?”

4a. How much did you receive in wages and salary from ALL jobs before taxes and other deductions in 2014? Be sure to include wages and salary from ALL jobs in 2014

Nearly all interviewees expressed having difficulty with this question. Interviewees were generally confused as to what to include and what to exclude in this amount, especially after encountering the remaining questions (5 – 6). Three respondents revised their initial answers to this question once they attempted to answer the following questions (5 – 6), mentioning that they thought first 4a included everything, but were then unsure. Several interviewees noted that they excluded part-time jobs or irregular pay (e.g. payment for babysitting, participation in surveys, gambling winnings, tips, bonuses).

Most interviewees mentioned that the amount they provided was a rough estimate; however, two said they had simply reported what was on their W-2, Wage and Tax Statement, which they had recently received. Two interviewees arrived at an amount by first calculating approximate monthly earnings before taxes and then multiplying by 12 months.

A majority of respondents noted that the amount which they provided excluded taxes and deductions, and had no trouble interpreting the term “deductions.” Two interviewees expressed confusion over how to enter the numeric amount given the “.00” at the end of the blank field.

All but one of the interviewees noted that they had no reservations providing income information, and did not feel the questions infringed on their privacy.

Recommendation:

Restructuring the questionnaire to include all income questions as subparts of one question, using a common introduction (see recommendation to question 4) should eliminate double counting as it will become obvious to respondents that questions 4a-6a are subcategories that should sum up to their total annual income.

To further ensure tips and bonuses are not double counted, follow the approach of the American Community Survey and combine questions 4a and 5a, asking respondents to report on “wages, salary, commissions, bonuses, or tips from all jobs.” Include instructions that all jobs should be considered whether or not they are related to the program of study.

In addition, use a box, rather than a blank field, where the requested dollar amount can be entered.

5. Did you receive any additional tips, overtime pay, bonuses, or commissions?

Three respondents indicated “No” on this question because they had already reported overtime and commission pay in the previous question (4a). Two interviewees treated this question as a catch-all category and interpreted it as asking about money made in addition to salary and wages, including “under the table” pay or for money made in small temporary jobs. One interviewee indicated “No” because she did not report her tips for tax purposes, and one mentioned excluding tips and commissions due to their job not being related to their degree program.

Recommendation:

Include tips, bonuses, overtime pay, and commissions as part of the overall income from wages and salaries in questions 4a.

5a. How much did you receive in tips, overtime pay, bonuses, or commissions from ALL jobs before taxes and other deductions in 2014? Be sure to include ALL tips, overtime pay, bonuses, or commissions in 2014.

Similarly to the previous question, the interpretation of this question and its relationship to question 4a was puzzling to most respondents. Five of the eight eligible to answer this question noted that they had already included tips, overtime pay, bonuses and commissions in their response to question 4a. Several interviewees mentioned that this figure would be hard to come up with; some of the recall strategies mentioned were estimation based on a minimum yearly figure, and estimation based on a monthly figure.

A self-employed person was confused whether to include his commission under 4a, or 6a, but never considered it in response to this question, that explicitly states commissions.

Recommendation:

Include instructions in question 4a to include this figure in the overall income from wages and salaries.

6. Did you receive any self-employment income? Report income from own business (farm or non-farm) including proprietorships and partnerships.

Interviewees generally had no issues determining how to answer this question. One interviewee who holds a part-time job in a salon, was unsure how to answer this question, as she rents space in the salon and pays for her own supplies. Several interviewees who were not self-employed, expressed difficulty determining the difference between proprietorships and partnerships when probed on the meaning by interviewers.

Recommendation:

None.

6a. What was the amount you received in self-employment income before taxes and other deductions in 2014? Report net income after operating expenses. If net income was a loss, mark the “Loss” box to the right of the dollar amount.

Four interviewees expressed confusion over what to enter in this field because they had already reported their self-employment income in question 4a.

While most of the interviewees had no issue identifying what net income after operating expenses was conceptually, two noted that they had reported the amount before operating expenses.

Three interviewees noted confusion over what “loss” would mean, with one believing that expenses for equipment would be considered a loss, and one thinking that “loss” pertained only to those dependent on the stock market for their income.

Recommendation:

Restructure the questionnaire to include all income questions as subparts of one question, using a common introduction (see recommendation for question 4). Graphically (using a different font or underlining) emphasize that the requested amount should be reported after business expenses. Remove the instructions for “loss” from the question stem, as they seem to confuse rather than help respondents.

Overall Impressions

Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that they had no privacy concerns about any of the questions and would leave none blank, but several emphasized that they would participate only if the survey looked official and was clearly from their prior institution or the U.S. Department of Education. Four interviewees emphasized their confusion about where to report what income, and what should be included and excluded in each question.

Graduate Employment and Earnings Survey

Cognitive Interviews, Round 2 - Results and Recommendations

1. Recruitment and Participation

Cognitive testing was conducted from March 17 to April 23, 2015, in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago, IL. To recruit graduates, RTI posted web advertisements on Craigslist, Backpage.com, and Ebay Classifieds, and used direct outreach though advertisement postings at area businesses. A total of 41 individuals responded to the posted advertisements; of these, seven were ineligible, 15 were eligible and participated, and eight were eligible but did not participate due to cancellation, difficulty scheduling a time to participate, or not coming to their scheduled appointment. Participants were recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. Of those who participated, 11 were female and four male. Table 2 provides a distribution of the final number of interview participants, by control and level of institution.

Table 2: Number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	5
	2-year	Vocational certificates and associates	3
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	1
Public	Less-than-2-year and 2-year	Vocational certificates and associates	4

Participants were given the short paper survey and asked to complete it on their own. Upon completion, they were asked to think aloud retrospectively in order to identify potential problems related to question comprehension, recall, information editing due to sensitivity or social desirability and response mapping. On average, the interviews took 22 minutes to complete.

2. Results and Recommendations

Introduction

The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment.

The information you provide on this survey will be grouped with information from other graduates of your program. Average earnings for all graduates of the program will be calculated and reported to the U.S. Department of Education. Your individual earnings will not ever be reported, disclosed or used in any other manner unless required by law.

About half of the cognitive interview participants actually read the introductory statement. Participants seemed to have a good understanding about the purpose of the survey. Only one participant commented that she did not quite understand the introduction as it contained a lot of information; the rest of the respondents did not seem to have problems with it.

Recommendation:

No changes.

Survey Questions

1. Please verify that you are <First-Name Last-Name>. Is this correct? If you have legally changed your name, but once went by <First-Name Last-Name>, please answer Yes below.

☐ No Thank you. Please return the survey in the enclosed envelope so that we can remove you from our list and you are not contacted again.

☐ Yes

Most interviewees reported that if the name was completely wrong, they would likely return the survey rather than proceeding to fill it out. Four said they would likely discard the interview, as it would be easier than mailing back. Two noted that if there was a telephone number to contact, they would call if the name was incorrect. Two interviewees expressed some difficulty with the legally changed name portion of this question.

Recommendation:

No changes to the question.

We would encourage institutions to designate a telephone number when launching the survey and include that number in all communications with sample members.

2. Did you receive any wages, salary, tips, overtime pay, bonuses or commissions from an employer in 2013?

☐ No Skip to question 3.

☐ Yes

Fourteen interviewees confirmed that they were recalling information from the 2013 year, and that recall was not difficult. Several respondents noted that they used key events (e.g., being pregnant) in 2013 to help aid in recall; while others mentioned that they recently reviewed their 2014 taxes while preparing their 2015 taxes. One respondent thought that the question was asking about the current year.

When answering this question, respondents reported including income from full-time jobs, self-employment, casual employment (e.g. babysitting, nanny work, geriatric care), part-time employment, and tips in their consideration of this question. Two respondents excluded student financial aid. One respondent was unsure of whether to include child support income, work-study income, or grant money. One respondent excluded overtime pay. One respondent expressed hesitance to include “off the books” income.

Recommendation:

No changes.

2a. Think about the job you held the longest during 2013. In total, how much did you earn from that employer, before taxes and other deductions, in 2013?

$_____________

2b. Now think about all OTHER jobs you held during 2013. In total, how much did you receive from all other employers, before taxes and other deductions, in 2013? If you had no additional jobs, please enter a zero.

$_____________

At least half of the participants had only one job in 2013. For those with more than one, the question about the job held the longest did not seem to present a problem. One respondent interpreted the question to mean her “primary” job (i.e, the job where she gained the larger portion of her income).

One respondent commented that initially, he thought question 2b should include any miscellaneous income, but later saw question 4, so concluded that question 2b referred to regular pay only. Another respondent, who had only one job, entered his income amount here and left question 2a blank. Jobs reported under question 2b included part-time jobs such as baby-sitting, CNA for a private person, or other “part time odd jobs” that were not reported in respondents’ taxes. Three respondents incorrectly reported self-employment.

Recommendation:

Combine questions 2a and 2b into one question, asking about wages, salary, tips, overtime pay, bonuses or commissions from all employers in 2013. Since several participants commented that they reviewed their previous year’s taxes before doing their current taxes, reporting income from all jobs might be an easier cognitive task. The initial intention of the longest job question was to help recall, but respondents seemed to use personal landmark events instead. Furthermore, this question will not be relevant for respondents with only one job and might unnecessarily confuse them on where to report their income. The example of types of income reported in question 2b suggests that there might be an overlap with what is reported in question 4 (see below for some evidence). In order to avoid double counting, we recommend the following overarching structure:

1. Earnings from all employers

2. Earnings from self-employment

3. Earnings from other jobs, such as freelancing, consulting, etc.

To follow this structure, we recommend combining questions 2a and 2b into the following one question:

In total, how much did you earn from all employers, before taxes and other deductions, in 2013?

3. Excluding any income mentioned above, did you receive any self-employment income, including income from your own business?

☐ No Skip to question 4.

☐ Yes

3a. What was the amount you received in net self-employment income in 2013? Please report income minus expenses. If this amount is negative, please enter zero.

$_____________

All but one participant correctly defined self-employment as “working for yourself” and “running/owning your business.” Two respondents suggested that “1099” be included in the question as an example of self-employment. Four participants seemed to treat this question as a catch-all category, and noted that they would include tips, any other on-the-side income, or income earned outside of the traditional 40 hour work week.

One respondent was confused by what “net self-employment” income meant and she guessed it was income before taxes. One respondent missed the routing instructions in 3, therefore the question did not make sense. The respondent suggested that the skip instruction and arrow should be more prominent.

Six out of the eight respondents who answered question 3a did not have problems with the “income minus expenses” concept and were able to provide their own examples.

Recommendation:

No changes.

4. People sometimes earn extra money doing work outside of their regular jobs, such as freelancing, consulting, moonlighting, or doing other casual jobs. In 2013, did you earn any other money from work you did that you have not already reported on this survey?

☐ No Thank you. Please return the survey in the envelope provided.

☐ Yes

4a. What was the amount of your additional earnings from work in 2013 that are not already reported above?

$_____________

Three respondents found this question overlapping with the self-employment question (question 3), as they considered jobs such as doing hair and baby-sitting. Most participants interpreted the question to mean “side gigs,” participating in clinical trials, lawn mowing, or other ways of making extra money and being paid in cash. Two respondents double-counted by considering income already reported in 2b. One respondent who was previously confused about whether she should include grant money, internship funding, and child support noted that she would likely have included that income here in 4. This respondent, however, was still not sure if they should include Social Security Income, veterans’ benefits, or Social Security benefits. Two participants noted they were unsure of the meaning of “moonlighting” and “freelancing,” and one respondent expressed that they were confused about the term “casual jobs.” One individual commented that this question was an invasion of his privacy.

Recommendation:

No changes to the question. The issue with occasional double counting between this question and questions 2b and 3 will be resolved with the new structure, proposed in the recommendations for questions 2a and 2b.

Overall Impressions

Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that they had no privacy concerns about any of the questions. Six said that they would be willing to complete the survey if they received it. However four said they would not likely complete the survey. Among reasons for not wanting to complete the survey, respondents noted that they already receive “too many things in the mail”; they would want more information about the study and how the information would be used; there were too many questions about income and did not feel comfortable providing such information; concerned that the questions might be used in an IRS audit. One interviewee noted that some questions (e.g. question 2) look like a question from a tax form, and therefore people might be suspicious that the information would be matched or compared directly to tax information. This respondent suggested including a confidentiality note explaining that the information provided would not affect your income for the year or your eligibility for grants and loans.

Two interviewees emphasized their confusion about where to report what income, and what should be included and excluded in each question. One commented that the layout was difficult to navigate and the font was too small to read, and that the interview needed more use of highlighting and bolding to emphasize important portions. Another respondent noted that some of the wording needs to be simplified, but did not provide any specific examples. One participant expressed a concern over the sensitivity of the questions, and noted that they would want to seal and tape the envelope to ensure the information was secure. Another respondent noted that the survey would need to look official and include a letter with signature and toll-free phone number for questions.

A revised survey, incorporating the recommended revisions, is provided below.

Recent Graduates Employment and Earnings Survey

The purpose of the Recent Graduates Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment.

The information you provide on this survey will be grouped with information from other graduates of your program. Average earnings for all graduates of the program will be calculated and reported to the U.S. Department of Education. Your individual earnings will not ever be reported, disclosed or used in any other manner.

1. Please verify that you are <First-Name Last-Name>. Is this correct? If you have legally changed your name, but once went by <First-Name Last-Name>, please answer Yes below.

☐ No Thank you. Please return the survey in the enclosed envelope so that we can remove you from our list and you are not contacted again.

☐ Yes

The next few questions are about income during the 2013 calendar year, NOT the most recent 2014 calendar year.

This survey asks about wages and self-employment income separately. In some cases, it may not be clear to you where income you received should be reported. Please just use your best judgment about this, but please do NOT report the same income more than once. Remember to include income from ALL jobs.

2. Did you receive any wages, salary, tips, overtime pay, bonuses or commissions from an employer in 2013?

☐ No Skip to question 3.

☐ Yes

2a. In total, how much did you earn from all employers, before taxes and other deductions, in 2013?

$_____________

3. Excluding any income mentioned above, did you receive any self-employment income, including income from your own business?

☐ No Skip to question 4.

☐ Yes

3a. What was the amount you received in net self-employment income in 2013? Please report income minus expenses. If this amount is negative, please enter zero.

$_____________

4. People sometimes earn extra money doing work outside of their regular jobs, such as freelancing, consulting, moonlighting, or doing other casual jobs. In 2013, did you earn any other money from work you did that you have not already reported on this survey?

☐ No Thank you. Please return the survey in the envelope provided.

☐ Yes

4a. What was the amount of your additional earnings from work in 2013 that are not already reported above?

$_____________

Thank you for your participation.

Please return the survey in the envelope provided.

15