Memo

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The purpose of this research is to conduct focus groups and cognitive interviews to review and improve the content and features of the mailing materials to motivate response to the National Survey of College Graduates (NSCG). This research will be conducted with people who were in sample for the 2015 NSCG as well as people who are eligible to participate in the NSCG but have not been sampled. The 2015 mailing materials are attached to this letter. The testing is sponsored by the National Science Foundation (NSF). Attached to this letter are five protocols for the research.

This research will begin in January 2016. Staff from the Survey Methodology (SM) area of the Demographic Statistical Methods Division will work with staff from the Center for Survey Methodology in the Research and Methodology Directorate to conduct four focus groups. Each of the four attached protocols corresponds to a group. The first group will consist primarily of people who were in sample for the 2015 NSCG but did not respond, while the second group consists of sampled cases that responded in 2013 and were sampled again in 2015 but did not respond. If we are not able to recruit enough attritors for a focus group, we will instead talk to them in the cognitive interviews, as the sponsor is particularly interested in this group. The third group will consist of 2015 respondents that did not respond until they had received multiple invitations and reminders and the fourth group will consist of people who are eligible to participate in the NSCG, but have not been sampled. The groups will be differentiated by their final disposition in the 2015 NSCG. Talking with prior respondents, nonrespondents, and people unfamiliar with the survey will create a complete picture of what each groups’ preferences and motivations are.

We will begin this research with focus groups to understand the impact of the current mailing materials. Each focus group will consist of 7-10 participants, for a total of 28-40 participants. We plan to recruit 12 participants for each group in case some participants do not show up, but we will cap the group at 10 participants. Participants who arrive after the maximum number of participants is met will be paid the honorarium and excused. Each session will be recorded to facilitate analysis of results.

In these focus groups, participants will be asked about how they react to postal mail and letter content, their overall impressions of surveys, and their opinions on the NSF and NSCG. After introductions, participants will first be asked about what stands out in their postal mail, particularly what envelope features, both overall and specific to the initial NSCG mailing package. Subsequently, we will ask about what information they expect to see in a survey request letter and what language motivates them. Again, they will be asked these questions generically, but also specific to the initial NSCG letter. We will also ask questions about how they feel about being contacted using different modes (e.g., by email and text message) as well as their familiarity with the NSF and NSCG. Finally, we will ask about their experience with surveys, including but not limited to the NSCG. Specifically, we will ask whether they typically respond to surveys, what factors influence their decision to respond, and what about the NSCG makes them more or less likely to respond to that survey than others.

The focus groups will inform what features and content should be included in the NSCG mailing packages. We will use these results to redesign and update the existing mail materials and then use cognitive interviews to assess two new designs. The updates will include the design and messaging of the envelopes and letters.

There will be two rounds of cognitive interviews that will take place between March 2016 and May 2016. A contractor will be hired to conduct 15 interviews per round, for a total of 30 interviews. We will use the results of the first round to modify each set of mailing materials and retest them in the second round. In each round, we will test two alternative sets of mailing materials. The current sett (used in the 2015 NSCG cycle) contains:

1. 10 letters

2. 1 postcard

3. 1 email

We will not know the exact content of the materials that will be used for cognitive testing until we have analyzed the results of the focus groups.

The goal of the cognitive testing is to 1) determine respondent format preference and the motivational reasons to complete the NSCG and 2) find any cognitive problems respondents have with the wording or design of the materials.

All testing will be conducted with prior NSCG sample cases or individuals with a bachelors degree or higher who are eligible to participate in the NSCG but have not been sampled. Ideally, participants will represent 2015 respondents, nonrespondents, attritors, and nonsampled cases that would be eligible to participate with a range of characteristics (e.g., gender, race/ethnicity, age, education, field of degree). All testing will use a retrospective probing method. A copy of the cognitive interviewing protocol is enclosed.

Recruiting efforts for the focus groups will be conducted by CSM, while the contractor will do the recruiting for the cognitive testing. We will use existing phone number information to recruit prior respondents and sampled cases. To recruit non-sampled participants, we will use the Census Bureau’s existing database of potential participants. The telephone script and screener questions are attached to this memo. The script will determine eligibility. Eligibility is the same for the focus groups and the cognitive testing. Participants must have a bachelors degree or higher, be employed outside the Federal Government, speak English, and have been sampled in the 2015 NSCG, where applicable. Focus groups will be conducted at libraries and other similar public facilities in the local metropolitan area (DC, Maryland, and Virginia) while the cognitive interviews will take place at the contractor’s site. Participants will receive a $75 honorarium for the focus groups and $40 honorarium for the cognitive testing to offset costs associated with participation such as transportation and child care.

All focus groups and interviews will be tape-recorded to facilitate a summary of the results. Participants will be asked to sign consent forms and give permission to be recorded. All participants will be informed that their response is voluntary and that the information they provide is confidential.

In general, for Census Bureau staff, it requires 10 screener conversations to recruit one participant. Each screener conversation lasts approximately three minutes. We estimate it will take 35 hours to screen and recruit 70 participants (combined number of participants for the focus groups and cognitive interviews). The estimated time for each focus group is one and a half hours and one hour for the cognitive interview. The estimated burden for the focus groups is 60 hours, the estimated burden for the cognitive interviews is 70 hours. Thus, the total burden is 165 hours. A list of attachments is included in this memo.

The contact person for questions regarding data collection and study design is:

Rachel Horwitz

Survey Methodology Pool

Demographic Statistical Methods Division

U.S. Census Bureau

Room 7H164A

Washington, D.C. 20233

(301) 763-2834

[email protected]

List of Attachments:

1. Focus group protocol 1

2. Focus group protocol 2

3. Focus group protocol 3

4. Focus group protocol 4

5. Cognitive interview protocol

6. Recruiter telephone script and screener

7. Current mailing materials

2