Note to Reviewer - Wording Techniques

February 8, 2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Robin Kaplan Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Survey Question Wording Techniques Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 150 hours.

If there are any questions regarding this project, please contact Robin Kaplan at 202-691-7383.

I. Introduction

An interviewer’s task in the data collection process is a complex one, with many judgments and decisions being made from moment to moment as they ask questions and probe respondents for more information. Amongst the challenges interviewers face is asking respondents to answer questions without knowing beforehand the respondent’s attitude toward the question topic or the respondent’s “true value” (true answer) for that question. Although responses to common survey questions and topics (e.g., labor force participation) are fairly typical, some respondents may provide atypical, or unexpected responses to these questions depending on their circumstances (e.g., Barnett, 1998; De Schrijver, 2012). For instance, a respondent who recently graduated from college (a fairly typical context) may respond differently to questions about employment compared to someone who is currently underemployed and actively looking for a job with more hours (a more atypical context).

How interviewers differentially respond typical and atypical answers that respondents provide has received little empirical investigation, although survey designers have acknowledged that contextual factors and question wording affects interviewer-respondent interactions. As such, survey protocols and questions sometimes have special wording built into them to aid interviewers in their communication with respondents, making the potentially atypical response seem more common. In particular, three wording techniques for interviewers are often emphasized in surveys and in interviewer training: forgiving wording, distancing, and apologizing. However these wording techniques have received little empirical attention beyond informal interviewer debriefing sessions and anecdotal evidence about their function or their effectiveness. Examples of each wording technique are outlined below.

Forgiving Wording

The Current Population Survey (CPS) currently loads positive ‘forgiving wording’ to the front of the involuntary part-time work question (italicized below):

“Some people work part time because they cannot find full time work or because business is poor. Others work part time because of family obligations or other personal reasons. What is your main reason for working part time?”

In the above example about part time work, the forgiving introduction is embedded into the scripted question and provides external attributions for why a respondent may not be able to find full-time work (Tourangeau & Smith, 1996; Naher & Krumpal, 2012; Peter & Valekenburg, 2011). This introduction may affect multiple aspects of the interviewer-respondent interaction. For respondents, it might encourage honest answers. For interviewers, it might make the question more comfortable to ask, particularly if they know beforehand that the respondent’s circumstances and “true” values to related survey questions are atypical. It might also build rapport between the respondent and interviewer. However, empirical research studies on what effect forgiving wording has on the interviewer-respondent interaction is limited, and the few studies that have been done focus on respondent behaviors (Peter & Valekenburg, 2011; Sudman & Bradburn, 1982; Tourangeau & Smith, 1996; Peter & Valekenburg, 2011). Although in the CPS example the forgiving wording introduction is written into the question, little is known about whether interviewers may spontaneously use forgiving wording during the course of the interview as a technique to make atypical survey responses seem more common, and the effectiveness of such a technique.

Distancing and Apologizing

In addition to the use of forgiving wording, interviewers also receive training in techniques to make the question-asking process go more smoothly and to maintain rapport when unexpected responses or atypical survey responses occur, such as distancing themselves from the survey (e.g., “I have to ask every question as worded”) or apologizing (e.g., “I’m sorry to ask this…”). Interviewers are often trained to use such techniques when respondents express frustration about the survey questions (e.g., if a respondent states that the questions are repetitive, difficult to answer, or do not apply to them). Distancing might demonstrate that the interviewer acknowledges a question might be repetitive or irrelevant to the respondent’s circumstances (e.g., asking about vehicle expenses after the respondent tells the interviewer he does not own a vehicle), thus attributing interaction problems to the survey instead of the interviewer. Apologizing may address a slightly different concern where if interviewers display empathy and concern for the respondent, this may help build rapport. This technique may be ideal for the interviewer to use when the interviewer wants to appear trustworthy, when a question seems inappropriate, or when the respondent expresses frustration about the survey or their own circumstances (e.g., the respondent was laid-off from his job).

These special wording techniques (forgiving wording, distancing, and apologizing) are components of everyday conversational norms, and allowing interviewers to incorporate them into the survey interview without changing the meaning of the original question may improve data quality and rapport (e.g., Grice, 1975; Schober & Conrad, 1997). These techniques are also recommended in other interviewing contexts, for instance for medical professionals to elicit honest answers about medical history and past behaviors from their patients (e.g., Bickley & Szilagyi, 2012). However, no research to our knowledge has investigated which wording techniques are most effective for particular survey contexts. For instance, one of these wording techniques may improve data quality over simply asking the question directly using standardized interviewing techniques. Another technique may be helpful for maintaining rapport with respondents and gaining trust with the interviewer and the survey organization. Despite questions sometimes having forgiving wording written into them, and the widespread recommendations to use techniques such as distancing, and apologizing, research findings on their effectiveness is limited. Further, it is has not yet been explored why interviewers might select one technique over another during the course of an interview, or respondents’ reactions to the use of these wording techniques across different survey contexts.

In addition, there is a growing literature on interviewer variance and how factors such as interviewers’ attitudes, personality, and other traits influence the survey process (Jäckle et al., 2013). The decision to use particular wording techniques during a survey interview may also be related to individual differences. In addition, probing is often a personal decision rather than something that interviewers all do consistently. Thus, both the decision to probe and what wording techniques interviewers decide to use may have an impact on interviewer variance. However, no research has investigated what traits across interviewers are associated with use of various wording techniques when probing.

Research Goals

This research will explore why interviewers may select different question techniques over others, the reasons behind these selections, and perceptions of these wording techniques across varying survey contexts. This research may point to the most effective wording techniques to use for survey interviews for particular subsets of respondents and help to develop standardized probes and questions that can be applied in such contexts. In addition, interviewer variance and individual differences can play a role in the interviewer-respondent interaction (e.g., Jäckle et al., 2013). Thus, this research will also explore how individual difference factors that may impact use of different wording techniques. A more detailed description of the methodology follows.

The proposed research is exploratory in nature and will serve as a precursor to future research studies that will investigate survey contexts and questions using actual field interviewers as participants. This research is conceptually similar to a previous OSMR research study and expands on those findings. In this preliminary work, our goal is to explore why participants (taking on the role of survey interviewers for this research) may select different wording techniques across typical and atypical survey contexts where use of such wording techniques (forgiving wording, distancing, or apologizing) are likely to be helpful (e.g., interviewing a respondent about his or her employment situation who is struggling to find work due to age discrimination) relative to a more typical context (e.g., a respondent who is voluntarily retired). Because we don’t know much about the function or effectiveness of these wording techniques, participants will also have the opportunity to indicate if they think the wording technique is promoting some other goal besides enhancing data quality or building rapport. These data will be used to generate additional hypotheses about the use of wording techniques.

We seek to determine whether participants might be more likely to use such techniques across these varying contexts, why they might select one technique over another, and the effect that each wording technique has on perceptions of that survey question. Although the participants in this study are not trained interviewers, we believe that they are qualified to participate because the content of the research questions is centered on human interactions. Participants will be asked to take on the perspective of a survey interviewer using hypothetical vignettes as a proxy for how real interviewers may interact with and react to respondents in the field, an approach used successfully in a previous OSMR research study. Vignettes are a widely used tool for researchers to use in exploratory work to assess reactions, interpretations, and attitudes to survey questions and contexts (e.g., Beck, 2010). They also provide the benefit of not asking respondents to reveal any information about themselves, but rather for hypothetical situations involving fictional others (e.g., Lee, 1993). As such, this study will use hypothetical, fictional vignettes and will not ask participants to answer for their own situations. Additionally, the findings from this study will be used primarily to inform future research with interviewers as the online setting of this study is likely to differ from interviewer-administered surveys that occur via telephone or in person, and will not be used to make final recommendations about interviewer training or question wording.

There are four main exploratory research questions: (a) when faced with typical and atypical survey contexts, do participants decide to use forgiving wording, distancing, apologizing, or direct question techniques?, (b) what do participants believe are the effect of these wording techniques (i.e., do they promote accurate reporting, help build rapport, or something else?), (c) how do participants perceive questions that make use of these different wording techniques (i.e., promote data quality, help build rapport, or something else?), and (d) how do individual differences factors (e.g., perspective-taking ability and social desirability, e.g., Jäckle et al., 2013; Paulhus, 1984) affect the selection and perceptions of these wording techniques?

II. Methodology

To explore these questions, we will conduct an online experimental study. Participants will complete a task where they are instructed to take on the role of survey interviewer in a series of six online interactions with fictional survey respondents who are asked about their current employment status based on questions from the CPS.

Participants will be randomly assigned to one of three survey goal conditions in a single factor between-groups design: (a) prioritize the importance of accurate data, (b) prioritize the importance of rapport and getting future survey responses, or (c) no explicit survey goals provided (control group). These survey goals were selected because they are frequently cited as goals of federal statistical surveys and incorporated into interviewer training. We hypothesize that interviewers have these goals in mind and that the goals influence interviewer behaviors such as the selection of wording techniques and probes.

Secondly, to understand the effect of survey context on selection of wording techniques, half of the survey excerpts (interactions between interviewer and respondent) will be placed in a context where the use of special wording techniques may be particularly useful (e.g., a respondent struggling to find work due to age discrimination), and half will be placed in a fairly typical survey context (e.g., a respondent who is voluntarily retired), such that each participant sees all of these excerpts. The order of these survey excerpt interactions will be randomized. Participants will then select the question wording technique (forgiving wording, distancing, apologizing, or a direct question) they would like to use with each respondent in the given context. The order of these response options will also be randomized. Importantly, each survey excerpt interaction will have a typical and an atypical context to serve as a control comparison for the other:

Typical survey context	Atypical survey context
Recent college graduate	Recent worker who was laid off
Retired worker	Discouraged worker
Voluntary part-time worker	Involuntary part-time worker

Afterward, participants will be asked to indicate why they made each wording selection (to increase data quality, to improve rapport, or to specify some other reason). They will also provide ratings of how comfortable it was for them to ask each question, how honest they think the respondent was in answering the question, and comfortable they think it was for the respondent to answer, given the wording technique they selected. These measures are included to provide insight into the impact of each question and as a manipulation check for the interviewer goal condition. Finally, participants will be asked to complete a brief set of individual differences questions shown to be related to the interpretation of questions containing forgiving wording, including perspective-taking ability and social desirability (see Peter & Valekenburg, 2011; Jäckle et al., 2013). A summary of the study design is below:

Study Design

	Context (within-subjects): Survey Interaction Excerpt: 3 levels x Respondent Typicality: 2 levels
Interviewer Goal (between-subjects): 3 levels	Unemployed, looking	Unemployed worker, not looking	Part-time worker
Try to get the most accurate answers from your partner. We need the most accurate information possible to make sure we get the highest quality data about the US economy and employment.	Typical College grad Atypical Laid off worker	Typical Recently retired Atypical Discouraged	Typical Voluntary Atypical Involuntary
Try to maintain a good relationship with your respondent. We conduct this survey each month, and we need them to be willing to participate in future surveys about the US economy and employment.	Typical College grad Atypical Laid off worker	Typical Recently retired Atypical Discouraged	Typical Voluntary Atypical Involuntary
No specific goals mentioned (Control)	Typical College grad Atypical Laid off worker	Typical Recently retired Atypical Discouraged	Typical Voluntary Atypical Involuntary

A list of outcome variables is below (these are explained in more detail in the following sections):

Dependent Variables:

• Ranking of the effectiveness of each wording technique at getting accurate high quality data

• Ranking of the effectiveness of each wording technique at maintaining a good relationship with your respondent

• Rating of how comfortable the question was for the participant (Interviewer proxy) to ask the question

• Rating of how honest they think the respondent was answering the selected question

• Rating of how comfortable they think the respondent was answering the selected question

• Perspective taking scale

• Social desirability scale

• Demographics

This research is exploratory and there is very little literature on which to base predictions and specific hypotheses. However, we expect that interviews that take place in atypical contexts may lead to higher use of special wording techniques relative to the more typical interview contexts. We also expect that the use of special wording techniques will make the questions more comfortable for the interviewers to ask and for the respondents to answer in these atypical contexts. We expect that direct wording techniques will be selected more frequently in the more typical survey contexts. We also expect some participants to indicate novel reasons they selected each wording technique, and this will be used for future hypothesis generation.

In addition, it is possible that participants who have the goal of eliciting high quality responses may opt to use forgiving wording more frequently compared to participants with the goal of maintaining positive relationships, as forgiving wording sometimes reduces social desirability concerns with answering questions (Peter & Valekenburg, 2011; Sudman & Bradburn, 1982; Tourangeau & Smith, 1996). These participants may also perceive respondents as providing more honest answers. Participants who have the goal of maintaining positive relationships with their respondents may focus more on rapport building and opt to use techniques like distancing and/or apologizing more frequently. They may also perceive respondents as feeling more comfortable answering the questions.

Finally, people who score high on perspective-taking ability and/or social desirability may opt to use more special wording techniques than those lower in these traits as they may be more attuned to the social context of the interview (e.g., Peter & Valekenburg, 2011; Jäckle et al., 2013).

We plan to conduct a 3 (survey goal condition) X 2 (survey context typicality) Chi-Square analysis on each of the 3 survey interaction contexts to determine if there are differences in wording technique selection. In addition, we will use multinomial regression analysis to determine the odds of selecting each type of wording technique across the goals and typical and atypical survey contexts, building in covariates (i.e., all dependent variables listed above) to explore all possible outcomes and relationships amongst wording technique selections, and individual difference factors.

Procedure

The full survey instrument is included as appendices; each segment of the survey appears as a separate appendix. An overview of the protocol is given below, and a detailed description about each segment follows.

Participants will be introduced to the survey (Appendix A) and read about information describing the survey and their survey partner (the respondent) and excerpts of their survey responses where they will make their wording technique selections (Appendix B). After completing each of 6 survey interactions, participants will answer retrospective ratings about the effectiveness of each wording technique, and how comfortable they felt as survey interviewer asking each question and how comfortable they perceive the respondent felt answering the question that was selected (Appendix C). Then participants will complete individual differences inventories (Appendix D), some demographic questions (Appendix E), and finally they will be thanked for their participation and asked to indicate any thoughts about the study (Appendix F). A more detailed description of each appendix is outlined below, and the full version of each appendix is attached at the bottom of this document.

• Appendix A: Participants are introduced to the study. We explain that we are conducting research to understand how to best ask questions in surveys about the US economy and employment, and we are testing different ways that survey interviewers can ask questions to understand which way works best. They are assigned randomly to one of the three study goals:

  • Data quality condition: Try to get the most accurate answers from your partner. We need the most accurate information possible to make sure we get the highest quality data about the US economy and employment.

  • Rapport condition: Try to maintain a good relationship with your respondent. We conduct this survey each month, and we need them to be willing to participate in future surveys about the US economy and employment.

  • Control condition: This is an important survey about the US economy and employment.

See Appendix A for the full text of the introduction to the study.

• Appendix B: Participants will engage in six different survey interactions placed in varying contexts (typical vs. atypical) with fictional survey partners. The presentation order of the six survey interviews will be randomized. The six types of survey contexts are as follows:

Typical survey context	Atypical survey context
Recent college graduate	Recent worker who was laid off
Retired worker	Discouraged worker
Voluntary part-time worker	Involuntary part-time worker

Participants will learn about the individual context of the fictional survey partner during the introduction of the survey topic. Then they will be asked to select a follow-up question to ask from a set of questions that represent the specialized wording techniques (i.e., forgiving wording, distancing, or apologizing), or using the direct question. The presentation order of the questions will be randomized. They will continue this process until they’ve completed each interview. For a full transcript of each interaction, please see Appendix B.

• Appendix C: After completing the survey interactions, we will ask participants to think back to each interaction and why they made each wording selection choice. We will provide excerpts from each survey interaction and display the wording technique selected as a retrieval cue. We present these retrospectively so as not to draw attention to the goals of the study during the survey interaction portion of the study. We will ask participants to rank each wording technique as follows (the order of the wording techniques will be randomized):

• Rank the four options as to how effective you think each would be at getting accurate high quality data:

  • [Insert: Forgiving wording technique version of the question]

  • [Insert: Apologizing wording technique version of the question]

  • [Insert: Direct wording technique version of the question]

  • [Insert: Distancing wording technique version of the question]

• Rank the four options again, this time as to how effective you think each would be at maintaining a good relationship with your respondent:

  • [Insert: Forgiving wording technique version of the question]

  • [Insert: Apologizing wording technique version of the question]

  • [Insert: Direct wording technique version of the question]

  • [Insert: Distancing wording technique version of the question]

We will also ask for ratings of how comfortable the participant felt asking the question and perceptions of how comfortable the respondent felt answering them. This process will be repeated for each of the survey interactions. See Appendix C for the exact wording of these questions.

• Appendix D: Participants will also complete individual differences scales that are expected to relate to the outcome variables in this study, described below:

  • Perspective Taking Scale (Jäckle, 2013). Because the tasks in this research study require participants to take on the perspective of interviewers, one potential covariate for wording technique selection in this study is perspective-taking ability. Participants will complete a scale developed to assess these traits. The perspective taking subscale contains 6 items designed to assess the ability to read others and been shown to affect interviewer variance in other research (Jäckle, 2013).

  • Balanced Inventory of Socially Desirable Responding (BIDR; Paulhus, 1984). The tasks in this research study refer to situations where survey respondents might exhibit a social desirability bias when answering questions. For instances, participants who exhibit high versus low levels of social desirability and may be differentially affected by forgiving wording (see Peter & Valekenburg, 2011). Thus, another potential covariate in this study is participants’ own tendency to respond in socially desirable ways. Participants will complete a short version of Paulhus’s (1984) Balanced Inventory of Socially Desirable Responding (BIDR). This is a commonly used scale to assess two dimensions of social desirability. The first dimension is Self-Deceptive Enhancement, or the tendency to give self-reports that are believed but have a positivity bias and contains 10 items. The second dimension is Impression Management and assesses deliberate self-presentation to an audience. It also contains 10 items.

See Appendix D for the complete version of each scale.

1. Appendix E: At the end of the task, participants will be asked to provide basic demographic information: their gender, age, race, and education.

2. Appendix F: The participants will be thanked for their participation and given an opportunity to leave any comments in an open-ended text entry box.

III. Participants

Participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult U.S. citizens (18 years and older); this study is focused on internal validity rather than representativeness of any population. This research design requires a sample of 450 participants in order to sufficiently explore the range of variables of interest and because we expect a very small effect size since, as the study manipulations are subtle for online surveys of this nature. These participants will be randomly assigned to the one of the three between-subjects survey goals; (each participant will also experience the 2 within subjects-factors: excerpt and typicality) with 150 participants per group.

IV. Burden Hours

Our goal is to obtain responses from 450 participants recruited from Amazon Mechanical Turk. Each session is expected to take no more than 20 minutes to complete, for a total of 150 burden hours. The survey will be administered completely online at the time and location of the participant’s choosing. Participants will be compensated $1.50 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks.

V. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Participants will be informed that the study is about their perceptions of different types of questions. Once participants are recruited into the study, they will be sent a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We estimate that it will take 20 minutes to complete this survey. . Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

1. Attachments

Appendices A through F

References

Appendix A - Survey Introduction

We are conducting research to understand how to best ask questions in federal government surveys about the US economy and employment. This study is about testing different ways that survey interviewers can ask questions and understanding which ways work best. We want to learn what you – as a person who takes surveys – would want to hear as a respondent. How would you want someone to ask you survey questions?

Please do not use your browser's back button. Remember to maximize your browser screen.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We estimate that it will take 20 minutes to complete this survey. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

---- page break ----

In this study, imagine that you have been hired as a Survey Interviewer and it is your job to ask people survey questions. These interviews take place online through messaging windows, like perhaps you have seen for customer service help on some websites. You will see the survey questions that need to be asked. You’ll be interacting with a partner whose responses represent what we hear from real survey respondents. Your task is to ask the questions with these goals in mind:

1.      The question as written represents the information that we need to collect.

2.      [INSERT CONDITION]:

• [Data quality condition]: Try to get the most honest and truthful answers from your partner. We need the most accurate information possible to make sure we get the highest quality data about the US economy and employment.

• [Rapport condition]: Try to make your partner feel comfortable answering the questions. We conduct this survey each month, and we need them to be willing to participate in future surveys about the US economy and employment.

• Control condition: This is an important survey about the US economy and employment.

3. Remember: We want to learn what you – as a person who takes surveys – would want to hear as a respondent. How would you want someone to ask you survey questions? Please think carefully about each question you ask, as you will be asked questions about them later on.

Appendix B – Survey Interaction Excerpts

Interview 1 (Alex / Unemployed, looking for work / Atypical Context)

You are interviewing Alex for the General Social Research Survey.

You start with the standard introduction:

"Hi Alex, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Alex, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

“Are you ready to start?”

Alex says:

"Yeah yeah. I know the drill. I’m not looking forward to those questions on unemployment this time though since I lost my job…"

The next question is written as:

“Are you currently doing any work for pay?”

What would you like to say to Alex?

• “The first question I am supposed to ask you is about employment. We have to ask the same question of everybody – Are you currently doing any work for pay?”

• “Are you currently doing any work for pay?”

• “Some people may not work because the economy is still recovering, they are waiting for the right job offer, or other personal reasons – Are you currently doing any work for pay?”

• “I’m sorry to ask about this, but – Are you currently doing any work for pay?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Alex says:

"Uh, no. No, I'm not."

The next question is written as:

“During the last four weeks, have you been doing anything to look for work?”

What would you like to say to Alex?

• "During the last four weeks, have you been doing anything to look for work?"

• “I'm sorry to ask about this again but - During the last four weeks, have you been doing anything to look for work?"

• "Now they want me to ask you - During the last four weeks, have you been doing anything to look for work?"

• “Some people may not look for work because business is poor, family obligations, or other personal reasons. During the last four weeks, have you been doing anything to look for work?"

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Alex says:

 

"Of course I looked for work. I went to the jobs center three times and I put my resume in at least 12 places."

---- page break ----

That completes Interview 1. 

Please wait to begin Interview 2.

---- page break ----

Interview 2 (Pat, Unemployed, recent college graduate looking for work, Typical Context)

You are interviewing Pat for the General Social Research Survey.

You start with the standard introduction:

"Hi Pat, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Pat, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

"Are you ready to start?"

Pat says:

"Oh, right. Since I just graduated from college with my degree, I’m still looking for a job at the moment…"

The next question is written as:

“Are you currently doing any work for pay?”

What would you like to say to Pat?

• "The first question I am supposed to ask you is about employment. We have to ask the same question of everybody - Are you currently doing any work for pay?"

• "Are you currently doing any work for pay?"

• “Some people may not work because the economy is still recovering, they are waiting for the right job offer, or other personal reasons – Are you currently doing any work for pay?”

• "I’m sorry to ask about this, but – Are you currently doing any work for pay?"

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Pat says:

"No.”

The next question is written as:

“During the last four weeks, have you been doing anything to look for work?”

What would you like to say to Pat?

• "During the last four weeks, have you been doing anything to look for work?"

• “I'm sorry to ask about this again but - During the last four weeks, have you been doing anything to look for work?"

• "Now they want me to ask you - During the last four weeks, have you been doing anything to look for work?"

• “Some people may not look for work because business is poor, family obligations, or other personal reasons. During the last four weeks, have you been doing anything to look for work?"

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Pat says:

 

"Yes, I submitted resumes and talked to my school’s job counselor.”

---- page break ----

That completes Interview 2. 

Please wait to begin Interview 3.

Interview 3 (Frankie, Discouraged worker, Atypical Context)

You are interviewing Frankie for the General Social Research Survey.

You start with the standard introduction:

"Hi Frankie, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Frankie, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

"Are you ready to start?"

Frankie says:

"Oh… I’m so tired of getting questions about my employment. I keep hearing: ‘Your resume is excellent but the position requires someone more up-to-date on the newest techniques.’ I can't help how old I am!”

The next question is written as:

“During the last four weeks, have you been doing anything to look for work?”

What would you like to say to Frankie?

• "The first question I am supposed to ask you is about employment. We have to ask the same question of everybody. During the last four weeks, have you been doing anything to look for work?”

• "During the last four weeks, have you been doing anything to look for work?"

• “Even the most qualified people may not have recently looked for work because the economy is still recovering, they are waiting for the right job offer, or personal reasons. During the last four weeks, have you been doing anything to look for work?”

• "I’m sorry to ask about this, but – During the last four weeks, have you been doing anything to look for work?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Frankie says:

“I looked for work for 14 straight weeks and came up with nothing. That’s more than anyone else I know. My friend John looked for only 12 weeks. I won't go through that again. There's no job out there for me.”

The next question is written as:

“What is the main reason you were not looking for work during the last 4 weeks?”

What would you like to say to Frankie?

• “What is the main reason you were not looking for work during the last 4 weeks?”

• “I'm sorry to ask about this again but - What is the main reason you were not looking for work during the last 4 weeks?”

• "Now they want me to ask you - What is the main reason you were not looking for work during the last 4 weeks?”

• “Some people might not look for work because business is poor, family obligations, or other personal reasons. What is the main reason you were not looking for work during the last 4 weeks?"

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Frankie says:

 

"Like I said, there is just no job out there for me. People don’t want to hire someone my age.”

---- page break ----

That completes Interview 3. 

Please wait to begin Interview 4.

Interview 4 (Jean, Retired worker, Typical Context)

You are interviewing Jean for the General Social Research Survey.

You start with the standard introduction:

"Hi Jean, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Jean, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

"Are you ready to start?"

Jean says:

"Sure, but I just recently retired after over 35 years at my company. I’ve been looking for a part-time job just for fun, but haven’t found anything.”

The next question is written as:

“During the last four weeks, have you been doing anything to look for work?”

What would you like to say to Jean?

• "The first question I am supposed to ask you is about employment. We have to ask the same question of everybody. During the last four weeks, have you been doing anything to look for work?”

• "During the last four weeks, have you been doing anything to look for work?"

• “Even the most qualified people may not have recently looked for work because the economy is still recovering, they are waiting for the right job offer, or personal reasons. During the last four weeks, have you been doing anything to look for work?”

"I’m sorry to ask about this, but – During the last four weeks, have you been doing anything to look for work?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Jean says:

“Hm… No I don’t think I looked in the past 4 weeks. Like I mentioned, I’m retired now but occasionally look for part-time jobs just for fun and to fill my time.”

The next question is written as:

“What is the main reason you were not looking for work during the last 4 weeks?”

What would you like to say to Jean?

• “What is the main reason you were not looking for work during the last 4 weeks?”

• “I'm sorry to ask about this again but - What is the main reason you were not looking for work during the last 4 weeks?”

• "Now they want me to ask you - What is the main reason you were not looking for work during the last 4 weeks?”

• “Some people may not look for work because business is poor, family obligations, or other personal reasons. What is the main reason you were not looking for work during the last 4 weeks?"

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Jean says:

 

"The main reason is because I’m retired I don’t really need to work anymore. I’ve been looking on and off, but not too seriously.”

---- page break ----

That completes Interview 4. 

Please wait to begin Interview 5.

Interview 5 (Sam, Involuntary Part Time Worker, Atypical Context)

You are interviewing Sam for the General Social Research Survey.

You start with the standard introduction:

"Hi Sam, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Sam, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

"Are you ready to start?”

Sam says:

“Ugh - I am so frustrated with my job. My hours recently got cut to part-time and I haven’t been able to find another full time job...”

The next question is written as:

“Do you want to work a full time workweek of 35 hours or more per week?”

What would you like to say to Sam?

• "The first question I am supposed to ask you is about working full time. We have to ask the same question of everybody. Do you want to work a full time workweek of 35 hours or more per week?”

• "Do you want to work a full time workweek of 35 hours or more per week?"

• “Many qualified workers in the US work part time instead of full time for a variety of reasons. Do you want to work a full time workweek of 35 hours or more per week?”

• "I’m sorry to ask about this, but – Do you want to work a full time workweek of 35 hours or more per week?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Sam says:

“I’ve been trying for weeks, but can’t another full-time position. I applied to 6 jobs. I even am applying to jobs as far as 55 miles away from home.”

The next question is written as:

“What is your main reason for working part time?”

What would you like to say to Sam?

• “What is your main reason for working part time?”

• “I'm sorry to ask about this again, but - What is your main reason for working part time?”

• "Now they want me to ask you - What is your main reason for working part time?”

• “Some people work part time because they cannot find full time work or because business is poor. Others work part time because of family obligations or other personal reasons. What is your main reason for working part time?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Sam says:

 

"Like I said before it is very difficult to find another full time position that I’m qualified for, so I am stuck working part-time… It’s not that I’m not trying. I’m having a hard time making ends meet with so few hours.”

---- page break ----

That completes Interview 5. 

Please wait to begin Interview 6.

Interview 6 (Sam, Voluntary Part Time Worker, Typical Context)

You are interviewing Frances for the General Social Research Survey.

You start with the standard introduction:

"Hi Frances, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy. 

"Are you ready to start?"

---- page break ----

You said:

"Hi Frances, thanks for participating in the General Social Research Survey on behalf of the Institute for General Social Research. I’ll be asking about your employment and outlook on the US economy.”

"Are you ready to start?"

Frances says:

“Okay, sure. But I’ve only been working part time, 10 to 15 hours per week lately since I decided to go back to school.”

The next question is written as:

“Do you want to work a full time workweek of 35 hours or more per week?”

What would you like to say to Frances?

• "The first question I am supposed to ask you is about working full time. We have to ask the same question of everybody. Do you want to work a full time workweek of 35 hours or more per week?”

• "Do you want to work a full time workweek of 35 hours or more per week?"

• “Many qualified workers in the US work part time instead of full time for a variety of reasons. Do you want to work a full time workweek of 35 hours or more per week?”

• "I’m sorry to ask about this, but – Do you want to work a full time workweek of 35 hours or more per week?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Frances says:

“No, I decided to cut back my hours to try to finish school more quickly. I also help out with some family obligations.”

The next question is written as:

“What is your main reason for working part time?”

What would you like to say to Frances?

• “What is your main reason for working part time?”

• “I'm sorry to ask about this again, but - What is your main reason for working part time?”

• "Now they want me to ask you - What is your main reason for working part time?”

• “Some people work part time because they cannot find full time work or because business is poor. Others work part time because of family obligations or other personal reasons. What is your main reason for working part time?”

---- page break ----

You said:

<INSERT WORDING TECHNIQUE SELECTION>

Frances says:

 

"The main reason is going back to school.”

---- page break ----

That completes Interview 6. 

Please click Next to continue.

Appendix C - Follow-up Questions

Now we are interested in understanding the reasons you selected the questions you did. On the next few pages, you will see excerpts from the survey interviews you just completed – including the questions you selected and your survey partners answered. As you re-read each excerpt, please think carefully about why you chose the question type you did.

Earlier, you interviewed Alex. Here is an excerpt from your conversation:

<INSERT EXCERPT FROM QUESTION/ANSWER EXCHANGE>

You had the option to ask Alex one of four questions. Rank the four options as to how effective you think each would be at getting accurate high quality data from Alex:

• [Insert four question options, forgiving, distancing, apologizing, direct, in random order]

Rank the four options again, this time as to how effective you think each would be at maintaining a good relationship with Alex:

• [Insert four question options, forgiving, distancing, apologizing, direct, in random order]

----page break---

Earlier, you chose to ask Alex: [INSERT QUESTION SELECTION]

How comfortable did you feel asking Alex this question?

• Not at all comfortable

• Slightly comfortable

• Moderately comfortable

• Very comfortable

• Completely comfortable

How honest do you think Alex was answering this question?

• Not at all honest

• Slightly honest

• Moderately honest

• Very honest

• Completely honest

How comfortable do you think Alex felt answering this question?

• Not at all comfortable

• Slightly comfortable

• Moderately comfortable

• Very comfortable

• Completely comfortable

<REPEAT SEQUENCE FOR EACH OF THE INTERVIEWS>

Appendix D - Individual Differences Questions

The following questions are about how you see yourself as a person. Please select the number which best describes how you see yourself where 1 means ‘does not apply to me at all’ and 7 means ‘applies to me perfectly’.

Please describe yourself as you generally are now, not as you wish to be in the future. Describe yourself as you honestly see yourself, in relation to other people you know of the same sex as you are, and roughly your age.

Perspective taking scale – Grid with all questions for the scale appearing on one page

1 2 3 4 5 6 7

(does not apply to me at all) (applies to me perfectly)

I see myself as someone who…

1. Is good at sensing what others are feeling

2. Anticipates the needs of others

3. Senses others’ wishes

4. Can tell a lot about people from how they live

5. Is very aware of my surroundings

6. Knows what to say to make people feel good

Balanced Inventory of Socially Desirable Responding – 2 Grids

The following questions are about how you see yourself as a person. Please select the number which best describes how you see yourself where 1 means ‘strongly disagree and 7 means ‘strongly agree’.

Please describe yourself as you generally are now, not as you wish to be in the future. Describe yourself as you honestly see yourself, in relation to other people you know of the same sex as you are, and roughly your age.

1 2 3 4 5

(strongly disagree) (strongly agree)

1. My first impressions of people usually turn out to be right.

2. It would be hard for me to break any of my bad habits.

3. I have not always been honest with myself.

4. I always know why I like things.

5. Once I’ve made up my mind, other people can seldom change my opinion.

6. It’s hard for me to shut off a disturbing thought.

7. I never regret my decisions.

8. I rarely appreciate criticism.

9. I am very confident of my judgments.

10. I don’t always know the reasons why I do things.

1 2 3 4 5

(strongly disagree) (strongly agree)

1. I sometimes tell lies if I have to.

2. I never cover up my mistakes.

3. I always obey laws, even if I am unlikely to get caught.

4. I have said something bad about a friend behind his or her back.

5. When I hear people talking privately, I avoid listening.

6. I have received too much change from a salesperson without telling him or her.

7. When I was young I sometimes stole things.

8. I have done things that I don’t tell other people about.

9. I never take things that don’t belong to me.

10. I don’t gossip about other people’s business.

Appendix E - Demographic Questions

We’re almost done – we just have a few more questions we’d like to know about you.

1. How old are you? ___ [validate two digits]

2. What is your gender?

   • Male

   • Female

3. Which of the following best describes you?

   • Employed full time

   • Employed part time

   • Unemployed

   • Student

   • Retired

4. Are you Hispanic or Latino?

   • Yes

   • No

5. What is your race? Please select one or more.

• American Indian or Alaska Native

• Asian

• Black or African American

• Native Hawaiian or Other Pacific Islander

• White

6. Which of the following best describes your highest level of education?

   • Less than high school

   • High school diploma or equivalent

   • Some college

   • Associate’s degree or Bachelor’s degree

   • Master’s degree or Doctoral degree

Appendix F - Thank you page

Thank you for participating in our study.

If you have any comments you would like to share, please use the space below.

[text entry box]

References

Barnett, J. (1998). Sensitive questions and response effects: an evaluation. Journal of Managerial Psychology, 13(1/2), 63-76.

Beck, J. (2010). On the Usefulness of Pretesting Vignettes in Exploratory Research. Survey Methodology, 02.

Bickley, L., & Szilagyi, P. G. (2012). Bates' guide to physical examination and history-taking. Lippincott Williams & Wilkins.

Davis, M. H. (1983). Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of personality and social psychology, 44(1), 113.

De Schrijver, A. (2012). Sample survey on sensitive topics: Investigating respondents' understanding and trust in alternative versions of the randomized response technique. Journal of Research Practice, 8(1).

Lee, R. (1993). Doing Research on Sensitive Topics, Thousand Oaks, CA: Sage.

Jäckle, A. E., Lynn, P., Sinibaldi, J., & Tipping, S. (2013). The effect of interviewer experience, attitudes, personality and skills on respondent co-operation with face-to-face surveys. In Survey research methods (Vol. 7, No. 1, pp. 1-15). European Survey Research Association.

Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions the impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275-304.

Näher, A. F., & Krumpal, I. (2012). Asking sensitive questions: the impact of forgiving wording and question context on social desirability bias. Quality & Quantity, 46(5), 1601-1616.

Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of personality and social psychology, 46(3), 598.

Peter, J., & Valkenburg, P. M. (2011). The Impact of “Forgiving” Introductions on the Reporting of Sensitive Behavior in Surveys: The Role of Social Desirability Response Style and Developmental Status. Public Opinion Quarterly, 75(4), 779–787.

Schober, M. F., & Conrad, F. G. (1997). Does conversational interviewing reduce survey measurement error?. Public opinion quarterly, 576-602.

Sudman, S., & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design.

31