Note to Review for Classifying Employment Situations Study

OMB_package_classifying_fricker2.docx

Cognitive and Psychological Research

Note to Review for Classifying Employment Situations Study

OMB: 1220-0141

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1220-0141 can be found here:

Document [docx]

Download: docx | pdf

July 4, 2012

NOTE TO THE

REVIEWER OF:

OMB CLEARANCE 1220-0141

“Cognitive and Psychological Research”

FROM:

Scott Fricker

Research Psychologist

Office of Survey Methods Research

SUBJECT:

Submission of Materials for the Classifying Employment Situations Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is a maximum of 90 hours.

If there are any questions regarding this project, please contact Scott Fricker at

202-691-7390.

Introduction and Purpose

Survey designers attempt to provide respondents with response options that are mutually exclusive and unambiguous. In some instances, however, the concepts underlying the set of response options reflect the survey designers’ technical viewpoint rather than the kinds of conceptual distinctions that respondents are likely (or even able) to make. This disconnect undermines the effectiveness of category labels and can reduce the accuracy of survey responses (Schober and Conrad, 1997; Tourangeau et al., 2006). Ambiguity also can arise from the nature of the concept itself; concepts differ in their complexity and specificity. Some concepts are unidimensional (i.e., membership is determined by a single factor, like concepts of ‘height’ or ‘weight’) and others are multidimensional (i.e., membership is determined by a set of factors, as with concepts like ‘healthy’ or ‘clothing’). Some have relatively dense representation (i.e., they have multiple intercorrelated features relevant for category membership, with only a few features being irrelevant), while others have sparse representation. Empirical work in the field of cognition has shown that people often have difficulty making categorical decisions about cognitively complex concepts (e.g., sparse concepts, or multidimensional concepts when some features fit the concept but others do not) (e.g., Rosch and Mervis, 1975; Kurtz and Gentner, 2001). These findings are mirrored in numerous survey methods studies that have demonstrated increases in respondent comprehension problems for questions asking about vague or broad concepts (see, e.g., Tourangeau, Rips, and Rasinski, 2000, for a review).

Moreover, we know that survey respondents presented with cognitively complex or ambiguous concepts often do not make the effort required to understand them as intended (e.g., Krosnick et al., 1996; Schober and Conrad, 1997). This finding has particular import for close-ended questions. Given a set of response options that vary in their conceptual complexity, respondents may avoid thoughtful consideration of the more complex options and instead select a response choice from among the remaining items (provided that it offers an acceptable – but not optimal – answer).

To explore this issue, Bosley, Fricker, and Gillman (2012) examined the impact of conceptual variability and question formatting on response distributions in the Classification of Worker (COW) question, a close-ended item taken from the Current Population Survey (CPS). Noting discrepancies between COW-based estimates of self-employment and those published from other sources, and drawing on theories of categorization, the authors argue that the concept of self-employment is more complex than concepts for the other three employment categories included in the response options (i.e., employed by government, employed by a private company, employed by a non-profit organization). They hypothesized that this differential conceptual complexity among the COW categories may induce respondent satisficing, resulting in the underutilization of the ‘self-employment’ category.

Participants in their study were administered a series of narrative vignettes describing the employment situation for fictional individuals, and then asked to classify those individuals using two different groupings of the COW classification. Half of the sample made choices from among the entire set of four response options (as in the existing COW item), while the other half were presented with only the “self-employed” option as a “yes-or-no” choice. The vignettes were presented a second time, and everyone classified the same jobs into just three employment categories —government, private or non-profit. The main study result showed that respondents were significantly less likely to categorize the person depicted in the vignette as ‘self employed’ when they responded to the 4-option item than when they answered the 2-option item. Respondent debriefings also revealed systematic differences in respondents’ ability to generate examples and definitions of self-employment vs. the other three work categories. Together these findings suggest that there may be something conceptually distinct about self employment, and that respondents’ ability and/or willingness to answer a class of worker item accurately will depend in part on the nature of the concepts being asked about and the form the question takes.

The purpose of the present study is to extend the results of Bosley, Fricker and Gillman (2012) in three ways. First, we will examine the effects of question wording (i.e., 4-option vs. 2-option item) on response distributions for all four COW categories: employed by government, employed by a private company, employed by a non-profit organization, and self employed. This will help us to validate earlier results and to evaluate whether there is something unique about the concept of self employment, or if all of the COW categories are equally influenced by question format. Second, we will examine potential response order effects, given that the ‘self employment’ option always appeared as the last response option (as is the case in the production COW question). Finally, we will further explore how the four COW concepts are cognitively represented in respondents’ memory using a feature-generation task that has been used in the categorization literature to differentiate different concept types (e.g., Goldwater, Markman, & Stillwell, 2010).

Research Design

The design of this study will largely follow that of Bosley, Fricker, and Gillman (2012). Participants will be given a series of narrative vignettes that briefly describe the work situations of different fictional characters (Attachment I). Test participants will be asked to indicate which COW category they believe best fits the characters’ work situations based on the information provided in each vignette (Attachment II). At the end of the test session, participants will be administered a set of debriefing questions to explore their reasoning for their vignette-based responses and their understanding of COW-related concepts (Attachment III).

As in the prior study, one of our key experimental manipulations will be the way in which the classification request is presented. Study participants will be asked to go through a set of 12 vignettes twice during the test session. In the first round, half of the study participants will be shown the vignettes and asked to classify the character based on the current COW question (Standard condition); the other half of the participants only will be asked to make a dichotomous decision about the character’s employment status (Split condition). Unlike in the previous study where respondents only were administered the 2-option item about self employment, respondents in this study will be asked a 2-option question about one of the four COS categories (i.e., Is person X employed by government or not employed by government?). Participants in both conditions then will be asked to go back through each of the 12 vignettes again and decide “the nature of the employment:” private, for profit; government; or non-profit. This design will enable us to capture the full set of possible class of worker responses for the split treatment participants, assess reliability between rounds of administration for the standard group respondents, and examine differences in response distributions under the standard and split conditions.

In addition to varying the number of response options across participants, this study also will examine potential response order effects. Half of the respondents in the standard condition will be shown self employed as the first response option and the other half will be shown it as the last option. We focused only on this one response option and these two positions to address one possible explanation for the results of the prior study and because it would have been prohibitively difficult to examine every combination and permutation of response option-by-position.

This study also will ask participants to list characteristics of ‘typical’ examples of each of the four COW employment situations. For this task, we adopt procedures and instructions used by Rosch (1973) (see Attachment III). These responses will be coded using a coding scheme developed by Barr and Caplan (1987) to distinguish between different concept types based on the kinds of properties respondents generate.

To address the research questions, two methods will be used: an in-person cognitive interview and a web-based questionnaire. In the in-person interview, participants will come into the Office of Survey Methods Research (OSMR) lab and begin by completing a self-administered questionnaire that contains the vignette-based items. They then will be asked as series of interviewer-administered questions aimed at assessing their comprehension of COW concepts, their response strategies and recall effort. Responses will be audio recorded and captured by the interviewer. Web participants will be given the same self-administered, vignette-based items as the lab participants, but the debriefing questions used to evaluate their comprehension of the COW concepts, recall strategies, and effort will be scripted into the web instrument. The time it takes to complete the study protocol will be recorded, and the verbal responses to the response strategy question will be transcribed.

TryMyUI

TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tests they qualify for. Each usability test is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. A video showing the participant completing the task is then provided.

For this study, this usability testing site is being used as a way to collect a large amount of data in an efficient (both in terms of financial and staff resources) way. TryMyUI is responsible for recruitment, screening, administration of the survey, and data collection. The company has been found to be able to provide high quality, qualitative data within hours of receiving the study from the sponsor. The same data would take weeks for OSMR to collect in the usability lab. Participants are used to providing qualitative feedback as they work on online tasks, and have been found to make very insightful, in-depth comments about their response process as they work on the self-administered survey. This allows for both quantitative and qualitative data to be collected in a very time efficient manner.

Participants will receive a link to an online survey (hosted by Qualtrics). For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMyUI, participants will be asked to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.

Vignettes

We selected 12 vignettes that were used in the earlier study. The vignettes described work being done in one of three employment sectors (i.e., government, private-for profit, and non-profit), and varied the amount of information that suggested self-employment (i.e., by making no direct reference to self-employment dimensions, referencing positive evidence of one or the other self-employment dimensions, or referencing negative evidence of one or the other self-employment dimensions). The following example illustrates these manipulations. In the first vignette below, the intent was to suggest that the job holder works for a governmental agency absent any mention of self-employment dimensions. (We refer to this as the “baseline” version of the vignette.)

“Marvin works full time collecting water samples and measuring air quality for his state’s environmental agency. He gathers environmental data at sampling locations in the northwest part of the state where he lives. He works mostly outdoors in the field in his car and on foot. He does occasionally attend meetings at agency headquarters in the state capital.”

This vignette is crafted deliberately to leave open the possibility that Marvin is in his own eyes self-employed even though it also clearly indicates the governmental nature of his work. That is, any holder of this job as described above could well respond to the CPS by selecting “self-employed” when the COW question is asked. As it stands, then, it is likely that some fraction of our study participants will classify Marvin as “self-employed” in the “Standard” condition¹. If this occurs, these responses would be “errors.” As we understand it, the model of employment on which the CPS’s work classification scheme rests treats a “government-related self-employed” response as impossible. Thus the proportion of this kind of “misclassification” provides an estimate of one type of potential response error in the CPS data.

We reproduced this vignette with information added that was intended to increase Marvin’s (or our study participants’) likelihood of judging him to be self-employed.

“Marvin works full time collecting water samples and measuring air quality for his state environmental agency. He gathers environmental data at certain locations in the northwest part of the state where he lives. He’s mostly outdoors in the field in his car and on foot. He occasionally attends meetings at agency headquarters in the state capital, but he does his work free from any day-to-day supervision or direction.”

This variation uses information about the critical indicator of supervision. Another variation involving the financial independence indicator but designed to lessen the appearance that Marvin is self-employed would replace the italicized addition above with the following information. “Marvin has to record his hours worked each day on a time-sheet that he submits bi-weekly in order to collect his pay.” (See Attachment I for the full set of baseline vignettes.)

Each test participant will receive a total of 12 vignettes (6 describing work done in a private company, 3 describing work in the government sector, and 3 describing work in the non-profit sector). In addition, four of the twelve vignettes given to participants will contain only the baseline information (i.e., no mention of self-employment dimensions), four will contain positive evidence of self-employment (i.e., two suggesting greater supervisory autonomy, and two suggesting greater financial autonomy), and four will contain negative evidence of self-employment (again, varying along the two self-employment dimensions). Individual participants will be randomly assigned only one version of each vignette, and the order of presentation will be randomized within each test session as well.

Participants

Participants for the in-person interviews will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. Web participants will be recruited by TryMyUI. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on their website. Participants will be selected with varying levels of age, gender, and education.

Payment

Lab-based participants will receive $40. For the web survey, participants are paid $10 per completed survey. This is a fixed amount set by TryMyUi.

5. Burden Hours

The web survey is limited to 20 minutes, with up to 150 participants completing it (i.e., 50 burden hours). The interviewer-administered interview is expected to take up to 60 minutes, with up to 40 participants completing it (i.e., 40 burden hours). Thus, there is a maximum total of 90 burden hours for this study.

5. Data Confidentiality

Lab participants will be informed as to the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of a labor force survey. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. OMB Control Number: 1220-0141. Prior to the interview, participants will be given a consent form to read and sign (Attachment IV).

Participants obtained through TryMyUI sign separate agreements with TryMyUI; no pledge of confidentiality is provided to these respondents. The following notice will be placed on the first page of the survey: “Note: This survey is being administered by Qualtrics.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”

References

Barr, R. and Caplan, L.(1987). “Category Representations and their implications for category structure.” Memory and Cognition, 15(5), 397-418.

Bosley, J., Fricker, S., and Gillman, D. (2012). “Effects of Conceptual Variability Among Response Category Options on Classification of Employment—Implications for Data Quality” Draft manuscript – Paper presented at the Annual Conference of the American Association for Public Opinion Research

Goldwater, M., Markman, A., and Stilwell (2011). “The empirical case for role-governed categories.” Cognition, 118, 359-376.

Krosnick, J. A., Narayan, S., and Smith, W.R. (1996). “Satisficing in Surveys: Initial Evidence.” In M.T. Braverman & J. K. Slater (Eds.), Advances in Survey Research. San Francisco: Jossey-Bass, 29-44

Kurtz, K. and Gentner, D. (2001). “Kinds of kinds: Sources of category coherence.” Proceedings of the 23^rd Annual Conference of the Cognitive Science Society, Mahwah, NJ: Erlbaum, 522-527

Rosch, E. and Mervis, C. (1975). “Family Resemblances: Studies in the internal structure of categories.” Cognitive Psychology, 7, 573-605.

Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization. Hillsdale, NJ: Erlbaum. Reprinted in: Margolis, E. and Laurence, S. (Eds.) (1999). Concepts: Core readings. Cambridge, MA: MIT Press.

Schober, M.F. & Conrad, F.G. (1997). “Does conversational interviewing reduce survey measurement error?” Public Opinion Quarterly, 61, 576-602.

Tourangeau, R., Rips, L., and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge: Cambridge University Press.

Tourangeau, R., Conrad, F., Arens, Z., Fricker, S., Lee, S. and Smith, E. (2006). “Everyday concepts and classification errors: Judgments of disability and residence.” Journal of Official Statistics, 22(3), 385-418.

Attachment I: Study Vignettes (Both In-Person and Web Respondents)

Vignette Type	Vignette Number	Baseline Vignettes
Private – For Profit (P)	1	Jacqueline is an organic gardener who sells her produce through a local food co-operative store. In the winter she sells dried flowers, and kitchen as well as medicinal herbs from the garden. Jackie’s husband and two teenagers help her tend the garden and prepare the dried plants for sale or shipment.
P	2	Joe is a digital imaging specialist for a large group medical practice. He set up and now runs the Picture-Archiving and Communications System (PACS) to store and display X-ray, CAT, and MRI scans of the group practice’s patients. Joe works under a renewable contract with the practice. This is the fifth medical group that Joe has worked for since PACS began to be used to manage electronic medical images.
P	3	Marcella has done graphic design work since she graduated from college. She is currently working on a long-term contract with an advertising agency in a city near her home. She works from her home as well as on-site at the client’s offices. Marcella gets new business by word-of-mouth and by putting ads in the local paper.
P	4	Randall is a PhD chemist who operates an independent testing laboratory. He has special training in methods for detecting and identifying toxic chemicals. This lab is licensed and accredited to analyze chemical samples for a range of clients from law enforcement agencies to agricultural chemical producers. Randall has six full-time lab workers and occasionally pays outside experts for help with difficult or unusual tests.
P	5	Sam is an energy expert who advises restaurant owners about how to become more energy-efficient and “green.” He owns an organic food restaurant himself that has energy-saving kitchen equipment. Sam has also taken some business courses on how to use less energy, create less waste and still be profitable. He works along with a building company that upgrades restaurants with more eco-friendly equipment and energy-saving structural upgrades.

Vignette Type	Vignette Number	Baseline Vignettes
P	6	William is a realtor associated with Modern Family Realtors, a local firm. He meets clients at the Modern Family office where he has a cubicle, a phone and conference room privileges. The company keeps $500 a month for the space and support services, plus 2% of William’s commissions for the month.
Non-Profit (NP)	7	Arnold is an experienced carpenter who trains low-income and unemployed workers in how to “weatherize” buildings. His work is paid for by a foundation that gets its funding from the local utility company and grants from the state department of workforce development. The Town Council turned over an unused building to the foundation to give Arnold a realistic place to teach by assigning “hands-on” projects. He temporarily hires master craftsmen whose names he keeps on file to teach advanced and specialized skills.
NP	8	Nicholas does the maintenance and repair work at three inner-city schools for “at-risk” youth. These students have dropped out of the regular school system but want to get more education or career training. A community-based foundation that is funded by corporate contributions operates these schools. Nicholas is skilled in plumbing, wiring, repairing HVAC equipment and other maintenance tasks.
NP	9	Philip is a trained nurse and a lay missionary for his church. He operates a small mission clinic on a Native American reservation. The clinic provides basic nursing care to reservation people on weekday mornings. In the afternoons, Philip visits homes to check on the health of older residents and other homebound persons and also to perform missionary work. He leads a religious service every Sunday in the tribal hall.

Vignette Type	Vignette Number	Baseline Vignettes
Gov’t (G)	10	Chizuko is a Japanese statistician who now lives in America. She developed a new, powerful computerized method for analyzing economic data from airline companies while she was at a Japanese university. A US government agency wants to use this new statistical method to calculate how competitive US airlines are with foreign airlines in the global travel market. This agency is paying Chizuko for the time she works with its staff to adapt her statistical software, and it provides her with an office at its headquarters and a secretary to help with administrative tasks while she’s working on this project.
G	11	Lawrence is an experienced automotive technician who maintains and repairs the vehicles (police, fire, ambulance, etc.) owned by his town. Lawrence works on the vehicles in a town garage, using his own tools. The city transportation department buys needed parts and supplies from its approved vendors, based on the work orders that Lawrence prepares and submits to the Town Clerk’s office.
G	12	Marvin works full time collecting environmental data (like water samples and air quality measurements). He does this for his state environmental agency and for the federal EPA. He covers a range of locations in the northwest part of the state where he lives. He’s mostly outdoors in the field in his car and on foot. He occasionally attends meetings at both agencies’ offices in the state capital.

Attachment II: Vignette-based Materials

WEB INTRODUCTION

The following survey is being conducted by the Bureau of Labor Statistics.

This voluntary study is being collected under OMB No. 1220-0141 and it is estimated it will take you no longer than 20 minutes to complete.

Note: This survey is being administered by Qualtrics.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.

IN-PERSON INTRODUCTION

Hi! Thank you for coming in today. My name is ( ). I work for ( ) in the Bureau of Labor Statistics. [This is my colleague ( ) from the ( ) office]

I [we] look forward to working with you today.

With your permission, I will be taping this session. This allows me to ensure the accuracy of your answers, and also allows me to focus on you and your responses instead of taking notes. Do you mind if I audiotape?

Before we get started, please read over the consent form. If you agree to participate in the study, print and sign your name as indicated on the bottom of the form. If you do not agree, please let me know now.

Let me explain a bit about what we’ll be doing today. We are going to be discussing an issue that has particular relevance for the Bureau’s labor force survey – the Current Population Survey – namely, how people describe their jobs and those of their household members. The CPS is an ongoing survey that produces the nation’s monthly unemployment rate as well as a wide variety of other demographic statistics, so your input here today will be used to help us improve the quality of those data.

To get us started we’re going to have you read through a set of fictional stories about peoples’ work situations and then ask you to classify the person described in the story into one of several work categories. After you’re done with the stories, you and I will talk a bit about the different types of job categories that the BLS uses and if you had any reactions to the questions we asked.

The purpose of today’s session is to help us find out more about how people, like you, think about these issues. We are not here to evaluate you; we are looking to improve the questions we ask, so there are no right or wrong answers. And, of course, anything you share with us today will be kept completely confidential.

Do you have any questions so far?

Great, let’s get started. [Hand participant the packet of vignette/questions]

WEB VIGNETTE ITEM INSTRUCTIONS

Welcome. Today we will be asking you to think about different kinds of work and workplaces where people can be employed.

On the pages that follow, you will read a brief story that describes some features of work a person might do to earn a living, and then you’ll be asked to classify the person described in the story into one of several work categories. You will see twelve different stories and answer just one question for each one. Then we will present these same twelve stories again and you will be asked a slightly different work classification question about each one. In some cases, your answers may be the same as before, and in some instances you may want to change your answer.

Please read each story carefully and completely before you pick an answer so that you have all the information it contains in your mind. We’d like you to pick your answer pretty quickly, even if you’re not 100% sure. We want to learn more about how people like you might answer questions like these on a real survey and so your own personal answer is the one we want, just based on what each story tells you. There are no “right” or “wrong” answers to any of the questions.

At the end of the session, we will ask you a few follow-up questions about work-related concepts and about your own work experiences.

Please think out loud to the extent possible, so we can tell what you are thinking about when you answer the questions.

IN-PERSON VIGNETTE PACKET INSTRUCTIONS

Welcome. Today we will be asking you to think about different kinds of work and workplaces where people can be employed.

If you have any questions about what we’re asking you to do with the stories, please ask them now before we begin. Our study rules don’t allow us to answer any questions about the actual stories, etc. after you start with the booklet.

THANK YOU FOR YOUR TIME!

[Vignette/question protocol identical for web and in-person respondents]

COW-related Items for Standard Condition: