Note to Reviewer - CPS Certification Study

April 24, 2014

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for CPS Certification/Licensure Cognitive Testing

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, I am submitting a brief description of the study.

The total estimated respondent burden hours for this study are 27 hours.

If there are any questions regarding this project, please direct them to Jennifer Edgar (202-691-7528).

1. Introduction and Purpose

The Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) has developed and validated national measures of the participation in and credentialing of education and training for work. The group has also built government-wide consensus for the adoption of these measures in key federal data collections. More specifically, GEMEnA is engaged in a rigorous process of survey item development to measure the core concepts of 1) the attainment of non-degree credentials, including industry-recognized certifications, occupational licenses, and educational certificates, and 2) enrollment in education and training, outside of traditional credit-bearing college coursework, that prepares people for work. To date, GEMEnA has developed a core set of validated survey items to measure the prevalence and key characteristics of certifications and licenses, and efforts are underway to deploy these items within the federal statistical system.

Current Population Survey (CPS) staff has been involved in the GEMEnA group and are looking to add a subset of the core certification and license items to the current CPS Educational Attainment Items, to be asked in months 1 and 5 of the 8 month panel. If the questions are found to be successful in this context, and CPS stakeholders approve, three little-used educational attainment questions will be removed (as shown in Appendix B) and the tested new questions will be added to the production CPS.

Although extensive pretesting has been done on the core questions, this study seeks to:

1. Look for evidence of context effects; whether the addition of the new questions impact the existing CPS questions

2. Test the proposed placement of the new certification questions within the existing educational attainment and labor force sections

3. Test minor wording changes to the questions developed by GEMEnA

4. Determine if introductory or transition language is necessary between the current education questions to the new certification questions

5. Determine what interviewer instructions or help screens are necessary to explain the key concepts

2. Research Design

The Office of Survey Methods Research (OSMR) at BLS will conduct 25 cognitive interviews. Most participants will be screened to be employed, have a recent or current certification, or to have proxy knowledge of one (Attachment A). A control group of participants without a certification or license will also be included. Participants with certifications/licenses will be scheduled first. It is expected that some screened in participants will not actually have a true certification or license, and these will form the control group. If the number of non-certification holders is fewer than 5 when the last 10 participants are being scheduled, we will schedule those explicitly.

Interviews will be conducted in-person in the OSMR cognitive lab and will be administered by OSMR staff. Each session will follow the cognitive interview protocol outlined in Attachment B. Sessions will be audio taped with the consent of the participant.

3. Procedures

Interviewers will follow the protocol outlined in Attachment B. After introductions, gaining informed consent, and a short explanation of the procedures for the cognitive interview, the participant will be asked to identify household members and specify their employment and certification/license status. Based on the responses to those preliminary questions, up to three household members will be selected for the CPS questions (to ensure there is time to conduct a full debriefing).

The interviewer will then administer the survey. The protocol will follow the designed skip patterns to mirror the planned implementation, and will be repeated for up to three household members, starting with the participant’s self-report. Following the administration of the survey, the interviewer will debrief the participant. The debriefing will be semi-structured and the interviewer will probe as necessary to obtain additional information.

4. Participants

Up to 25 participants will be interviewed for this cognitive interviewing study. Participants will be recruited by asking groups with members who have certifications to distribute or post fliers. Additional recruitment efforts will include targeted advertisement in newspapers and online bulletin boards, as well as contacting professional organizations of industries where certifications are common (e.g., health care, information technology, and teachers). Most participants will be screened to ensure either they or a member of their household has a current certification or license. A draft flier is included as Attachment C. A small group (5 to 7) of participants will be selected from the existing OSMR database, without any screening. This will allow testing of the questions among people who do not have these credentials and may not be familiar with the terms, as well as exploration of proxy reporting.

5. Burden Hours

We anticipate that each session will average 45 minutes. An additional 10 minutes is estimated for recruitment and screening of 50 people to obtain the 25 participants. Total burden hours is 27 hours.

6. Payment

For this study, we will be reimbursing participants $40 for their time.

7. Data Confidentiality

Participants will be informed of the voluntary nature of the study. Participants also will be informed that the study will be used for internal purposes to improve the design of the Current Population Survey. Participants will be given a consent form to read and sign (Appendix D) prior to beginning the test session. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

8. Attachments

Attachment A:	Screening Questions
Attachment B:	Cognitive Interviewing Protocol
Attachment C:	Recruitment Flier
Attachment D:	Consent Agreement Form and Privacy Act Statement

Attachment A: Screening Questions

Introduce the study

Explain we have to ask some questions to determine if they are eligible to participate

Ask each question, recording answers

Name
How did you hear about the study
Why do you think you’re eligible
For each member of the household, including caller, fill in below table:
First name
Do they have a certification or license?
Are they employed
If employed, how many jobs do they currently have?
If employed, what do they do?
General availability for interviews
Day of the week
Time of day
Best number to reach you
Best email address to reach you

Name	Certification/License? Y/N	Employed? Y/N	How many jobs?	Occupation

Explain we will call them if they’re eligible when interviews start, interviews are expected to run in May and June.

Attachment B: Cognitive Interviewing Protocol

• Hi! Thank you for coming in today.

• I am …... [This is my colleague ( ) who will be taking notes for us today]

• Have you participated in any of our studies before?

• Consent Form/Permission to audiotape

• Explanation:

  • We are going to be working with some questions from the Current Population Survey, an ongoing survey that provides a continuous flow of information about the characteristics of the American workforce.

  • What we are going to do today is go through a few questions from the survey and get your reactions to them.

  • The purpose of today’s session is to help us find out more about how people respond to these questions. Basically, we’re trying to find out what you think a question is asking and how hard it is to answer. We are not here to evaluate you, we are looking to improve the questions, so there are no wrong answers. All the information you give us will be kept completely confidential, and will be used to improve the survey questions.

• Any questions before we begin?

Question Number	Questions	Response Options	Who Receives Question
Roster1	Please tell me the names and ages of everyone who lives in your household	Open ended	Everyone
Roster 2	Is (name) employed?	Yes No	Everyone
Demographic Questions, ask as a block for each HH member
EDUCA	What is the highest level of school (name/you) (has/have) completed or the highest degree (name/you) (has/have) received?	(31) Less than 1st grade (32) 1st, 2nd, 3rd or 4th grade (For example: BA, AB, BS) (33) 5th or 6th grade (34) 7th or 8th grade (35) 9th grade (36) 10th grade (37) 11th grade (38) 12th grade NO DIPLOMA (39) HIGH SCHOOL GRADUATE- high school DIPLOMA or the equivalent (For example: GED) (40) Some college but no degree (41) Associate degree in college - Occupational/vocational program (42) Associate degree in college -- Academic program (43) Bachelor's degree (44) Master's degree (For example: MA, MS, MEng, MEd, MSW, MBA) (45) Professional School Degree (For example: MD,DDS,DVM,LLB,JD) (46) Doctorate degree (For example: PhD, EdD)	Everyone
DIPGED	People can get a High-School diploma in a variety of ways, such as graduating from High School or by getting a GED or other equivalent. How did (you/name) get (your/his/her) High-School diploma?	Graduation from High School GED or other equivalent	39 in EDUCA
HGCINO	What was the highest grade of regular school (you/name) completed before receiving (your/his/her) GED?	(1) Less than 1st grade (2) 1st, 2nd, 3rd, or 4th grade (3) 5th or 6th grade (4) 7th or 8th grade (5) 9th grade (6) 10th grade (7) 11th grade (8) 12th grade NO DIPLOMA	2 in DIPGED
CYC	(Including any time that may have been spent getting an Associate’s Degree,) How many years of college CREDIT (have/has) (name/you) completed? (Have/has) (name/you) completed	(1) Less than 1 year (include 0 years completed)? (2) The first, or FRESHMAN year? (3) The second, or SOPHOMORE year? (4) The third, or JUNIOR year? (5) Four or more years?	EDUCA=40, 41 or 42
EDUCA7	Since completing (your/his/her) Bachelor's degree (have/has) (you/name) taken any GRADUATE or PROFESSIONAL school courses for credit?	(1) Yes (2) No	Proposed for removal
EDUCA8	Did (you/name) complete SIX or MORE graduate or professional school courses?	(1) Yes (2) No	Proposed for removal
EDUCA9	Was (your/name's) Master's Degree program a 1-year, 2-year, or 3-year program?	(1) 1-year program (2) 2-year program (3) 3-year program (or longer)	Proposed for removal
CERT1	(Does/do) (name/you) have a currently active professional certification or a state or industry license?  Do not include business licenses, such as a liquor license or vending license. (Read if necessary:  a professional certification or license shows you are qualified to perform a specific job. Examples include a realtor license, a medical assistant certification, a Teacher’s License or an IT certification. Only include certifications or licenses obtained by an individual.)	(1) Yes (2) No	Everyone
CERT2	Were any of (your/his/her) certifications or licenses issued by the federal, state, or local government?	(1) Yes (2) No	CERT1 = 1
HSPNON	(Are / Is) (name/you) of Hispanic, Latino, or Spanish origin?	(1) Yes (2) No	Everyone
ORISPN	(Are / Is) (name/you) Mexican, Mexican American, or Chicano; Puerto Rican; Cuban, Cuban American, or of another Hispanic, Latino, or Spanish origin; for example, Argentinean, Colombian, Dominican, Nicaraguan, Salvadoran, Spaniard, and so on? If multiple answer, Probe: Which group (do/does) (you/he/she) most closely identify with?	Mexican Mexican American Chicano Puerto Rican Cuban Cuban-American Other	Everyone
S_OROTSP	What is that origin?		ORISPN=7
RACE	I am going to read you a list of race categories. You may choose one or more races. For this survey, Hispanic origin is not a race. Are you: White; Black or African American; American Indian or Alaska Native; Asian; OR Native Hawaiian or Other Pacific Islander? Do not probe unless response is Hispanic or a Hispanic origin. Enter all that apply, separate with commas	White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander Other - DO NOT READ	Everyone
RACEAS	You may choose one or more Asian groups. (Are / Is) (you/name) Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, or another Asian group, for example, Hmong, Laotian, Thai, Pakistani, Cambodian, and so on? Read each item. Enter all that apply, separate with commas	Asian Indian Chinese Filipino Japanese Korean Vietnamese Other Asian	RACE = 4
RACEPI	You may choose one or more Pacific Islander groups. (Are / Is) (you/name) Native Hawaiian; Guamanian or Chamorro; Samoan; or another Pacific Islander group, for example, Fijian, Tongan, and so on? Read each item. Enter all that apply, separate with commas	Native Hawaiian Guamanian or Chamorro Samoan Other Pacific Islander	RACE = 5
Labor Force Questions, for up to 3 HH employed members, identified during screening
LABOR	Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___		Everyone
PUWK	LAST WEEK, did you do ANY work for pay?	Yes, go to PUMJ No, go to PUABS Don't Know Refused	Everyone
PUABS	LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.	Yes No Don't Know Refused	Employed, PUWK = 1
PUMJ	LAST WEEK, did you have more than one job (or business), including part-time, evening or weekend work?	Yes, No, Don't Know Refused	Employed, PUABS = 1 PUMJ = 1
PUHRUSL1	How many hours per week do you USUALLY work at your (main) job? (If PUMJ is "yes") By main job we mean the one at which you USUALLY work the most hours.	If pumj=1 ask puhrusl2 else go to puio1int	All employed, fill based on more than one job, PUMJ=1
PUHRUSL2	How many hours per week do you USUALLY work at your other job(s)?		All employed, fill based on more than one job, PUMJ=1
PUIO1INT	Now I have a few questions about the (main) job - (a) (at which you worked LAST WEEK.) (b) (from which you were absent LAST WEEK.) (Was.../Were you) employed by government, by a private company, a non-profit organization, or (was.../were you) self employed (or working in a family business)?	Government Private company Non-profit organization incl. tax exempt and charitable organizations Self-employed Working in family business	All employed, fill based on more than one job, PUMJ=1
PUIO1INC	What is the name of the (company) (non-profit organization) (business) for which ... (works) (at your/his/her Main job)?	Open-ended	All employed, fill based on more than one job, PUMJ=1
PUIO2INC	"What kind of business or industry is this?" READ IF NECESSARY What do they make or do where you work?	Open-ended	All employed
PUIO1OCC	What kind of work (do) you do, that is, what (is) your occupation? (For example: plumber, typist, farmer)	Open-ended	All employed
PUIO1DT	What (are) your usual activities or duties at this job? (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)	Open-ended	All employed
CERT3	Earlier you told me (you/name) had a currently active professional certification or license.  Is (your/his/her) certification or license required for (your/his/her) main job?	(1) Yes (2) No	All employed CERT1=1
PUSCHENR	LAST WEEK, were you enrolled in a high school, college, or university? (Mark "Yes" if currently on holiday or seasonal vacation. Mark "No" for summer vacation.)	(1) Yes (2) No	Everyone
SCHLVL	(if necessary) Would that be high school, college, or university?	High school College or University Don't Know Refused	Everyone

Debriefing

I’d like to begin by asking you about your general reactions to the survey.

1. What was it like for you to participate in this survey?

1. Were there any questions you found difficult or confusing?

2. What were the main topics that survey asked about?

CERT1. Now I’d like to talk about some of the specific questions you answered. The first one asked about

CERT1

(Does/do) (name/you) have a currently active professional certification or a state or industry license? Do not include business licenses, such as liquor license or vending license. (Read if necessary, if participant shows any confusion or hesitation before answering:  a professional certification or license shows you are qualified to perform a specific job. Examples include a realtor license, a medical assistance certification, a Teacher’s License or an IT certification. Only include certifications or licenses obtained by an individual.)

(1) Yes (2) No

Everyone

1. What did this question mean to you?

2. (if 1) You answered “Yes” to this question. Can you tell me about that? What type of certification or license do you have?

3. (if 2) You answered “No” to this question. Have you ever had a certification or license?

   1. (if yes) Can you tell me about that? What type is it?

4. What types of things did you think of when you heard “professional certification or a state or industry license?”

5. The questions that came before the question about certifications and licenses asked about how much education you (and others in your household had). Do you think the questions about certification or license fit in with those questions?

   1. Do you consider a certification or license a type of education?

   2. (if they are a type of education) Are there any types of certifications or licenses that are not educational?

   3. (if they are not a type of education) Are there some types of certifications and licenses that you would include when answering the question about what is the highest level of education you received?

   4. (if something else) what term would you use to classify certification or licenses?

6. There were also some questions about the highest level of education you’ve received, do you consider your certification/license as part of your education?

   1. (if no) How are they different? Is there another category you would put certifications in?

   2. (if yes) Do you consider your certification as your highest level of education? Does it go above your high school/college/graduate degree?

   3. Did the questions make sense to you, to ask about education and then certifications and licenses?

FOR PROXY

7. (if yes for proxy) You said that (name) has a license or certification. What type?

   1. Is it currently active? How do you know its current status?

8. (if no for proxy) You said that (name) didn’t have a license or certification. Has (name) ever had a certification or license?

   1. (if yes) Can you tell me about that? What type is it? Is it currently active?

9. How sure are you of your answers?

   1. Do you feel like you have a strong knowledge of the certifications/licenses of other people in your household?

CERT2 (if received). Another question asked

CERT2

Were any of (your/his/her) certifications or licenses issued by the federal, state or local government?

(1) Yes (2) No

CERT1 = 1

NOTE: be sure to probe able all certifications and licenses if the P has more than one

1. What did this question mean to you?

2. (if 1) You answered “Yes” to this question. Who awarded your certificate or license?

3. (if 2) You answered “No” to this question. Who awarded your certificate or license?

   1. How sure are you of the __ that awarded your certificate or license?

4. (if only one) The question asked about ANY certifications/licenses. Did you think this question applied to you, even though you only had one certification /license ?

5. (if more than one) The question asked about “Any of your certifications…” Which certifications/licenses were you thinking about?

FOR PROXY

6. You said that (name’s) license/certification was/was not awarded by the federal, state or local government. Who awarded it?

   1. How sure are you of the __ that awarded your certificate or license?

7. How sure are you of your answers?

CERT3 (if received) After asking a series of questions about your job, we asked:

CERT3

Earlier you told me (you/name) had a currently active professional certification or license.  Is (your/his/her) certification or license required for (your/his/her) main job?

(1) Yes (2) No

CERT1=1

NOTE: be sure to probe able all certifications and licenses if the P has more than one

NOTE: be sure to probe able all jobs if the P has more than one

1. What did this question mean to you?

2. What was I referring to when I said “earlier you told me you had a currently active certification or license?”

   1. When did you give that information?

   2. Was it clear to you what the question was referring to?

3. (if 1) You answered “Yes” to this question. Can you explain that to me?

   1. Who requires that you have the certification or license?

   2. How do you know that it is required?

   3. (if more than one job) What job were you thinking about in this question?

   4. (if more than one job) (Does/Do) your other job(s) require a certification or license?

4. (if 2) You answered “No” to this question. Can you explain that to me?

   1. (if more than one job) What job were you thinking about in this question?

   2. (if more than one job) (Does/Do) your other job(s) require a certification or license?

FOR PROXY

1. What was I referring to when I said “earlier you told me [NAME] had a currently active certification or license”?

   1. When did you give that information?

   2. Was it clear to you what the question was referring to?

2. (if proxy was yes) You said that (name’s) certification/license is required for their job. Can you explain that to me?

   1. Who requires that they have the certification or license?

   2. How do you know that it is required?

3. (if proxy was no) You said that (name’s) certification/license is not required for their job. Can you explain that to me?

   1. Are there any types of certifications or licenses required for your job?

   2. Can you think of any types of jobs that require a certification or license?

4. (if more than one job) What job were you thinking about in this question?

5. How sure are you of your answers?

The last set of questions asked about current school enrollment:

PUSCHENR

LAST WEEK, were you enrolled in a high school, college, or university? (Mark "Yes" if currently on holiday or seasonal vacation. Mark "No" for summer vacation.)

(1) Yes (2) No

Everyone

1. There were also questions about school enrollment, if you are currently enrolled in school. What type of school did you think about for those questions?

2. If you were currently working on getting a new certification or license, would you say you were currently enrolled in school?

3. What type of schools do you think should be included in this question?

Closing

• (If participant exhibited uncertainty for any of the educational attainment/school enrollment questions, probe those)

• Thank you for your help with this, do you have anything else to add?

Attachment C: Recruiting Flier

Do you have a certification or professional license?

The Bureau of Labor Statistics is looking for people with certifications or licenses to participate in an interview.

You’re also eligible if anyone who lives in your household has a certification or license.

You will receive $40 for an interview.

We are conveniently located on the Red Line, next to Union Station.

Call 202-691-xxxx or email [email protected] for more information.

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Bureau of Labor Statistics [Address] [Phone] [E-mail]

Attachment D: Consent agreement form and Privacy Act statement

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: [02-28-2015]

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

23