Attachment H - Certification Cognitive Testing Results

Memo

To:	Dori Allard
From:	Jennifer Edgar, OSMR
cc:	Bill Mockovak, Brandon Kopp
Date:	July 8, 2014
Re:	Results from CPS Certification Cognitive Testing

Background

The goal of this study was to test the addition of three certification/license questions to the CPS. Although extensive pretesting has been done on the core questions, this study aimed to:

1. Look for evidence of context effects; whether the addition of the new questions impacts the existing CPS questions

2. Test the proposed placement of the new certification questions within the existing educational attainment and labor force sections

3. Test minor wording changes to the questions developed by the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA)

4. Determine if introductory or transition language is necessary between the current education questions and the new certification questions

5. Determine if interviewer instructions or help screens are necessary to explain the key concepts

Participants

Recruiting

Since the existing OSMR database does not track certification/license status, new participants were needed for this study. To find participants who were both employed and had a certification or professional license (or had someone in their household with one), recruitment ads were placed in several locations:

• Craigslist: Resumes, Domestic Gigs, Computer Gigs, Labor Gigs

• Capitol Hill businesses: Nail salon, hair salon, barber shop, child care center, University of Phoenix, Architect of the Capitol

• Student and Alumni Associations: American University, Catholic University, University of DC

• Social Networking: Mothers on the Hill list serve, Facebook accounts

• Internal BLS: Sent an email to IT staff , office pantries, health center, fitness center staff

From these sources, 24 participants were recruited: 10 females and 14 males. Participants ranged in age from 23 to 64, with an average of 39.4. Five participants reported that someone else in their household had an active certification or license.

Despite extensive recruiting efforts, the majority of participants were recruited from the BLS IT staff and Craigslist. This led to less diversity of education and certification/license types than we had hoped for (Table 1).

Table 1. Education Level

Number	Highest Degree Obtained
1	Associate’s Degree
11	Bachelor’s Degree
11	Master’s Degree
1	Doctoral Degree

Most (19) participants had one certification or license, four had two and one person had three. The combinations are shown in Table 2.

Table 2. Reported Certifications or Professional Licenses

Reported Certifications or Professional Licenses

American College of Sports Medicine Certification

Brokerage/securities license

Certificate of interpretation; Certification of transliteration

Certified Information System Security Professional (CISSP)

Change Management Advance Practitioner Certification

COMP TIA A+ Certification

Dual certified in special and elementary education

International Board Certified Lactation Consultant

Juris Doctor (JD)

Master Human Capital Strategist

Personal Training Certification

Pilot's license, open water diving certificate, Teaching English as a Second Language certificate

Project Management Institute (PMI) certificate

Project Management Professional Certification (PMP) (5 participants)

Real estate license

Security +, ITIL certification

Survey design and data analysis certificate

VMware Certified Professional (VCP), Microsoft Certified Professional (MCP), Microsoft Certified Technology Specialist (MCTS) certification

All participants were employed, working on average 40.4 hours per week (ranging from 10 to 70), in a variety of job types (Table 3).

Table 3. Reported Occupation

Occupation

Clerk 

Communications Consultant 

Dell Software Associate 

Driver/Transportation 

Economist (4 participants)

Grant Writing 

Human Resources 

Information Specialist Manager 

Information Technology (2 participants)

Insurance Broker 

Lawyer 

Personal Trainer 

Project Manager

Sales 

Scholar 

Software Engineer 

Special Education Teacher

Supervisory Computer Specialist 

Systems Administrator 

Questions Tested

Three new questions were tested in this study. The first two were placed in the demographic section, following the education question and preceding the race and ethnicity questions.

CERT1	(Does/do) (name/you) have a currently active professional certification or a state or industry license?  Do not include business licenses, such as a liquor license or vending license. (Read if necessary:  a professional certification or license shows you are qualified to perform a specific job. Examples include a realtor license, a medical assistant certification, a Teacher’s License or an IT certification. Only include certifications or licenses obtained by an individual.)	Yes No	Universe: All participants
CERT2	Were any of (your/his/her) certifications or licenses issued by the federal, state, or local government?	(1) Yes (2) No	Universe: CERT1 = 1

The third question, to be asked only of respondents who are currently employed, was placed in the workforce section¹, following the question about usual job duties and preceding a question about current school enrollment.

CERT3

Earlier you told me (you/name) had a currently active professional certification or license.  Is (your/his/her) certification or license required for (your/his/her) main job?

(1) Yes (2) No

Universe: All employed & CERT1=1

Results

CERT1 Results

Overall this question worked as intended. Most participants were correctly able to explain what the question meant, such as one participant who said it was asking “if you have a license related to your job, not a degree. Like a project management license. Seems clear enough.” Several participants even noted that the question was asking only for current certifications or licenses with paraphrasing such as “Do you currently have a license that's useable, that you can present to someone.” One participant correctly responded “No” to this question because his certification had recently expired, noting “I don’t have any [certifications] that are currently active.”

Although participants didn’t show uncertainty when answering the questions, there were some yes responses that seemed questionable and likely to be certificates, although the participant considered them to be certifications. Participant inability to differentiate between a certificate and certification was an issue identified throughout the GEMEnA testing, and is not unique to the CPS implementation. The cases found during this testing were:

• Change Management Advance Practitioner Certification, from Georgetown University

• Survey Design and Data Analysis Certificate, from George Washington University

Participants seemed to focus on the part of the question relevant to them and had no issues with that part of the question. A few did express confusion about the part of the question that did not apply to them, though this didn’t impact their answers: “Not sure what "state issued license" means, but because I'm just not familiar with those. Professional certifications on the IT side are straightforward, there are a lot there.” This issue did not impact their ability to answer the question correctly.

Overall, we concluded this question worked as well as found in the GEMEnA testing, the placement in the CPS survey introduced no new issues, and it should be implemented as tested.

CERT2 Results

Of the three questions, CERT2 was found to have the most issues. However, the key issue identified; that is, participant uncertainty about the meaning of the question leading to inaccurate answers, was confirmed to have been identified in GEMEnA testing as well. Of all the participants, a majority answered the question correctly and were able to correctly explain the meaning of the question: “is it from the government and not a private company.”

A few participants incorrectly interpreted the question, or incorrectly applied it to their situation. These cases included participants who said yes because:

• the certificate was awarded by George Washington University, which is accredited by the Department of Education;

• the state (Maryland Transit Authority) paid for her brother’s certification

• the license is recognized across different states

• since the certification provided by the American College of Sports Medicine (ACSM) was recognized across all the states, he thought it must be federal government

OSMR and CPS staff reviewed these issues and debated changing the wording and conducting additional testing, but since almost all participants were able to correctly answer the question, and the GEMEnA group has also recommended similar wording based on their testing, the decision was made to recommend the question be implemented as tested.

CERT3 Results

CERT3 was found to work without any issue for the participants. Participants were able to explain what the question was asking: “Certification can help me in my job, but if I didn't have it I wouldn’t lose my job or anything.”

All participants appeared to correctly answer the question, though there is no way of verifying the information. The explanations given for who requires them to have a certification or license, and how they know seem reasonable: “Certification is required by my job, to be hired.” Six participants said that their certification or license was required for their job, and their certification and their job were:

Table 4. Occupation and Certification / License

Occupation	Certification or License
Insurance Broker	Brokerage/securities license
Lawyer	JD
Personal Trainer	American College of Sports Medicine certification
Real Estate License (2 participants)	Realtor
Special Education Teacher	Dual certified in special and elementary education

Since the interviews took around 15 minutes, the probes about the CERT3 introduction (“Earlier you told me…”) were not particularly effective. Participants chuckled or otherwise indicated the answer was obvious, and no one had any issue with the question’s reference back to their previous CERT1 response.

Overall, we concluded that this question worked well, the placement in the CPS survey introduced no new issues, and it should be implemented as tested.

Evidence of Context Effects

Since CERT1 and CERT2 were placed among education questions, participants were asked how certifications and licenses related to education. There was a range of responses, with about half of the participants saying that certifications/licenses are a type of education and half saying they are not. Those who believed it was education cited the training and testing required to obtain the certification or license. Those who said it was not education said that a certification or license is related to your job and in your professional field while your education may or may not be.

Despite the inconsistency in thinking of the relationship between certifications, licenses and education, CERT1 and CERT2 follow the ‘highest education’ question, so they cannot impact the answers to that existing question. Also, no participants reported their certification or license in response to the ‘highest education’ question. Therefore, there are no expected context effects related to CERT1 and CERT2.

CERT3 is placed in the workforce section, immediately preceding the existing ‘currently enrolled’ question. This placement creates the potential for context effects, if respondents include certification or license work in the ‘currently enrolled’ question. While we didn’t have any participants who were actively pursuing a certification or license, we did ask them how they would handle the situation hypothetically.

Most participants said no, they would not have answered yes to the ‘currently enrolled’ question if they were actively pursuing a new certification or license. The few that said they would, noted it was because they got their certification or license through a university or community college, and so they would have been enrolled in these institutions: “Since my program was at Georgetown, we were registered with the university and I would have said yes.” Although these answers suggest that the participant may not have actually have a true certification or license (likely a certificate), they do not indicate an incorrect answer to the ‘currently enrolled’ question would be given. Therefore, we conclude that no major context effects are expected as a result of asking CERT3.

Proposed Placement

The second aim of this study was to test the proposed placement of the new certification questions within the existing educational attainment and labor force sections. Interviewers found that the flow of the questions worked, and participants reported that the inclusion of certification and license questions among education questions (CERT1 and CERT2) was logical. CERT3 also performed well in its proposed location in the workforce section, and participants were able to easily connect the question to their main job. We recommend that the proposed placement be used in implementation.

Question Wording

CPS and OSMR staff made minor wording changes to the questions used in the GEMEnA testing. These changes (e.g., “Were any of…” and “Earlier you told me…”) were necessary based on the context of the questions within the CPS, and the limited number of questions that we were able to add. The changes were not found to introduce any new issues previously unidentified by GEMEnA testing, and were found to work well. We recommend that the tested wording be implemented.

Transition Language

Since the new certification and license questions are being placed within an existing survey, we were concerned that language may be needed to introduce the new topic to respondents (e.g., “Now I’m going to ask about…”). We tested the questions without any transition language to minimize the number of words used in the interview. No evidence of confusion was found, and as noted previously, participants felt the new questions fit well in the existing sections. We recommend implementing the questions as tested, without the addition of transition language.

Interviewer Instructions

When survey changes are made, interviewers need to be trained on the objectives of the new questions, and on how to handle any respondent questions or errors. Based on our testing, we recommend that interviewer training be developed to address how to record answers for the main educational attainment question if respondents report certifications instead of education (this is an existing CPS issue), and how to handle questions about CERT2 if the respondent shows uncertainty or confusion about the question meaning.

Conclusion and Recommendation

Based on this cognitive interviewing study, we found all three questions to work reasonably well, did not introduce any problems to existing CPS questions, and recommend that they be implemented as tested.

1 In addition to employed respondents, a small number of unemployed respondents who previously held jobs will also be asked this question. Since this subgroup (previously employed people with a certification or license who have done active job search in the past 4 weeks) was difficult to recruit, this implementation of the question was not tested.