Note to Reviewer - Email Study

Note to Reviewer - Email Design Study final.docx

Cognitive and Psychological Research

Note to Reviewer - Email Study

OMB: 1220-0141

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1220-0141 can be found here:

Document [docx]

Download: docx | pdf

June 21, 2017

NOTE TO THE

REVIEWER OF:

OMB CLEARANCE 1220-0141

“Cognitive and Psychological Research”

FROM:

Jennifer Edgar Research Statistician

Office of Survey Methods Research

SUBJECT:

Submission of Materials for the Testing of Email Designs

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 108.

If there are any questions regarding this project, please contact Jennifer Edgar at

202-691-7528.

Introduction and Purpose

In the light of budget constraints and declining response rates, several BLS programs have been exploring how to improve their email notifications to increase the attention respondents pay to the emails. While some guidelines exist about how to word or format these types of emails, there are not clear guidelines as to what format of email best conveys the intended message to respondents, catching and maintaining their attention. The goal of this study is to evaluate the impact of email formatting on participant comprehension and reactions.

Research Design

Information will be collected from both online participants, through the MTurk platform, as well as lab participants who will provide eye tracking data. In both collections, the same study design will be used.

This study will compare three email formats, each of which has been used within BLS. Participants will be randomly assigned to the format, asked to review the email as if they received it and were being asked to participate in a BLS survey. The three conditions (shown in Appendix A) will be:

Paragraphs: the common format in most BLS programs, with main points communicated in full sentences and paragraphs.
Bullets: a streamlined content, with main points identified in bulleted lists.
Graphics: a streamlined content, with main points identified in bulleted lists (same format as the bulleted design), but with graphics to illustrate the BLS logo and other main points

After viewing the email all participants will be asked the same debriefing questions (shown in Appendix B), the results from which will be compared across groups. The debriefing questions will ask participants to list the main points they recall from the email. Then we will show participants a list of concepts and phrases, some of which were included in the email and some were not, and asked to mark off those they recognize. Following the recall and recognition tasks, all participants will be asked questions about their response to the email and whether they would or would not be inclined to complete the survey based on the email.

Both the online and eye tracking will complete the tasks online, the only difference is that the eye tracking participants will be monitored using OSMR’s eye tracker as they look at the email. The rest of the data will be collected without any eye tracking monitoring.

3. Participants

Up to 600 Amazon Mechanical Turk participants will participate. This sample size was determined by similar online evaluation studies, and allows for enough analytical power for statistical comparison between groups, as the differences are expected to be small. Participants will be given a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers.

Additionally, up to 30 participants will be recruited from the OSMR database to participate in the eye tacking study. Based on past studies with the eye tracker, we anticipate having to screen 40 participants to find 30 who are good candidates for the study.

Burden Hours

A total of 108 burden hours will be used for this study:

	# of Participants Screened	Minutes per participant for Screening	Total Screening Burden Minutes	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden Minutes	Total Burden Hours (Screening + Collection)
Mechanical Turk	0	0	0	600	10	6000	100
Eye Tracking	40	5	200	30	10	300	8
							108 hours

5. Payment to Respondents

Mechanical Turk participants will receive $1.00 for participating in this survey, a typical rate for similar tasks. A total of 600 participants will participate. The costs for the main survey total $840.00 (1.00 x 600, plus a $240 commission fee to the company).

Lab participants will be compensated $40 each.

6. Data Confidentiality

Recruiting of online participants will be handled by MTurk, with the ‘task’ posted on the worker’s platform. Participants will be informed that the study is about their perceptions of different types of questions. Once participants are recruited into the study, they will be sent a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Participants in this study will not be asked to provide any personally identifiable information. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality (As shown in Appendix D.)

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2017). You are not required to respond to this collection unless it displays a currently valid OMB control number. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Lab participants will sign the standard consent form, included as attachment C.

Appendix A – Email Formats

Paragraphs

From: Occupational Employment Statistics Program <[email protected]>
Subject: Occupational Employment Report Notice

Dear Employer,

You are receiving this email because you indicated that you would prefer to be contacted via email, or because you have previously responded to the Occupational Employment Statistics Report via email.

The Maryland Department of Labor, in cooperation with the U.S. Bureau of Labor Statistics (BLS), needs your help to collect workforce information for Maryland. Your participation is critical, since businesses like yours are our only source of information. We cannot gain insight into the economy without your help. We statistically sample from businesses around the country, contacting a wide range of company types and sizes. Your company was selected because it represents similar businesses. Without your cooperation, we will have a gap in our understanding of businesses like yours.

In most cases, all we need from you are the job titles and wages for the employees at the location specified below who worked during the pay period that included May 12, 2017. Including department is optional, but can help reduce the need for follow up phone calls. Please consider reporting by emailing [email protected] or online. To save government resources, consider reporting online or by emailing [email protected].

Online instructions:

Go to https://idcfoes.bls.gov. Enter your IDCF number, 1234-568, complete the security check, and click "I Accept." Follow the instructions on screen and refer to the information below to complete the report. When you are finished, you will receive an email confirming your submission. Please save the confirmation email in case we have questions about your data.

No password is necessary for the OES Report.

A sample of the information required is located at https://www.bls.gov/respondents/oes/instructions.htm.

If you are reporting for a school, please visit https://www.bls.gov/respondents/oes/instructions.htm#school for additional instructions.

Please complete your OES Report within 14 days. If you have problems or questions, please contact us at [email protected] or 301-555-5555. Additional information regarding this report can be found at https://www.bls.gov/respondents/oes.

Thank you,

The Maryland Department of Labor and the US Bureau of Labor Statistics

The Maryland Department of Labor and the Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent, except in the case of State and local governments. The BLS publishes statistical tabulations from this report that may reveal the information reported by State and local governments. Upon request, however, the BLS will hold the information provided by State and local governments on this report in confidence. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

We estimate that the time required to complete this report will vary from 10 minutes to 2 hours, depending on factors such as the size of the establishment. This includes time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this information. If you have any comments regarding these estimates or any other aspects of this report, including suggestions for reducing this burden, send them to the U.S. Bureau of Labor Statistics, Division of Occupational Employment Statistics (1220-0042), 2 Massachusetts Avenue NE, Suite 2135, Washington, DC 20212. This report is authorized by law, 29 U.S.C. §2. Your voluntary cooperation is needed to make the results of this report comprehensive, accurate, and timely. You do not have to complete this questionnaire if it does not display a currently valid OMB control number. Form Approved, O.M.B. No. 1220-0042.

Bureau of Labor Statistics - Statement to Respondent on the Use of Electronic Data Transmission

As a participant in a Bureau of Labor Statistics (BLS) statistical survey, you should be aware that use of electronic transmittal methods in reporting data to BLS involves certain inherent risks to the confidentiality of those data. Further, you should be aware that responsible electronic transmittal practices employed by the BLS cannot completely eliminate those risks. BLS is committed to the responsible treatment of confidential information and takes rigorous security measures to protect confidential information in its possession.

Bullets

From: Occupational Employment Statistics Program <[email protected]>
Subject: Occupational Employment Report Notice

Dear Employer,

Thank you for agreeing to be contacted by email regarding the Occupational Employment Statistics Report, or for providing your information via email in the past!

The Maryland Department of Labor in cooperation with the U.S. Bureau of Labor Statistics (BLS) needs your help! The Occupational Employment Statistics Report collects workforce information for Maryland.

Your participation is critical:

Businesses like yours are our only source of information; we cannot gain insight into the economy without your help.
We statistically sample from businesses around the country, contacting a wide range of company types and sizes.
Your company was selected because it represents similar businesses in Maryland.
Without your cooperation, we will have a gap in our understanding of businesses like yours.
Additional information regarding this report can be found at https://www.bls.gov/respondents/oes.

Your task is straightforward:

In most cases, all we need from you are the job titles and wages for the employees at the location specified below who worked during the pay period that included May 12, 2017.
Including department is optional, but can help reduce the need for follow up phone calls.
To save government resources, consider reporting online or by emailing [email protected].

Instructions:

Within the next 14 days, Go to https://idcfoes.bls.gov.
Enter your IDCF number, 1234-568, complete the security check, and click "I Accept."
1. No password is necessary for the OES report
Follow the instructions on screen and refer to the information below to complete the report.
You can find information about reporting electronically at https://www.bls.gov/respondents/oes/instructions.htm
You will receive a confirmation email when you’re finished, please save this email in case we have questions about your data.

If you have problems or questions, please contact us at [email protected] or 301-555-5555. Additional information regarding this report can be found at https://www.bls.gov/respondents/oes.

Thank you,

The Maryland Department of Labor and the US Bureau of Labor Statistics

Bureau of Labor Statistics - Statement to Respondent on the Use of Electronic Data Transmission

Graphics

From: Occupational Employment Statistics Program <[email protected]>
To: [resp_email]
Subject: Occupational Employment Report Notice

Dear Employer,

Thank you for agreeing to be contacted by email regarding the Occupational Employment Statistics Report, or for providing your information via email in the past!

Your participation is critical:

Businesses like yours are our only source of information; we cannot gain insight into the economy without your help.
We statistically sample from businesses around the country, contacting a wide range of company types and sizes.
Your company was selected because it represents similar businesses in Maryland.
Without your cooperation, we will have a gap in our understanding of businesses like yours.
Additional information regarding this report can be found at https://www.bls.gov/respondents/oes.

Y our task is straightforward:

In most cases, all we need from you are the job titles and wages for the employees at the location specified below who worked during the pay period that included May 12, 2017.
Including department is optional, but can help reduce the need for follow up phone calls.
To save government resources, report online or by emailing [email protected].

Shape2

Within the next 14 days, Go to https://idcfoes.bls.gov.
Enter your IDCF number, 1234-568, complete the security check, and click "I Accept."
Follow the instructions on screen and refer to the information below to complete the report.
You will receive a confirmation email when you’re finished, please save this email in case we have questions about your data.

I nstructions:

Y ou can find information about reporting electronically at https://www.bls.gov/respondents/oes/instructions.htm

If you have problems or questions, please contact us at [email protected] or 301-555-5555. Additional information regarding this report can be found at https://www.bls.gov/respondents/oes.

Thank you!

The Maryland Department of Labor and the US Bureau of Labor Statistics

Bureau of Labor Statistics - Statement to Respondent on the Use of Electronic Data Transmission

Appendix B – Debriefing Questions

Thanks for reviewing the emails! Please answer the following questions, without looking back at the previous screen, to give us your feedback on them.

Please list as many of the main points as you can remember from the email.

[page break]

Which of these phrases were included in the email?

	In the email	Not in the email	Not sure
May 12^th
Mandatory
Voluntary
Confidential
Statistical Sample
Represent businesses like yours
Occupational Employment Statistics Report
14 days
31 days
Instructions
Report online
Report by email
Report by fax
Report by mail
Save government resources
Job titles
Wages
State of Maryland
Department of Labor
Bureau of Labor Statistics
Department of Commerce
IRS
Cybersecurity

[page break]

Just a few more questions. As you answer these, remember, these emails would be sent to you at your place of work. To refresh your memory, here’s the email again.

Based on this email only (excluding your or your company’s attitudes or policies towards surveys), how likely would you be to click the link and complete the survey?
1. Very Likely
2. Somewhat Likely
3. Somewhat Unlikely
4. Very Unlikely
5. Comments:

Please rate the email on the following factors:

	Very	Somewhat	Not Very	Not At All
Engaging
Interesting
Professional
Motivating

Comments:

How similar is this email to others that you receive at work?
1. Very Similar
2. Somewhat Similar
3. Somewhat Dissimilar
4. Very Dissimilar
5. Comments:

Here are two other versions of the email that we’re considering. Which of the three do you prefer [screen shots of all three, with labels]:
1. Paragraphs
2. Formatted
3. Graphic
4. Comments:
What recommendations do you have to improve these emails?

[page break]

To allow us to understand who is participating in our research, please answer the following questions about yourself:

What is your sex?
- Male
- Female
What is your age?

_____
Prefer not to say

Are you of Hispanic, Latino or Spanish origin?

Yes
No
Don’t know
Prefer not to say

What is your race? (You may select more than one)

White
Black or African American
American Indian or Alaska Native
Asian
Native Hawaiian or Other Pacific Islander

What is the highest level of education you’ve completed?

Less than high school
High school diploma or GED
Some college
Associate degree
Bachelor’s degree
Graduate school degree
Prefer not to say

Are you currently employed:

Full time (go to 7)
Part time (go to 7)
Unemployed (go to next page)
Student (go to next page)
Retired (go to next page)
Other, specify (go to 7)
Prefer not to say (go to 7)

What is your annual income? ____

Don’t know
Prefer not to say

[If 7 is blank], In which of these ranges does your annual income fall?

Less than $25,000
$25,000 - $50,000
$50,001 – $75,000
$75,001 - $100,000
More than $100,000
Don’t know
Prefer not to say

Appendix C – Consent Form

CONSENT FORM OMB Control Number: 1220-0141

Expiration Date: April 30, 2018

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 10 minutes to participate in this research..

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 (DOL/BLS – 14 BLS Behavioral Science Research Laboratory Project Files (81 FR 47418)), as amended (5 U.S.C. 552a) , you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

Appendix D – Online Survey Introductory Screens

Thank you for participating in our study! This short study will ask you to review an email and give your feedback about it.

This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

We are looking for information about how respondents react to our survey invitations. Please take your time as you answer these questions. The information you provide will contribute to valuable research at the BLS, one of the Federal statistical agencies.

[page break]

For this study we are interested in your reactions to an email that we send to companies when they are selected to be part of our government survey.

Please think about this email in the context of your job, as if you received this email at work.

[Insert one of three email formats]

Please click to confirm you've read the email.

Confirmed

[page break]

[go to debriefing questions]

Page 23 of 23

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	kopp_b
File Modified	0000-00-00
File Created	2021-01-22