Note to Reviewer - Evaluation of BLS Brochure

July 8, 2013

NOTE TO THE REVIEWER OF:	OMB No. 1220-0141 Cognitive and Psychological Research
FROM:	William Mockovak Office of Survey Methods Research (OSMR)
SUBJECT:	Evaluation of the BLS brochure “How We Serve the Nation and You’’

Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.”

In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.

We anticipate this research will require approximately 126.7 burden hours.

If there are any questions regarding this project, please contact Bill Mockovak (202) 691-7414.

I. Introduction and Purpose

A BLS team has redesigned the general purpose 14-page brochure ‘How We Serve the Nation and You’ (see Attachment A), and has requested feedback from potential users about its effectiveness.

The brochure has been designed for a diverse audience including respondents to household and business surveys, educators and career counselors, economists, researchers, and attendees at professional conferences.

II. Planned Evaluation Steps and Data Collection Methodology

We propose to obtain feedback using several approaches.

First, staff from the Behavioral Science Research Center has completed an expert review of the brochure that focused on formatting, identification of possible jargon, consistency of presentation, and perceived overall effectiveness of the brochure. The Brochure Team is in the process of reviewing these comments and will consider them in the context of results from additional evaluation activities described in this proposal.

The current evaluation approach will rely on the use of respondents/reviewers obtained from three online services: Trymyui¹, SurveyMonkey², and Mechanical Turk.³

SurveyMonkey and Mechanical Turk will be used to obtain reactions to the brochure and individual pages using a short, brief questionnaire, whereas the Trymyui service will be used to obtain more detailed, in-depth reactions.

Mechanical Turk is an online testing service that OSMR has not used before, and which we would like to gain experience with. Therefore, a small comparative test will be run between SurveyMonkey and Mechanical Turk to see if comparable results are obtained.

The online services provide samples of users that can meet different selection criteria. SurveyMonkey provides samples at a cost of about $1 per respondent for a 100-person sample, with the general guideline that a survey should focus on only a few key points and be kept brief. The cost of a 20-minute video session with Trymyui costs about $35, but can be less depending on the number of sessions ordered.

Mechanical Turk allows you to set up tasks (for example, complete a short survey) and pay participants accordingly. In this case, we will pay $1 to each participant for completing a brief questionnaire to make the experience comparable to SurveyMonkey.

To limit costs, only selected pages and content from the brochure will be tested.

A. Proposed Evaluation Approach

We will use SurveyMonkey, Trymyui, and Mechanical Turk to obtain feedback on topics of interest and specific pages from the 14-page BLS brochure. The content/topics shown in Tables 1 and 2 below were selected because they were considered most important in terms of their potential impact on the motivation of potential BLS respondents and data users.

B. Embedded Experiments

There will be two small embedded experiments.

• The first experiment will compare results when a page from the brochure is embedded in the survey instrument versus requiring the respondent to click a link to see the page, and when a progress indicator is used or not used (see Table 2).

• The second experiment will compare the results between SurveyMonkey and Mechanical Turk for the Brochure, in general, and for the “Your Participation Counts!” page. In these comparisons, there will be a link in the instrument to the illustrations and a progress indicator will be used (see Table 1).

Although illustrations can be embedded in a SurveyMonkey instrument, file size limitations mean that the page’s resolution may be negatively affected and impact a respondent’s reactions. On the other hand, clicking a link and waiting for the brochure page to display in a separate window takes additional time, so requiring this step might irritate or even possibly confuse respondents and lead to higher nonresponse or lower ratings, especially among respondents who are used to short, simple surveys. Therefore, it would be informative to compare both approaches and their effect on subsequent ratings.

Table 1: Pages/Content to be tested from the BLS Brochure (instruments displaying this

content will include a progress indicator, and there will be a link in the instrument

to the content or page)

		Sample Size:
Brochure Page(s)	Desired Audience	SurveyMonkey	Mech Turk	Trymyui Sample Size
Brochure, in general	Ages 21-65	100	100	5
Front and Back Covers	Ages 21-65	100		54
‘Your Participation Counts!’ pages	Ages 21-65	100	100	5
‘Financial Investors’ page	Ages 21-65	100		5
Total		400	200	20

Table 2: Embedded vs. Linked Illustrations (sample sizes shown

In parentheses)

		Progress Indicator		Total: Linked & Embedded Sample	Trymyui Sample
Page from Brochure		Yes	No
How We Serve	Embedded	100	100		3
	Linked	100	100		2
Students/Educators	Embedded	100	100		2
	Linked	100	100		3
		400	400	800	10

The use of progress indicators has led to inconclusive findings in online surveys, but in the tests proposed in this study, their use could be beneficial because we are probably tapping audiences with minimal motivation. Therefore, such audiences might use progress indicators more than in other contexts.

A page from the brochure is geared towards both students and educators, but we do not think we will be able to obtain a good sample of Educators using SurveyMonkey or Mechanical Turk. Therefore, we’re proposing only using SurveyMonkey and Mechanical Turk for obtaining reactions from students, and using Trymyui to pick up a small number of educators, because more control over recruiting is possible in Trymyui. In addition to the 20 Trymyui testers shown in Table 1, each brochure page in Table 2 will also be tested by 5 Trymyui testers (10 total), for a total of 30 in Tables 1 and 2.

The sample sizes shown in Tables 1 and 2 would be the maximum. Actual sample sizes and the number of pages tested will depend on available resources.

C. Analysis

The experiment comparing use of a progress indicator and how an illustration is accessed (embedded vs. hyperlink) will rely on responses to three 5-point Likert items (see Attachment B for the complete questions):

• How would you rate the overall appearance of this page?

• Considering the writing style and readability of the page you reviewed, how easy was it to read?

• After reading the information on this page, how likely is it that you will visit the Bureau of Labor Statistic’s website to learn more about statistics that interest or affect you?

An ANOVA ( 2 x 2 fixed) will be run separately with responses to each question as the dependent variable. The power of each F test is 0.9 assuming standard deviations of 0.23 for each row and column, 0.33 for row*column cells, 1.0 for total within, and 100 observations per factor combination. With these values, a between-groups effect of 0.28 (for example, use or non-use of a progress indicator) can be detected with a power of 0.8 (Tukey/HSD, alpha = 0.05, two sided).⁵

Ratings obtained using SurveyMonkey and Mechanical Turk for the “Your Participation Counts” page and “The Brochure, in general” will be compared using a two sample t test (two tailed). Assuming equal standard deviations of 0.4, the study will be able to detect a true difference of 0.2 with a power of 0.94.

D. Possible Evaluation Questions

Attachment B shows the SurveyMonkey and Mechanical Turk questions that will be asked for each topic of interest or brochure page(s). Each survey is expected to last less than an average of 5 minutes.

Attachment C shows the Trymyui protocol that will be used. Trymyui will actually first link to a SurveyMonkey instrument so that the relevant part of the brochure can be displayed.

Notice that some of the questions are trying to assess the direct impact of the page that is read. Since some of the evaluation questions are identical, they will also allow comparisons between pages of the brochure as a measure of their relative effectiveness.

III. Burden Hours

Burden hours were estimated as follows:

• 1,400 online respondents at 5 minutes per survey = 116.7 burden hours

• 30 Trymyui respondents at 20 minutes per session = 10.0

Total 126.7 burden hours

IV. Data Confidentiality

Recruiting of participants will be handled by SurveyMonkey, Mechanical Turk, and Trymyui. Participants will be informed that the study will be used to improve the design of the BLS Brochure ‘How We Serve the Nation and You.’ No pledge of confidentiality will be given to survey respondents.

V. Attachments

A. How We Serve the Nation and You Brochure

B. Proposed SurveyMonkey evaluation questions for:

• Brochure, in general

• Front and Back Covers

• ‘Your Participation Counts!’ page

• ‘How We Serve’ page

• ‘Financial Investors’ page

• ‘Students and Educators’ page

C. Trymyui Instructions

D. Instructions for Mechanical Turk

Attachment B - SurveyMonkey Questions about the Brochure, in General

Introduction: You will be reviewing a brochure from the Bureau of Labor Statistics, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. When you click the following link, you will be taken to a brochure. Please take a couple of minutes to scan it. [link to pdf of brochure]

3. How would you rate the overall appearance of the brochure?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

• No opinion

4. Considering the writing style and readability of the brochure, how easy was it to read?

• Very easy

• Easy

• Neither easy nor hard

• Hard

• Very hard

5. Would you say the overall length of the brochure was …

• Too short?

• About right?

• Too long?

• No opinion or not able to judge

5. Would you prefer a paper brochure or would you prefer to view the brochure on a webpage with hyperlinks? (A hyperlink would take you to more detailed information)

• Paper brochure

• Webpage with hyperlinks

• Either

• No opinion

5. When you scanned the brochure, did you learn of BLS data products that would be of interest to you?

• Yes

• No

5. Have you ever used information from the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

5. How likely is it that you will visit the Bureau of Labor Statistics’ website to learn more about statistics that interest or affect you?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

5. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

5. Please enter your age: _____

Sex: Male

Female

5. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

5. Thank you!  Any comments about the brochure are welcome.

SurveyMonkey Questions about the Front and Back Covers

Introduction: You will be asked to review the covers of a brochure, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. When you click the following link, you will be taken to a brochure. Please scan the front and back covers (pages). [link to pdf of front & back covers]

3. How would you rate the overall appearance of the front cover?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

• Other comments

4. How would you rate the overall appearance of the back cover?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

• Other comments

5. What did you think of the pictures on the front cover?

• No specific reactions – skip to next question

Enter comments in box below

6. Based on the front cover, what type of information did you expect the brochure to contain?

• No specific expectations – skip to next question

7. Have you ever used information from the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

7. Have you ever participated in a Bureau of Labor Statistics’ survey?

• Yes

• No

• Don’t know

7. Please enter your age: _____

Sex: Male

Female

8. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

9. Thank you!  Any other comments about the front and back covers are welcome.

SurveyMonkey Questions for the “Your Participation Counts!” Page

Introduction: You will be reviewing a single page from a Bureau of Labor Statistics brochure, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. If you were asked to voluntarily participate in a Bureau of Labor Statistics survey, how likely is it that you would participate?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

• Don’t know/It depends

3. Please click the following link and read the information presented on the page.  [link to “Your Participation Counts!” page]

4. How would you rate the overall appearance of this page?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

4. Considering the writing style and readability of the page you reviewed, how easy was it to read?

• Very easy

• Easy

• Neither easy nor hard

• Hard

• Very hard

5. How convincing would the following points be for someone who was asked to participate in a Bureau of Labor Statistics survey?

	Not at all convincing	A little convincing	Somewhat convincing	Very convincing
You count (your participation is important)
You represent many other Americans
By participating you have a voice in government statistics
Your information will be kept confidential

6. After reading the information “Your Participation Counts!” would you participate in a survey?

• Yes

• No

• It depends – Please explain

(If ‘It Depends’) Which of the following would probably be most important in your

decision? (Check all that apply)

• Purpose or content of the survey

• Length of the survey

• Confidentiality of data

• Whether I am paid

• Type of interview – phone, mail, Internet, face-to-face

• Other – please explain

7. Have you ever participated in a Bureau of Labor Statistics’ survey?

• Yes

• No

• Don’t know

8. Have you ever used information from the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

9. Please enter your age: _____

Sex: Male

Female

10. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

11. Thank you!  Any other comments about the page you reviewed are welcome.

SurveyMonkey Questions for the “Financial Investors” Page

Introduction: You will be reviewing a single page from a Bureau of Labor Statistics brochure, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. Please click the following link and read the information presented on the page.  [link to “Financial Investors” page]

3. How clearly does this page describe information that financial investors can use on the Bureau of Labor Statistics’ website?

• Very clearly

• Clearly

• Neither clearly nor unclearly

• Unclearly

• Very unclearly

4. How would you rate the overall appearance of this page?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

5. Considering the writing style and readability of the page you reviewed, how easy was it to read?

• Very easy

• Easy

• Neither easy nor hard

• Hard

• Very hard

6. Have you ever used information from the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

7. After reading the information on this page, how likely is it that you will visit the Bureau of Labor Statistic’s website to learn more about statistics that interest or affect you?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

8. Please enter your age: _____

Sex: Male

Female

9. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

10. Thank you!  Any other comments about the page you reviewed are welcome.

SurveyMonkey Questions for the “How We Serve …?” Page

Introduction: You will be reviewing a single page from a Bureau of Labor Statistics brochure, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. If you were asked to voluntarily participate in a Bureau of Labor Statistics survey, how likely is it that you would participate?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

• Don’t know/It depends on the survey

3. Please click the following link and read the information presented on the page.  [link to “How We Serve …?” page]

4. How would you rate the overall appearance of the page you just reviewed?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

4. Considering the writing style and readability of the page you reviewed, how easy was it to read?

• Very easy

• Easy

• Neither easy nor hard

• Hard

• Very hard

5. How well does this page explain the Bureau of Labor Statistics’ mission?

• Very Well

• Well

• Neither well nor poorly

• Poorly

• Very Poorly

6. Have you ever used data from the Bureau of Labor Statistics website?

• Yes

• No

• Don’t know

7. After reading the information on this page, how likely is it that you will visit the Bureau of Labor Statistic’s website to learn more about statistics that interest or affect you?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

8. After reading the information on this page would you participate in a survey?

• Yes

• No

• It depends

(If ‘It Depends’) Which of the following would probably be most important in your decision? (Check all that apply)

• Purpose or content of the survey

• Length of the survey

• Confidentiality of data

• Whether I am paid

• Type of interview – phone, mail, Internet, face-to-face

• Other – please explain

9. Have you ever participated in a Bureau of Labor Statistics Survey?

• Yes

• No

• Don’t know

10. Please enter your age: _____

Sex: Male

Female

11. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

12. Thank you!  Any other comments about the pages you reviewed are welcome.

SurveyMonkey Questions for the “Students and Educators” Page

Introduction: You will be reviewing a single page from a Bureau of Labor Statistics brochure, and asked for your opinions about it. This survey is being done to help the Bureau of Labor Statistics improve its products.

Please note that this survey is being administered by SurveyMonkey and resides on a server outside of the Bureau of Labor Statistics (BLS) domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of personally-identifiable information -- such as your full name, phone number, e-mail address, etc. -- in any response.

The OMB control number for this voluntary survey is 1220-0141.  Without a currently valid number BLS would not be able to conduct this survey.  The current expiration date for this control number is February 28, 2015.

1. Do you know what the Bureau of Labor Statistics (BLS) does?

• Yes

• No

• Not sure

2. Please click the following link and read the information presented on the page.  [link to “Students and Educators” page]

3. How clearly does this page describe information that students and educators can use on the Bureau of Labor Statistics’ website?

• Very clearly

• Clearly

• Neither clearly nor unclearly

• Unclearly

• Very unclearly

4. How would you rate the overall appearance of this page?

• Very attractive

• Attractive

• Neither attractive nor unattractive

• Unattractive

• Very unattractive

4. Considering the writing style and readability of the page you reviewed, how easy was it to read?

• Very easy

• Easy

• Neither easy nor hard

• Hard

• Very hard

8. Have you ever used information from the Bureau of Labor Statistics’ website, such as the Occupational Outlook Handbook?

• Yes

• No

• Don’t know

8. After reading the information on this page, how likely is it that you will visit the Bureau of Labor Statistic’s website to learn more about statistics that interest or affect you?

• Very likely

• Likely

• Neither likely nor unlikely

• Unlikely

• Very unlikely

8. Please enter your age: _____

Sex: Male

Female

8. Which of the following best describes you?

• Student

• Educator

• Business leader

• Professional

• Jobseeker

• Human resources

• Consumer

• Other – please describe

8. Thank you!  Any other comments about the page you reviewed are welcome.

Attachment C: Trymyui Instructions – These instructions will be used for the Brochure, in

general

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion of a brochure that will be given to people asked to participate in government surveys. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and in SurveyMonkey

Please take a few minutes to scan the brochure. Please share your thoughts and reactions as you complete this review. Please answer the following questions when you are ready.

1. How would you rate the overall appearance of the brochure? What do you like about it? Dislike?

2. What do you think about the brochure's length? Is it about right, too short, or too long?

3. What did you think of the writing style or readability? Was it easy to read? Who do you think would best be able to read and use this?

4. If you were given this brochure, how likely is it that you would go to the BLS website to learn more about a topic?

5. After scanning the brochure, which topics were of most interest to you?

6. Would this brochure influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Trymyui Instructions – These instructions will be used for the Front and Back covers of the Brochure

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion of a brochure that will be given to people asked to participate in government surveys. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and SurveyMonkey

The focus of this evaluation is on the front and back covers of the brochure, but please scan the contents of the brochure as well. Share any thoughts or reactions as you complete this review. When you are ready, answer the following questions.

1. How would you rate the appearance of the front cover? Of the back cover? What do you like about either one? Dislike?

2. What do you think of the pictures used on the front cover? What type of information did you think the brochure would contain?

3. What do you think about the brochure's length? Is it about right, too short, or too long?

4. What do you think about the brochure, in general?

5. After scanning the brochure, which topics were of most interest to you?

6. Would this brochure influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Trymyui Instructions – These instructions will be used for the ‘Your Participation Counts!’ page

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion a page from a brochure that will be given to people asked to participate in government surveys. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and in SurveyMonkey

Take a few minutes to read this page from the brochure. You do not need to read out loud, but feel free to share any thoughts or reactions. Please answer the following questions when you are ready.

1. How would you rate the overall appearance of this page? Is it attractive or unattractive?

2. What caught your attention on this page? Which points were most persuasive?

3. What did you think of the writing style or readability? Was it easy to read? Who do you think would best be able to read and use this?

4. After reading this page, how likely is it that you would go to the BLS website to learn more about a topic?

5. Would this page influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Trymyui Instructions – These instructions will be used for the ‘Financial Investors’ page

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion of a page from a brochure that will be given to people asked to participate in government surveys. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and in SurveyMonkey

Take a few minutes to read the page from the brochure. You do not need to read out loud, but feel free to share any thoughts or reactions. Please answer the following questions when you are ready.

1. How would you rate the overall appearance of this page? Would you say it is attractive or unattractive?

2. What caught your attention on this page? Which points were most persuasive?

3. What did you think of the writing style or readability? Was it easy to read? Who do you think would best be able to read and use this?

4. After reading this page, how likely is it that you would go to the BLS website to learn more about a topic?

5. Would this page influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Trymyui Instructions – These instructions will be used for the ‘How We Serve’ page

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion of a page from a brochure that will be given to people asked to participate in government surveys. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and in SurveyMonkey

Take a few minutes to read the page from the brochure. You do not need to read out loud, but feel free to share any thoughts or reactions. Please answer the following questions when you are ready.

1. How would you rate the overall appearance of this page? Would you say it is attractive or unattractive?

2. What caught your attention on this page? Which points were most persuasive?

3. What did you think of the writing style or readability? Was it easy to read? Who do you think would best be able to read and use this?

4. After reading this page, how likely is it that you would go to the BLS website to learn more about a topic?

5. Would this page influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Trymyui Instructions – These instructions will be used for the ‘Students & Educators’ page

Instructions and Questions Displayed for Participants in Trymyui

This test differs from your typical usability test. The Bureau of Labor Statistics would like your opinion of a page from a brochure that will be given to people asked to participate in government surveys. When reviewing, please ignore the poor image resolution and watermarks on the images and respond as if the images were clear and unblemished. Please follow the instructions that appear on the web page to your right. Click Next when you are ready to move from one page to another.

Instructions and Questions Displayed in a Task Pane in Trymyui and in SurveyMonkey

Take a few minutes to read the page from the brochure. You do not need to read out loud, but feel free to share any thoughts or reactions. Please answer the following questions when you are ready.

1. How would you rate the overall appearance of this page? Would you say it is attractive or unattractive?

2. What caught your attention on this page? Which points were most persuasive?

3. What did you think of the writing style or readability? Was it easy to read? Who do you think would best be able to read and use this?

4. After reading this page, how likely is it that you would go to the BLS website to learn more about a topic?

5. Would this page influence your decision to participate in a Bureau of Labor Statistics survey?

Follow-up Trymyui Questions

1. Have you ever visited the Bureau of Labor Statistics’ website?

• Yes

• No

• Don’t know

2. Have you ever participated in a BLS survey?

• Yes

• No

• Don’t know

Attachment D – Instructions for Mechanical Turk

Note: Participants in Mechanical Turk will click a link that takes them to the SurveyMonkey instrument. Once in SurveyMonkey, Mechanical Turk and SurveyMonkey participants will see the same instructions and questions. The standard disclaimer will appear in the SurveyMonkey instructions.

A – These instructions will be used for the Brochure, in general

Instructions for Participants in Mechanical Turk

The Bureau of Labor Statistics would like your opinion of a brochure that will be given to people asked to participate in government surveys. You will be asked to look at the brochure and give your general reactions. This should take less than 5 minutes. You will be paid $1 for completing this task. Click the link that follows to begin this survey.

B - These instructions will be used for the “Your Participation Counts” page

Instructions for Participants in Mechanical Turk

The Bureau of Labor Statistics would like your opinion of a page from a brochure that will be given to people asked to participate in government surveys. You will be asked to read the page and give your general reactions. This should take less than 5 minutes. You will be paid $1 for completing this task. Click the link that follows to begin this survey.

1 Trymyui: http://www.trymyui.com/

2 SurveyMonkey: http://www.surveymonkey.com/

3 Mechanical Turk: https://www.mturk.com/mturk/

4 The focus will be on the front and back pages, but since a Trymyui session can last up to 20 minutes, in this

case the participants will also be asked to scan the brochure and provide general comments.

5 Lenth, R. V. (2006-9).  Java Applets for Power and Sample Size [Computer software].  Retrieved July 24, 2013, from http://www.stat.uiowa.edu/~rlenth/Power

0