Note to Reviewe - 2013 Web CI Study

July 11, 2012

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the 2013 Web Cognitive Interviewing Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for the study is 33.3.

If there are any questions regarding this project, please contact Jennifer Edgar at

202-691-7528.

1. Introduction and Purpose

Building upon previous research (Edgar, 2012 & 2013), this study seeks to further explore the use of self-administered, think-aloud, web surveys to collect information typically collected during a face-to-face cognitive interview. Cognitive interviewing is a research method used to gain insight about a participant’s response process. Typically, there is a significant interaction between the participant and the interviewer, with the interviewer using both scripted and spontaneous probes to ask follow-up questions about the participant’s response.

Recruiting for and conducting cognitive interviews is a resource intensive process, with much time spent screening participants to identify those with the desired characteristics and ability to verbalize their thought processes even before the interview is conducted. By using an online panel, pre-screened for verbal ability and qualifications, the amount of time spent recruiting for a study can be significantly diminished. Further gains are realized when an interviewer is not needed to conduct each interview, instead a researcher can devote time to designing the self-administered web survey and analyzing the resulting data.

Thus far, evidence has been found that self-administered, think-aloud, web surveys are a feasible way to collect information about question comprehension and response processes, at least for the questions tested with potentially significant time savings. There has also been mixed findings on the comparability of results across modes, with some questions exhibiting different issues when testing in person and on the internet (Edgar, 2013). This study seeks to extend this line research by replicating past interviews, conducted in both the OSMR cognitive lab and through alternative methods.

2. Research Design

To address the research questions, two methods will be used a self-administered web survey and an interviewer-administered cognitive interview. Additionally, two recruiting methods will be used: standard OSMR recruiting and the TryMyUI panel.

Interviewer-administered Interview

A separate group of participants will complete a standard cognitive interview in the OSMR lab, using the protocol included as Attachment B. The protocol was developed when the questions were being originally tested for the CE. The protocols were approved by OMB with original data collection in 2007 and 2009, as well as a global clothing question, tested in 2010. Additionally, two questions previously tested by RTI using alternative methods (Mechanical Turk and Facebook) are included for comparison purposes.

Self-administered Web Survey

Participants will complete a web survey hosted by Survey Monkey, given by TryMyUI to participants. The proposed draft survey can be viewed at https://www.surveymonkey.com/s/WebCI2013, and is Attachment A. The survey contains a series of survey questions, each with several follow-up questions which will seek to evaluate comprehension, judgment and/or response process. The survey was designed to match the cognitive interviewing protocols (which were developed for testing of production Consumer Expenditure Survey questions several years ago) to the extent possible. Participants will be also asked to complete a series of categorization-type questions to gain additional information about their understanding of the key concepts.

TryMyUI

TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tasks for which they qualify. Each usability task is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. An audio and video recording showing the participant completing the task is then provided.

For this study, this usability testing site is being used as a way to collect a large amount of data efficiently (both in terms of financial and staff resources). TryMYUI is responsible for recruitment, screening, administration of the survey, and data collection. After completing each study, participants are evaluated by the researchers, and only those found to be able to provide thoughtful and complete talk-aloud responses are kept in the panel. This evaluation and screening process helps ensure that all participants provide useable data, a confidence we do not have with lab participants, whose verbal abilities are often unknown. Additionally, TryMyUI is able to provide data within hours of receiving the study from the sponsor. The same data would take weeks for the Office of Survey Methods Research (OSMR) to collect in the usability lab.

Participants will receive a link to an online survey hosted by SurveyMonkey. The survey will be developed to mirror the existing cognitive interview protocol as closely as possible. For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMYUI, a few questions will have instructions for participants to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.

Analysis

Audio recordings of the existing and new cognitive interviews and video recordings of the web will be analyzed. Each interview will be reviewed and coded, using a coding scheme developed to evaluate the amount and quality of information collected via each mode. Analysis of the time spent recruiting, scheduling and conducting interviews, and spent analyzing the data, will also be done to compare the efficiency of each mode. The cost of each method will also be compared.

3. Participants

Participants will be recruited by TryMyUI and OSMR. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on their website. Participants will be selected with varying levels of age, gender, and education.

Lab participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. Participants will be screened to ensure they’ve purchased clothing within the past month. Lab participants will be randomly assigned to either a standard cognitive interview or a self-administered cognitive interview.

Study Sample Sizes

	OSMR Recruiting	TryMyUI
Standard lab CI	201	n/a
Self-administered CI	20	20

4. Payment

Self-administered Survey

In the TryMyUI framework, participants are paid $10 per completed survey. This is a fixed amount set by the company.

Interviewer-administered Interview

Participants will receive $40.

5. Burden Hours

The self-administered survey is limited to twenty minutes, with up to 40 participants completing it. The interviewer-administered interview is expected to take up to 60 minutes, with up to 20 participants completing it, for a total of 33.3 burden hours.

5. Data Confidentiality

Interviewer-administered Interview

Participants will be informed as to the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of a national consumer expenditure interview survey. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. Prior to the interview, participants will be given a consent form to read and sign (Attachment C).

Self-administered Survey

Participants obtained through TryMyUI sign separate agreements with TryMyUI, and although BLS follows steps to ensure that their confidentiality is maintained, no pledge of confidentiality is provided to these respondents.

The following notice will be placed on the first page of the survey: “Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”

6. References

Edgar, J (2012). Cognitive Interviews without the Cognitive Interviewer? Paper Presented at the American Association of Public Opinion Research, Orlando FL.

Edgar, J. (2013) Self-Administered Cognitive Interviewing. Paper Presented at the American Association of Public Opinion Research, Boston, MA.

Attachment A: Web Survey

-- See Separate File --

Attachment B: Interviewing Protocols

Note: These are the original protocols that were used for cognitive interviews already collected. New cognitive interviews will be conducted using the same protocols, and the self-administered web survey was developed to match (with addition of instructions for when to think aloud)

Mortgage: (Screener: Do you currently make mortgage payments? If no, skip to next topic.

Main Question(s)

Now I will ask some questions about your primary mortgage. These questions refer to the mortgage you are currently making payments on.

1. What was the amount of the mortgage when you first obtained it, not including any interest?

2. In what year did you make the first payment on this mortgage?

3. What is the current interest rate on your mortgage?

4. How much is your mortgage payment per month?

5. How much of that amount is for principal and interest?

6. On your most recent payment, which of these were included?

   1. Principal

   2. Interest

   3. Property Taxes

   4. Property insurance

   5. Mortgage guarantee insurance (PMI)

   6. Any other payments – specify

Follow-up Questions

1. First, when I asked you what the amount of your mortgage was when you first obtained it, how did you arrive at your answer?

2. How easy or hard was it for you to remember what year you made your first payment?

3. When I asked you the current interest rate on your loan, On a scale from one to ten, with ten being absolutely certain and 1 meaning you made a wild guess, how sure are you that your answer is accurate?

4. When I asked you how much of your monthly payment was for principal and interest, how did you arrive at your answer?

Sportswear

Main Question(s)

Since the first of [Refmonth] have you or any member of your household purchased any swimsuits or warm-up or ski suits?

Follow-up Questions

1. What types of items did you think of when you heard the question?

2. How sure were you that those expenses were what were meant by this question?

Let me repeat the question. “Since the first of [Refmonth] have you or any member of your household purchased any swimsuits or warm-up or ski suits?

Now I’m going to name some items, tell me if they would be included in this question:

1. a swimsuit?

2. a jogging suit?

3. a track suit?

4. a warm-up jacket?

5. a snow suit?

6. sneakers

7. are there other items that would be included in this question that we haven’t mentioned?

Delivery & Installation Charges

Now we’re going to talk about expenses for delivery and installation charges. Please listen to the following question, give me your answer, then I’ll ask some follow-up questions about it.

In the past year, have you purchased or rented any items that had extra charges for installation or delivery?

1. What did you purchase/rent?

2. What was the purchase price?

3. Does that include the charge for installation or delivery?

4. How much did installation or delivery cost?

Follow-up Questions

Let’s talk about those questions for a minute.

5. What types of items can have installation charges?

6. What types of items can have delivery charges?

Global Clothing

How much have you or any member of your household spent on clothing since the first of [Refmonth]?

Follow up questions

1. What did you think about when deciding how much you've spent on clothing?

2. When figuring out how much you've spent on clothing, did you think mostly about:

   1. Specific clothing items

   2. Specific shopping trips

   3. Specific events

   4. Clothing items in general

   5. Something else

3. Please explain your answer to the last question

4. On a scale from one to ten, how certain are you that the amount you've spent on clothing is accurate?

1 - Wild Guess

2

3

4

5 - Somewhat Certain

6

7

8

9

10 - Absolutely Certain

5. What clothing items have you or any member of your household bought since the first of October and how much did each cost? (Example: 1 shirt, $20)

6. Is there anything else you’d like to add about your answer to the clothing question?

Season (From RTI Study)

Have you had a flu shot this season? Yes, No

Follow up questions

1. In your own words, how do you define the word "season" when you read the question above? 

2. When you read "season" in the question above, what time period do you think of? 

Smoking (From RTI Study)

Have you smoked at least 100 cigarettes in your entire life?

Follow up questions

1. Do you think that you could accurately answer this survey question? (Yes, No)

2. If you were asked this survey question, how would you mentally calculate the answer? 

Attachment B: Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02/28/15

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

1 This group is to add to the interviews conducted in past years. There are currently about 40 interviews available for analysis

15