Web Cognitive Interviews Study

January 2, 2013

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Web Cognitive Interviews Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 37 hours.

If there are any questions regarding this project, please contact Jennifer Edgar at

202-691-7528.

Attachments

A – Cognitive Interviewing Protocols

B – Consent Form for lab testing

1. Introduction and Purpose

Building upon previous research (Edgar, 2012), this study seeks to further explore the use of self-administered, think-aloud, web surveys to collect information typically collected during a face-to-face cognitive interview. Cognitive interviewing is a research method used to gain insight about a participant’s response process. Typically, there is a significant interaction between the participant and the interviewer, with the interviewer using both scripted and spontaneous probes to ask follow-up questions about the participant’s response.

Recruiting for and conducting cognitive interviews is a resource intensive process, with much time spent screening participants to identify those with the desired characteristics and ability to verbalize their thought processes even before the interview is conducted. By using an online panel, pre-screened for verbal ability and qualifications, the amount of time spent recruiting for a study can be significantly diminished. Further gains are realized when an interviewer is not needed to conduct each interview, instead a researcher can devote time to designing the self-administered web survey and analyzing the resulting data.

Thus far, evidence has been found (Edgar, 2012) that self-administered, think-aloud, web surveys are a feasible way to collect information about question comprehension and response processes, at least for the questions tested. That study also found significant time savings. This study seeks to extend that research by testing additional types of survey questions and using different types of follow-up questions.

2. Research Design

To address the research questions, two methods will be used: a self-administered web survey and an interviewer-administered cognitive interview. To leverage existing cognitive interviewing data, a series of questions from the Consumer Expenditure Survey that were previously tested in the cognitive lab were selected.

Self-administered Web Survey

Participants will complete a web survey hosted by Survey Monkey, given by TryMyUI to participants. The proposed draft survey can be viewed at

https://www.surveymonkey.com/s/WebCI. The survey contains a series of survey questions, each with several follow-up questions which will seek to evaluate comprehension, judgment and/or response process. The survey was designed to match the cognitive interviewing protocols to the extent possible. Participants will be also asked to complete a series of categorization-type questions to gain additional information about their understanding of the key concepts.

TryMyUI

TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tasks for which they qualify. Each usability task is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. An audio and video recording showing the participant completing the task is then provided.

For this study, this usability testing site is being used as a way to collect a large amount of data efficiently (both in terms of financial and staff resources). TryMYUI is responsible for recruitment, screening, administration of the survey, and data collection. After completing each study, participants are evaluated by the researchers, and only those found to be able to provide thoughtful and complete talk-aloud responses are kept in the panel. This evaluation and screening process helps ensure that all participants provide useable data, a confidence we do not have with lab participants, whose verbal abilities are often unknown. Additionally, TryMyUI is able to provide data within hours of receiving the study from the sponsor. The same data would take weeks for the Office of Survey Methods Research (OSMR) to collect in the usability lab.

Participants will receive a link to an online survey hosted by SurveyMonkey. The survey will be developed to mirror the existing cognitive interview protocol as closely as possible. For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMYUI, a few questions will have instructions for participants to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.

Interviewer-administered Interview

A separate group of participants will complete a standard cognitive interview in the OSMR lab, using the protocol included as Attachment A. The protocol was developed when the questions were being originally tested, and will be used again when resources allow new data collection. The protocols were approved by OMB with original data collection in 2007 and 2009. In addition, a global clothing question, tested in 2010 is also included.

Analysis

Audio recordings of the existing and new cognitive interviews and video recordings of the web will be analyzed. Each interview will be reviewed and coded, using a coding scheme developed to evaluate the amount and quality of information collected via each mode. Analysis of the time spent recruiting, scheduling and conducting interviews, as well as the time spent analyzing the data, will also be done to compare the efficiency of each mode.

3. Participants

Self-administered Survey

Participants are recruited by TryMyUI. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on their website. Participants will be selected with varying levels of age, gender, and education.

Interviewer-administered Interview

When resources allow, participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process.

4. Payment

Self-administered Survey

In the TryMyUI framework, participants are paid $10 per completed survey. This is a fixed amount set by the company.

Interviewer-administered Interview

Participants will receive $40.

5. Burden Hours

The self-administered survey is limited to twenty minutes, with up to 50 participants completing it. The interviewer-administered interview is expected to take up to 60 minutes, with up to 20 participants completing it, for a total of 37 burden hours.

5. Data Confidentiality

Interviewer-administered Interview

Participants will be informed as to the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of a national consumer expenditure interview survey. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. OMB Control Number: 1220-0141. Prior to the interview, participants will be given a consent form to read and sign (Attachment B).

Self-administered Survey

Participants obtained through TryMyUI sign separate agreements with TryMyUI, and although BLS follows steps to ensure that their confidentiality is maintained, no pledge of confidentiality is provided to these respondents.

The following notice will be placed on the first page of the survey: “Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”

6. References

Edgar, J (2012). Cognitive Interviews without the Cognitive Interviewer? Paper Presented at the American Association of Public Opinion Research, Orlando FL.

Attachment A: Interviewing Protocols

Note: These are the original protocols that were used for cognitive interviews already collected. New cognitive interviews will be conducted using the same protocols, and the self-administered web survey will be developed to match (with addition of instructions for when to think aloud)

Mortgage: (Screener: How many mortgages do you currently have? If 0, skip to next topic.

Main Question(s)

Now I will ask some questions about your primary mortgage. These questions refer to the mortgage you are currently making payments on.

1. What was the amount of the mortgage when you first obtained it, not including any interest?

2. In what year did you make the first payment on this mortgage?

3. What is the current interest rate on your mortgage?

4. How much is your mortgage payment per month?

5. How much of that amount is for principal and interest?

6. On your most recent payment, which of these were included?

   1. Principal

   2. Interest

   3. Property Taxes

   4. Property insurance

   5. Mortgage guarantee insurance (PMI)

   6. Any other payments – specify

Follow-up Questions

1. First, when I asked you what the amount of your mortgage was when you first obtained it, how did you arrive at your answer?

2. How easy or hard was it for you to remember what year you made your first payment?

3. When I asked you the current interest rate on your loan, On a scale from one to ten, with ten being absolutely certain and 1 meaning you made a wild guess, how sure are you that your answer is accurate?

4. When I asked you how much of your monthly payment was for principal and interest, how did you arrive at your answer?

Sportswear

Main Question(s)

Since the first of [Refmonth] have you or any member of your household purchased any swimsuits or warm-up or ski suits?

Follow-up Questions

1. What types of items did you think of when you heard the question?

2. How sure were you that those expenses were what were meant by this question?

Let me repeat the question. “Since the first of [Refmonth] have you or any member of your household purchased any swimsuits or warm-up or ski suits?

Now I’m going to name some items, tell me if they would be included in this question:

1. a swimsuit?

2. a jogging suit?

3. a track suit?

4. a warm-up jacket?

5. a snow suit?

6. sneakers

7. are there other items that would be included in this question that we haven’t mentioned?

Delivery & Installation Charges

Now we’re going to talk about expenses for delivery and installation charges. Please listen to the following question, give me your answer, then I’ll ask some follow-up questions about it.

In the past year, have you purchased or rented any items that had extra charges for installation or delivery?

1. What did you purchase/rent?

2. What was the purchase price?

3. Does that include the charge for installation or delivery?

4. How much did installation or delivery cost?

Follow-up Questions

Let’s talk about those questions for a minute.

5. What types of items can have installation charges?

6. What types of items can have delivery charges?

Global Clothing

How much have you or any member of your household spent on clothing since the first of [Refmonth]?

Follow up questions

1. What did you think about when deciding how much you've spent on clothing?

2. When figuring out how much you've spent on clothing, did you think mostly about:

   1. Specific clothing items

   2. Specific shopping trips

   3. Specific events

   4. Clothing items in general

   5. Something else

3. Please explain your answer to the last question

4. On a scale from one to ten, how certain are you that the amount you've spent on clothing is accurate?

1 - Wild Guess

2

3

4

5 - Somewhat Certain

6

7

8

9

10 - Absolutely Certain

5. What clothing items have you or any member of your household bought since the first of October and how much did each cost? (Example: 1 shirt, $20)

6. Is there anything else you’d like to add about your answer to the clothing question?

Attachment B: Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02/28/15

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

12