Note to Reviewer

May 29, 2012

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jean Fox Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for Cognitive Testing of PPI Data Collection Forms

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, I am submitting a brief description of the study.

The total estimated respondent burden hours for this study are 35 hours.

If you have any questions regarding this project, please contact Jean Fox at [email protected] or 202-691-7370.

I. Introduction and Purpose

The Bureau of Labor Statistics (BLS) is revising the data collection form for the Producer Price Index (PPI). The current form has not been revised in almost 30 years, so when the PPI program upgrades the mail and fax system in the Fall of 2012, management would like to revise the form as well. We will focus our initial round of cognitive testing on a general form to be deployed first, then continue testing in subsequent rounds with one or more additional versions of the form. These versions will be more tailored to certain items or services. The goals of the revisions are (1) to address data collection problems with the current form and (2) to adjust the forms to meet the requirements of the new system.

II. Research Design

OSMR will conduct up to 35 cognitive interviews, in several rounds of testing. In the first round, we plan to interview approximately 15 PPI survey respondents (called “reporters”). In subsequent rounds, we expect to interview approximately 10 reporters per form (we expect to test up to two additional forms). We will not need as many participants in subsequent tests because these forms will be based on the initial form and because the populations using those forms will be more uniform. The exact number for each round will depend on the availability of participants. For all rounds of testing, we will attempt to obtain a sample with good diversity in the industries and geographic regions represented. The participants will be actual respondents for the PPI survey.

For the first round, approximately one-third to one-half of the tests will be conducted in the field. These participants will be recruited from an area requiring only local travel (e.g., approximately from Richmond to Philadelphia). We will interview the remaining participants by phone. This plan will allow the team to observe some of the participants completing the forms in person, while also allowing us to include respondents from a diverse geographical area. The exact ratio of in-person to remote interviews will depend on availability of participants and travel budget.

The items we will ask respondents to complete for the test will be similar to the items that they price for BLS, with a few modifications so we can evaluate how they complete the forms in a variety of situations.

III. Procedures

In general, the cognitive test will be very similar to a response analysis survey, where respondents complete a survey, then provide feedback in a subsequent interview. In this case, however, participants will be completing the forms using scenarios, rather than their real data, and some of the interviews will be done in person.

A. In-Person Interviews

For the in-person interviews, the interviewer will follow the protocol outlined in Attachment A, Section 1. We will ask each participant if they are comfortable with us audio recording the session, and we will record, or not, based on their response. Recording is helpful, since we will have a record of the session we can refer to and share with other team members who cannot attend, but it is not necessary.

After introductions, gaining informed consent, and a short explanation of the procedures for the cognitive test, the interviewer will provide the participant with the first item to price, along with a “scenario,” which will provide the information needed to complete the form.

We will create scenarios for each participant, based on the items or services that company provides. Each scenario will consist of a text description of the current situation we would like them to price and a form with the description of the item or service. This description will provide the information the participants need to complete the form, such as the price, changes to the item, and address changes. The form will have pre-printed data from the last reporting period, as a respondent would normally receive it. Sample scenarios are included in Attachment B, and a sample form is in Attachment C.

These scenarios should cover a range of situations, so we can test all parts of the form. Some situations will be rare or nonexistent for certain industries, so we may not be able to test all parts of the form with all participants. Overall, we expect that participants will price approximately five to seven items total.

We will ask the in-person participants to complete one scenario, then get their feedback on that scenario (questions 10-13 in the debriefing interview, see Attachment D). This will more accurately simulate how they would complete the form on their own, allowing us to get an estimate of the time needed. We will get feedback immediately after they complete the form, while it is still fresh on their mind.

Following the administration of the survey, the interviewer will debrief the participant (Attachment D). The debriefing will ask participants about their opinions of the new form.

B. Remote Interviews

The primary difference for the remote participants is that they will complete the forms ahead of time, and fax them back to us before the interview. This way, we will be able to look at their data as we are talking with them. For the remote participants, we will not be able to get their feedback after they complete each form, so we will start with their overall impressions. These interviews will be conducted by phone.

IV. Analysis

To evaluate the form, we will calculate how often participants make the changes we wanted them to make. We will also consider situations where participants made changes they should not have. In addition, we will evaluate the comments participants make while completing the form, as well as the time needed to complete the forms, and the results of the post-test debriefing. The team will recommend improvements to the draft form(s) based on the results of this test.

V. Participants and Burden Hours

We will have up to 35 participants in this study. The PPI Industry Analysts, who regularly communicate with PPI respondents for data collection, will identify and recruit the participants. They will look for participants who vary in terms of their geographic location, their industry, and the types of issues they may face when completing the forms. Should a participant with a disability be selected, we will make necessary accommodations to allow that person to participate. We expect that each test session will last under an hour, for a total of up to 35 burden hours.

VI. Payment

Participants will not receive any payments for participating in this cognitive test.

VII. Data Confidentiality

Participants will be informed of the voluntary nature of the study. Participants also will be informed that the study will be used for internal purposes to improve the design of the PPI data collection form. Participants will be given a consent form to read and sign (Appendix E) prior to beginning the test session. Information related to this study will not be released to the public in any way that would allow identification of individuals without their informed consent.

VIII. Attachments

Attachment A:	Cognitive Interview Protocol
Attachment B:	Sample Scenarios
Attachment C:	Sample Form
Attachment D:	Debriefing Items
Attachment E:	Consent Agreement Form and Confidentiality Statement

Attachment A: Cognitive Interview Protocol

1. In-person Interviews

• Thank you for participating today.

• I am …... [This is my colleague ( ) who will be taking notes for us today]

• Explain cognitive testing

  • The goal of our session is for us to evaluate a new form we are developing for PPI. We want to see how you complete the form. We are looking for what works well and what is more difficult.

  • During this session, we will give you several simulated forms to complete, along with scenarios that provide the information you will need to complete the forms. The items/services in the forms will be similar to what you already price for us. However, the scenarios may or may not match actual situations in your business.

  • Please use the information in the scenarios to complete the forms. If a scenario is significantly different from what you would normally experience, please let us know.

  • We are testing the form, not you, so any problems you have are problems with the form that we need to fix. We would like to hear all your comments and suggestions so we can improve the form as much as possible.

  • The information you provide today will only be used for the purposes of improving the form. It will not be used for any other purpose, and it is not part of the regular PPI data collection.

• Administer consent form.

• Have the participant complete the scenarios.

• Conduct the debriefing (see Attachment D).

• Thank the participant.

2. Remote Participants

Instructions during initial call

• Thank you for agreeing to participate.

• Explain cognitive testing

  • The goal of our session is for us to evaluate a new form we are developing for PPI. We want to see how you complete the form. We are looking for what works well and what is more difficult.

  • We will send you several simulated forms to complete, along with scenarios that provide the information to complete the forms. The items/services in the forms will be similar to what you already price for us. However, the scenarios may or may not match actual situations in your business.

  • Please use the information in the scenarios to complete the forms. If a scenario is significantly different from what you would normally experience, please let us know.

  • We will also send a consent form, which states that participation is voluntary, and that you may stop at any time.

  • As you are entering the data, please note how long it takes to complete each form.

  • We would like you to fax back the completed PPI forms along with the consent form.

  • We will then arrange a call with you, so you we can get feedback on your experience with the new form.

• Set up time to call back

Introduction during the test session

• Thank you for participating today.

• I am… [My colleague(s) <names> will also be listening today]

• Explain cognitive testing

  • We have a series of questions to ask you about the forms and your experience.

  • We are testing the form, not you, so any problems you have are problems with the form that we need to fix.

  • As a reminder, your participation is completely voluntary and you can decline to answer any question at any time.

  • The information you provide today will only be used for the purposes of improving the form. It will not be used for any other purpose, and it is not part of the regular PPI data collection.

  • We would like to hear all your comments and suggestions so we can improve the form as much as possible.

• Conduct the debriefing (see Attachment D).

• Thank the participant.

Attachment B: Sample Scenarios

Note: These samples were taken from a cognitive test for the PPI form we conducted using BLS employees. For these tests, we created scenarios to cover a wide range of conditions. In the test with actual respondents, we plan to tailor the scenarios to the types of items or services they report.

Scenario A

Your company produces grain combines. Sales of model number A123 have been low in the past year and your company discontinued producing this combine on March 1, 2012. The replacement grain combine, model B 1234, has an additional two rows and the cab has a premium windshield wiper and washer system. The price for model B1234 is $427,000. The mark up of both combines is the same. Model A123 cost you $150,000 to manufacture, while model B1234 costs you $200,000.

Scenario B

Your company produces a variety of lighting products, including flashlights. Over the last month your company has modified the flashlight’s housing, so it is now 4 ½ inches in diameter, and 7 inches long. The new model number is AB12, and the price of the new item is $53.

Scenario C

You are a large broker-dealer who specializes in making a marker in investment grade corporate bonds. You report to BLS the bid-ask spread for several corporate bonds. Several times throughout the year, you replace the bond that you report with a new bond from a similar company.

Using the information below, please fill out the form for BLS:

New Bond:

Marathon Oil Corp

Coupon Rate: 7.5%

Maturity: 2/15/2019

Credit rating: Ba

Ask: 118.19

Bid: 117.41

Attachment C: Sample Form

Attachment D: Debriefing Items

Complete questions 1 – 4 for each item or service.

For the in-person participants, we will ask these four questions after they complete each form, then start with question 5 (Q5) after they have finished all the forms. The remote participants will start with Q5, then insert the item-specific questions after Q13.

Q What steps did you take to complete the form? (Probe: What information did you report on the form?

Q Did you have any difficulty completing the form? (Probe: Was anything particularly confusing or challenging?)

• Yes

• No (go to Q4)

• Don’t Know/Not Sure/No Opinion (go to Q4)

Q What was difficult?

Q (this question will only be used as needed. The test moderator will review the form to determine if there are any other questions to ask, such as how the participant decided on a specific entry).

Q I’d like to begin by asking about your initial reaction to the form. (Probes: When you first saw the form, what did you think about it?)

Q How easy or difficult was it to complete the form? Very Easy, Easy, Neither Easy nor Difficult, Difficult, Very Difficult.

• Very Easy

• Easy

• Neither Easy nor Difficult

• Difficult

• Very Difficult

• No Opinion/Don’t Know/Not Sure

Q Did you look at the cover sheet at all? (check one)

• Yes

• No (go to Q10)

• Don’t Know / Not Sure / No Opinion (go to Q10)

Q How helpful was the cover sheet? Not At All Helpful, Slightly Helpful, Somewhat Helpful, or Very Helpful.

• Not At All Helpful

• Slightly Helpful

• Somewhat Helpful

• Very Helpful

Q When did you look at the coversheet? (check all that apply)

• Before starting

• When I wasn’t sure what to do

• Other, please specify

• Don’t Know / Not Sure / No Opinion

Q What was your general reaction to the coversheet? (Probes: When you first saw the coversheet, what did you think about it?)

Q Was there anything about the cover sheet that was particularly confusing or difficult to understand?

Q Was there anything about the cover sheet that was particularly helpful?

Q Do you have any other comments on the cover sheet?

For remote participants, Q1-Q4 will be here, repeated for each form

Instructions for remote participants: “Next I’d like to look at the form itself. Let’s start/continue with <brief item description & item code>.”

Q [Additional question for remote participants] Approximately how long did it take you to complete this form?

Q If you were completing these forms with real information, how much additional time would it take for you to locate the information you need for a single form?

Q Overall, what didn’t you like about the form?

Q Overall, what did you like about the form?

Q (this question will only be used as needed. The test moderator will review the discussion so far to determine if there are any other questions to ask).

Q Do you have any other comments that would help us improve the form?

Attachment E: Consent agreement form and Confidentiality statement

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data for the Producer Price Index.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

During this research you may be audiotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

12