Focus Group Supporting Statement B

Supporting Statement for Paperwork Reduction Act Submission

Certification Statement for Recovery Accountability and Transparency Board

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses or employ statistical methods” is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Total focus group participants: 72 (18 people in each of 4 markets)

Participants will be average American citizens that provide a diverse community based on the following criteria:

• Registered voter

• Government-related activities

• Awareness of ARRA

• Internet usage

• Current work status/occupation

• Level of education

• Ethnicity

• Household income

Reference Attachment A: Participant Screener

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Discovery: Focus Groups

To “field” test Recovery.gov enhancements, contractor will conduct qualitative research, specifically focus groups, representative of key user populations. Sufficient focus groups will help us engage audiences in a safe (non-threatening) environment, where thorough dialogue, group dynamics and observation of tasks, we can understand how these individuals, from specific audience profiles, might use the site to meet their needs.

The facilitated focus groups will provide probing opportunities so we can truly understand the “why” behind participant behaviors (actions) in the course of seeking information on the site as well as perceptions of the site. We believe that this approach can validate the design concepts and content while gathering any constructive feedback to complete site implementation in the most timely and efficient manner.

We recommend conducting 8 in-person focus groups in 4 geographically representative regions: Northeast, Southeast, Midwest, and West.

Reference Attachment B: Usability Testing Task List

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

A participant screener (Attachment A) will be developed to ensure focus group participants are within the specified audiences of concern for site success. Participants, except where not allowed for legal or ethical reasons, will receive an industry standard stipend of $100 each, delivered by check upon completion of the focus group or usability testing session.

In addition to general diversity in the markets, locations for the research were selected based on:

• Metropolitan Statistical Area: mix of metro, suburban, rural

• Range of unemployment rates

Market	Unemployment Rate	MSA Rank
BALTMORE	7.2	20
MEMPHIS	10.40	50
DETROIT	14.8	41
PHOENIX	8.4	12

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

Focus groups will be a two hour guided session that follows a pre-determined discussion guide. The recommended discussion guide is Attachment B.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Rob Groat

Chief Technology Officer

Office of the CTO

Smartronix, Inc

301.535.7703

Lindsay Wozniak

Senior Vice President

TMP Government

703.269.0095

Attachment A: Participant Screener

Record Gender: recruit a mix

Male

Female

NAME

ADDRESS

CITY/STATE/ZIP

HOME PHONE WORK PHONE

FAX E-MAIL

RECRUITED BY DATE

CONFIRMED BY DATE

PROJECT OVERVIEW

• Recruit 13 for at least 9-10 to participate per group

• Represent a mix of ethnicity, employment status, gender, age, education level, U.S. born vs. foreign born, and English as primary/secondary languages.

• Phase 1 Focus Groups will focus specifically on Average Citizens and Interested Citizens

• Ensure 508 compliance representation

Proposed Research Locations/Schedule

Locations recommended based on the following:

• Media market population: mix of metro, suburban and rural areas (based on Metropolitan Statistical Area rankings)

• Diversity in markets: gender, ethnicity, disabled residents

• Range of unemployment rates: high, low (based on BLS June 2009 data)

Day	Date	Region	Times	Market	Unemployment Rate	MSA Rank
Thursday	July 8	Northeast	5:30 and 7:30 p.m.	BALTMORE	7.2	20
Monday	July 12	Southeast	5:30 and 7:30 p.m.	MEMPHIS	10.40	50
Tuesday	July 13	Midwest	5:30 and 7:30 p.m.	DETROIT	14.8	41
Wednesday	July 14	West	5:30 and 7:30 p.m.	PHOENIX	8.4	12

Target Audience

• Group 1: Average citizens/Interested Citizens

• Group 2: Potential funding recipients/Recipients/Citizens that use Recovery.gov in a professional capacity.

And represent a mix of ethnicity, employment status, gender, age, education level, U.S. born vs. foreign born and English as primary/secondary languages.

Recruiting notes:

Since we are testing creative, we will not be able to include any no/low vision 508 compliance respondents

Hello, my name is _________________from C² Consumer Research. I am calling to invite you to join a paid market research study to discuss government websites. We will pay you $100 to simply share your thoughts and opinions for a little less than 2 hours of your time. This is not a sales presentation or recruiting event. It is merely for market research purposes. We have a number of evening sessions available based on your interests.

VM: If interested or would like to get more information please call us at (877) 222-2909 Ext. xx again that’s (877) 222-2909 Ext. xx.

Live: Does this sound like something you would be interested in participating in? (if yes) Is this a convenient time to ask you a few questions?

a. Yes 1 CONTINUE

b. No 2 ESTABLISH CONVENIENT TIME

TO CALL BACK

1. When was the last time, if ever, you participated in a market research study either at our facility or for another research company? (DO NOT READ)

a. Within the past 6 months 1 TERMINATE

b. Longer than 6 months ago 2 ASK Q2a/2b

c. Never 3 SKIP TO Q3

If within the past 6 months, terminate

2a. How many market research group discussions have you ever participated in?

2b. How many market research one-on-one, in-depth interviews have you ever participated in?

No more than 3 focus groups or one-on-one interviews ever

3. Which of the following categories includes your age? (READ) Recruit a mix.

a. Under 18 1 TERMINATE

b. 18-24 2 CONTINUE

c. 25-34 3 CONTINUE

d. 35-45 4 CONTINUE

e. 45-54 5 CONTINUE

f. 55-64 6 CONTINUE

g. 65+ 7 CONTINUE

4. Which of the following best describes your current residency status?

a. I am a U.S. Citizen 1 CONTINUE

b. I am a Naturalized U.S. Citizen 2 CONTINUE

c. I am not a U.S. Citizen 3 TERMINATE

5. What languages are spoken in your home? ___________________ Recruit a mix of English only and bilingual households

6. On a scale from 1 to 10, where 1= not very interested and 10= very interested, how interested are you in the following subjects?

a. Professional Sports (such as football, baseball, basketball, racing)

b. Fashion & Beauty

c. Internet and/or social networking websites (such as Facebook, Twitter, YouTube)

d. Travel

e. Gardening/Home Decoration/Architecture

f. Politics and government affairs

7. Are you a registered voter?

a. Yes 1 CONTINUE

b. No 2 CONTINUE

8. Which of the following activities have you ever participated in? Again, I want to stress that we are looking for a wide array of Americans so please be as honest as possible.

(Note to recruiter: If A-E—may qualify for Potential funding recipients, Recipients, Interested Citizens quotas. If F—Continue for Average Citizen Quota

a. I voted in the last General Election (November 2008)

b. I follow local, state and national politics closely

c. I have contacted an elected official regarding my thoughts on current events and government policies

d. I have searched the internet/watched TV/listened to the radio/read the newspaper/journals for more information on current events and government policies

e. I have campaigned/volunteered/actively supported initiatives of personal interest

f. None of the above—categorize as Average Citizen

9. Please share an example of your level of inquiry/interests_________________________

10. Are you aware of the American Recovery and Reinvestment Act of 2009 often referred to as the economic stimulus package or Recovery Act?

a. Yes 1 CONTINUE

b. No 2 CONTINUE

10b. What is it? ___________________________________________________________

11A. Do you use the internet?

a. Yes 1 CONTINUE

b. No 2 CONTINUE, SKIP TO 12

11B. If Yes, where do you access the internet?

a. Work 1 CONTINUE

b. Home 2 CONTINUE

c. Library 3 CONTINUE

d. Pay per use (e.g. internet café) 4 CONTINUE

12. On a scale from 1 to 10, where 1=not very comfortable and 10=very comfortable, how comfortable are you with using the Internet? _____________________ (Recruit a mix.)

12B. Which of the following government websites have you ever visited?

a. IRS.gov 1 CONTINUE

b. WhiteHouse.gov 2 CONTINUE

c. Recovery.gov 3 CONTINUE

d. Weather.gov 4 CONTINUE

e. Census2010.gov 5 CONTINUE

f. Other (specify) 6 CONTINUE

The following questions are intended to ensure we get a wide variety of respondents represented in our research study

Note to recruiters: Targets to include in the focus groups

• Group 1: Average citizens and Interested Citizens

• Group 2: Recipients and Potential funding recipients

13. Which of the following best describes your current work status or occupation? (Read. Recruit a mix.)

1. Professional/Executive/Owner

2. White Collar/ Administrative/ Clerical

3. Blue Collar/ Service/ Manufacturing 

4. Part Time Work Outside of the Home

5. Stay at Home Full Time/Do Not Work

6. Unemployed/Looking for Work

7. Student

8. Retired

9. Other_____ (Record verbatim, do not read.)

10. Do Not Know (Do not read.)

11. Refused (Do not read.)

If A-D ask:

13a. What is your current job title? _________________________________

13b. What is the name of your current employer? _________________________________

14. Have you ever researched or applied for government funding (e.g. grants, contracts or loans) on behalf of your organization?

1. Yes

2. No, I do not foresee the need to apply

3. No but we have considered it—continue for potential funding recipient

15. Did your organization receive government funding/grants/loans/contracts?

1. Yes—Classify as Funding Recipient

2. No—Classify as Potential Funding Recipient

The following questions are merely for classification purposes only.

16. Which of the following best describes your last level of education completed? (READ. Data collection purposes only.)

a. No high school diploma 1 CONTINUE

b. High School 2 CONTINUE

c. Associate’s Degree 3 CONTINUE

d. Bachelor’s Degree 4 CONTINUE

e. Masters Degree 5 CONTINUE

f. JD/PhD/MD 6 CONTINUE

g. Post Doctorate 7 CONTINUE

17. Which of the following best describes your ethnicity? (READ. Recruit a mix.)

a. Black/African American 1 CONTINUE

b. Asian 2 CONTINUE

c. White 3 CONTINUE

d. Native Hawaiian or other Pacific Islander 4 CONTINUE

e. Hispanic/Latino 5 CONTINUE

f. American Indian or Alaska Native 6 CONTINUE

g. Other 7 CONTINUE

h. Decline to state 8 CONTINUE

18. For classification purposes only, please tell me into which of the following categories your TOTAL HOUSEHOLD income falls before taxes? (READ LIST.)

a. Under $35,000 1 CONTINUE

b. Between $35,000 and $45,000 2 CONTINUE

c. Between $45,000 and $60,000 3 CONTINUE

d. Between $60,000 and $80,000 4 CONTINUE

e. Between $80,000 and $100,000 5 CONTINUE

f. Over $100,000 6 CONTINUE

g. Refused 7 CONTINUE

19. To ensure our research program is compliant with Section 508 of the Rehabilitation Act and accessible to all segments of the US population, please indicate which of the following applies to you:

(If possible—low/no vision respondents may not be possible for in-person creative concept testing)

a. No vision 1 CONTINUE

b. Low vision 2 CONTINUE

c. Hearing impaired 3 CONTINUE

d. Motor impairments 4 CONTINUE

e. Speech impairment 5 CONTINUE

f. Spinal impairment 6 CONTINUE

g. None of the above 7 CONTINUE

INVITATION

We would like to invite you to participate in a paid market research study to discuss government websites. The focus group is scheduled to last less than 2 hours and will take place on [insert date based on market].

Are you interested in participating in the research discussion?

a. Yes CONTINUE

b. No TERMINATE

That is great, the session will last approximately 2 hours. For your participation we will pay you $xxx. Does this sound like something you would be interested in participating in?

We are investing a great deal of time and effort to coordinate this research study. Your participation and attendance is vital to the success of our study. If for any reason, you are unable to attend the focus group, please notify us in advance so we may find a replacement.

Get mailing address, fax number or email address to send confirmation information.

We are counting on your attendance.  Please do not schedule yourself for this research if you feel that you are not going to be able to attend the focus group discussion. If an emergency does arise, it is imperative that you call us immediately at (877) 222-2909 Ext. xx as we will need to find your replacement as soon as possible. Since our clients expect our respondents to all show up for their research, and since it is necessary that we provide them with the most accurate information, you will be receiving 2 calls confirming your attendance. The first call will be 3-4 days after you are recruited and we will need to re-ask you some of the screening questions, you will also receive a final call to confirm the day before your scheduled appointment. We do need to speak with you on each of these occasions, and we appreciate your prompt return calls.  

Do you have any objection to us placing the 2 calls to confirm your information and attendance?

a. Yes TERMINATE

b. No CONTINUE

You will also be videotaped for market research purposes only during this discussion. Do you have any objection to the use of video?

a. Yes TERMINATE

b. No CONTINUE

If you wear glasses please be certain to bring with you to the session.

Thank you for your understanding and patience during these confirmation calls, and we look forward to having you participate in this study!

Attachment B: Focus Group Discussion Guide

Note to moderator: GROUP MAY NOT EXCEED 2 HOURS IN LENGTH. PLEASE MANAGE TIME ACCORDINGLY.

Introduction (10 minutes)

Hello my name is ____. I want to start off by letting you know that I am an independent moderator. What that means is that I do not work for [insert name of focus group facility]. I am not looking for you to say one thing or feel another. In fact, I am just here to guide the conversation along. There are no right or wrong answers.

I would also like to point out a few things before we get started. I am video taping this group so I could focus on our conversation and I do have some of my colleagues behind the glass taking notes for me. So please feel free to be candid as your individual information will be kept confidential as it relates to this project.

With regard to cell phones - If you have a cell phone with you tonight, please turn it to vibrate so it does not disrupt the flow of our conversation today.

Let’s start off by going around the room. Please introduce yourself, where you are originally from and how long you have lived in [insert city].

Section I: Awareness/Opinions of American Recovery and Reinvestment Act of 2009

But before we get to that point I would like to learn more about your thoughts on the current state of the economy.

Discuss what they know about the economy stimulus and road to recovery.

What do you know about the $787 billion stimulus package? What types of programs or projects is it funding?

What resources do you currently use/or would you use to learn more about the recovery process and allocation of the $787 billion stimulus package? (Create list on flip chart. See if Recovery.gov is mentioned on an unaided basis.)

If not mentioned, how many of you are aware of a website called Recovery.gov?

Section II. Design the Recovery.gov website

Transition: We are conducting this research to better assess what folks just like yourself might like to find on a website called Recovery.gov.

Read description about Recovery.gov: Recovery.gov is the U.S. government’s official Web site providing easy access to data related to Recovery Act spending and is a forum for reporting fraud, waste, and abuse.

Section III: Reveal Enhancements to Recovery.gov

Bring up Recovery.gov/ on a large television monitor.

Conduct an ad-hoc usability study critique of the existing site

Ask respondents to write down their initial response to the homepage. Where would they click first?

Probe all elements of a traditional usability test including homepage, content, navigation, nomenclature.

• Basic user information location and retrieving

• Ask respondents to look at each content category/label and supporting topics

• Discuss if headers/labels and supporting content is clear/intuitive

• Top-line navigation

• Aesthetics

• Economy, i.e. activity was worth the effort for reward achieved

• Total site architecture

• Data visualization components

Assess ways to improve the site.

Section V. Conclusion

In closing, please offer one piece of advice to improve Recovery.gov.

Thank respondents again for their time and participation. Please ask them to leave their paperwork behind.

23