One Pager

Request for Approval under the “Fast Track Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0704-0553)

TITLE OF INFORMATION COLLECTION: Artificial Intelligence (AI) Customer Usability/Acceptance

PURPOSE:

BLUF: Assess customer usability for a notional AI recommendation agent (Artemis).

Problem Statement: Use of Artificial Intelligence (AI) agents to provide recommendations is increasingly commonplace. This is a function of an explosion of data availability as well as budget cuts, and is true for both the federal (e.g., analytic AI agents) and private sectors (e.g., Amazon proposed products). However, it has been documented that customers of these services both over-accept (Logg et al., 2019) and over-reject (Dietvorst, 2016) recommendations compared to what is optimal or what they would have otherwise chosen themselves. This results in significant loss (time, accuracy, money, etc.) to both customers and organizations providing these services.

Task: In this questionnaire we propose to ask paid panelists (Amazon Mechanical Turk workers, $0.35/per) a series of questions to better design future AI agents for customer acceptance/user experience design. This task serves as an analog for customer acceptance/rejection of AI contributions to analytic tasks, but is intended for wider AI services application (e.g., business services, etc.), which is why we seek to survey a general audience vs. specifically analytic one. Specifically, we ask them to pretend they are film producers, and we ask them to look at a series of fake movie posters. Based on a recommendation of an AI agent (Artemis), we ask panelists to either accept or reject the proposed movies for publication. Following this short task, panelists complete a short set of questions assessing behavioral aspects of their experience. The survey should take each panelist approximately 5 minutes to complete.

DESCRIPTION OF RESPONDENTS: Respondents will be paid ($0.35) panelists from Amazon’s Mechanical Turk service.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [] Customer Satisfaction Survey

[X] Usability Testing (e.g., Website or Software [ ] Small Discussion Group

[ ] Focus Group [] Other: __ ______

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name:____Stephen Hood____________________________

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [X ] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ X ] Yes [ ] No

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Individuals or Households	400	5 minutes	33.33hrs
Totals	400		33.33hrs

PUBLIC COST: The estimated annual cost to the public is _$0.00___________

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [X ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

The universe of potential respondents is comprised of voluntary (paid, in our case $0.35/response) US panelists from a service provided by Amazon (Mechanical Turk). Previous research has shown that this population is representative of the larger US population, and so generalizes well. Because this is a paid service we do not have access to a customer list.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[ X ] Web-based or other forms of Social Media

[ ] Telephone

[ ] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [ ] Yes [ X ] No

3