3090-0297-Template_USA-gov-Boland_(Req-22)

Request for Approval under the “Generic Clearance for the Collection of
Routine Customer Feedback” (OMB Control Number: 3090-0297)
Req-22

TITLE OF INFORMATION COLLECTION:
Improving USAGov Communications for Email Subscribers

PURPOSE:
USAGov aims to increase relevance of USAGov's email and notifications’ content, structure, and
design to better serve the public. Additionally, through this work, USAGov and OES aim to build
the capacity of the USAGov team to conduct ongoing rapid evaluation, in support of ​OIRA
Memorandum “Behavioral Science Insights for Federal Forms”​ that encourages agencies to
implement rapid, no/low-cost randomized controlled trials and other rigorous evaluation
methods to measure the most effective and efficient design and structuring of information
collections, easing respondent burden, reducing misunderstanding, and improving data quality.
DESCRIPTION OF RESPONDENTS​:
Each day, USAGov and other websites on the USA.gov domain receive 130,000 unique visitors
(2.8 million per month). Some of these visitors subscribe to the USAGov email list(s) to receive
additional information from the Federal government. However, little is know about these
recipients - their demographic information or their interests, or which particular area(s) of the
broad topics USAGov covers that are of particular relevance to them.
Subscribers of USAGov emails will have the opportunity to complete a survey. This will also
enable USAGov to better satisfy the customer satisfaction metrics required by the Digital
Government Strategy. Participation is voluntary and anonymous.
TYPE OF COLLECTION:​ (Check one)
[ ] Customer Comment Card/Complaint Form
[ ] Usability Testing (​e.g.​, Website or Software
[ ] Focus Group

[x] Customer Satisfaction Survey
[ ] Small Discussion Group
[ ] Other:​ ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does ​not​ raise issues of concern to other federal
agencies.
4. The results are ​not​ intended to be disseminated to the public.
5. Information gathered will not be used for the purpose of ​substantially​ informing ​influential
policy decisions.
6. The collection is targeted to the solicitation of opinions from respondents who have
experience with the program or may have experience with the program in the future.
Name: ​Amira Choueiki Boland (Office of Evaluation Sciences, 202-412-9064), Victoria Wales
(USAGov, 202-812-5032 )

To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Is personally identifiable information (PII) collected? [ ] Yes [x] No
2. If Yes, will any information that is collected be included in records that are subject to the
Privacy Act of 1974? [ ] Yes [x] No
3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [x] No
Gifts or Payments:
Is an incentive (​e.g.​, money or reimbursement of expenses, token of appreciation) provided to
participants? [ ] Yes [x] No
BURDEN HOURS:
Category of Respondent
Total (Spread over multiple waves)
Individuals
Totals

No. of Respondents
44,280*
44,280*

Participation
Time
Appx. 2
minutes**
Appx. 2
minutes**

Burden
1476
hours
1476
hours

*Based on a sample of 737,994 individuals and an average completion rate of 6% , which is the current click-through rates for USAGov emails
**​Qualtrics calculation of current survey design (questions below)

FEDERAL COST:
Costs are negligible as surveys can be developed and analyzed as part of business-as-usual
activities of the Office of Evaluation Sciences and USAGov Outreach team. We estimate the
federal cost will be $3,305 (3 OES and USAGov team members at an average GS-14-1 x 20
hours each). These costs are based on time for survey design and development.
The cost of survey implementation, using Qualtrics, is also part of business-as-usual activities at
GSA. GSA estimates a cost of $0.37 for one Qualtrics survey response​. ​Since we estimate there
will be 44,280 responses, we expect a total cost of survey implementation of $16,384. However,
this dollar amount is within the GSA Enterprise contract with Qualtrics so at this time no new
dollars need to be allocated to this survey.
The total cost estimate is $19,689.
If you are conducting a focus group, survey, or plan to employ statistical methods, please
provide answers to the following questions:
The selection of your targeted respondents
1. Do you have a customer list or something similar that defines the universe of potential
respondents and do you have a sampling plan for selecting from this universe?
[x] Yes[ ] No
If the answer is yes, please provide a description of both below? If the answer is no, please
provide a description of how you plan to identify your potential group of respondents and how
you will select them?
1

See charts above and below of current subscribers.
Sampling Plan
USAGov has a total of 737,994 unique email subscribers. Of these subscribers we know:
○ Whether they are considered “active” or “inactive” (“inactive” means a
subscriber has not opened an email in the last 16 emails)
○ Which email lists they are subscribed to (Education, Health, etc.)
○ Whether they receive their emails in English or Spanish
Working with the Office of Evaluation Sciences, the USAGov team will conduct a randomized
controlled trial to build evidence about which messages presenting the survey to subscribers
generates the most open, click-through, and ultimately survey completion rates.
We will randomly assign each subscriber who receives emails in English into one of six message
arms - five “treatment” arms receiving a message informed by behavioral science insights and
one business as usual message after “blocking” on the three factors described above.
Subscribers who receive emails in Spanish will be assigned to a subset of the six messages, two
“treatment” arm messages informed by behavioral science insights and the business as usual
message, which could be meaningfully translated into Spanish.
We will conduct the randomization using R or Stata statistical software and randomization code
developed by OES team members. This code will be documented in an experimental design
document.
Current subscriber counts:
English
Total

Active

Inactive

Open
Rate

Total unique emails where we will
send english survey

646,213 283,351 362,862 N/A

Benefits, Grants, and Loans

276,705 107,743 168,962

Business

81,802

56,584

25,218

Click Rate
N/A

32.62

2.23 Biweekly
Dependent on Season [2 /
week in hurricane, bad snow
4.36 storms, extreme temps]

20,557

34,629

28.93

84,581

90,042

24.89
29.9

6.83 Biweekly

174,623
Education

15.01 Weekly

40.62

55,186
Disasters and Emergencies

Frequency Emails Are Sent

2 / Week [1 Teachers, 1
12.62 Parents [separate lists]]

Health

227,845

85,316 142,529

Housing

113,890

42,812

71,078

32.07

10.2 Biweekly

Jobs and Unemployment

143,569

57,654

85,915

36.15

20.78 Biweekly

79,319

31,295

48,024

36.42

11.82 Monthly

Money and Shopping

309,601 111,498 198,103

32.65

13.77 Weekly / 2/Week

Travel and Immigration

105,377

41,831

63,546

32.31

16.49 Biweekly

55,429

22,178

33,251

0

0 Seasonal

Military and Veterans

Voting and Elections

2

Spanish
Inactive

Open
Rate

Total

Active

Total unique emails where we will send
spanish survey

Click Rate

91,781

31,146

Beneficios y programas de ayuda del
Gobierno

12,010 -

Desastres y emergencias

47,102

15,371

31,731

24.5

Educación

20,036

5,804

14,232

0

Salud

33,345

9,950

23,395

25.1

1.1 Monthly

Vivienda y comunidad

60,199

19,101

41,098

27.3

4.48 Quarterly

Trabajo y desempleo

22,936

6,949

15,987

29.47

3.65 Monthly

Militares y veteranos

12,339

3,378

8,961

22.88

1.47 Quarterly

Dinero y temas del consumidor

26,127

8,063

18,064

33.19

6.63 Monthly/Bi-Monthly

Ciudadanía, inmigración y viajes

23,579

7,449

16,130

20.32

4.44 Quarterly

Votaciones y elecciones

12,314

3,391

8,923

0

0 Seasonal

Todos los temas (Spanish)

84,636

29,385

55,251

17

60,635 N/A

Frequency Emails Are Sent

N/A

12,010

Rarely
1.1 Sporadically/Seasonal
0 Rarely

3 Sporadically

The Goal
The goal of this project is twofold: learn more about who is accessing USAGov content and
resources (by respondents responding to the survey) and determine which communications about
surveys are most effective at increasing completion (by sending out this survey through six
different email messages). Because other agencies across government often conduct similar
activities, findings from this pilot can inform their efforts as well.
Right now, USAGov has little information about things like the age, regional location, or even
subject area of interest of the people opening their messages and clicking on their websites. If
USAGov had more details about their subscribers, they could tailor their content and strategies
to better serve the public. For example, if USAGov learns that there is a large population of
young veteran email subscribers, USAGov could increase the amount of content and reshape
their messaging on subjects like the GI Bill, PTSD resources, veteran startup grants, and home
loan benefits.
Sending surveys to learn more about customers is a common practice across private sector and
even other government websites. Many government agencies contract with “​ForSee​,” a
customer experience firm that helps agencies optimize their websites. An example of a
government entity that creates and embeds their own single-question surveys and pop-up
opportunities for feedback is ​https://www.stopbullying.gov/get-help-now/index.html​. This pilot
will help determine the type of communications that are effective at increasing feedback from
customers. OES will share aggregate results on which communications are most effective to
help other agencies improve customer experience and feedback.
Administration of the Instrument
1. How will you collect the information? (Check all that apply)

3

[x] Web-based or other forms of Social Media
[ ] Telephone [ ] In-person
[ ] Mail [ ] Other, Explain
2. Will interviewers or facilitators be used? [ ] Yes [ x] No
Enclosure:
USAGov - Survey Instrument and Email Communications (English)
USAGov - Survey Instrument and Email Communications (Spanish)

4