2017 Census Test Goals, Objectives, and Success Criteria

2017 Census Test
Goals, Objectives, Success Criteria (GOSC) and Research Questions
November 21, 2016
Version 3.1
Scope Refinement
Pursuant to Decision Memo 2016.21
Contents
List of the Operations Participating in the Test and Program Managers ........................................................................................................................................................................................................................ 2
2017 Census Test Timeline................................................................................................................................................................................................................................................................................................. 4
Key Information ................................................................................................................................................................................................................................................................................................................. 5
Site Selection Criteria .................................................................................................................................................................................................................................................................... 5
Test Focus ..................................................................................................................................................................................................................................................................................... 5
Scope and Limitations ................................................................................................................................................................................................................................................................... 5
Operations.................................................................................................................................................................................................................................................................................... 5
Overall Test Assumptions .............................................................................................................................................................................................................................................................. 5
Test Focus Operations with Test Objectives and Research Questions .............................................................................................................................................................................................................................. 6
Support Operations with Goals, Objectives, Success Criteria and Research Questions .................................................................................................................................................................................................... 7
Support Operations Only – Summary of How the Operation will Support the Test ......................................................................................................................................................................................................... 9

Page 1 of 17

List of the Operations Participating in the Test and Program Managers
Operation/Support
Program Manager
Geographic Programs
Non-ID Processing

Evan Moffett

Program Management

Deidre Hicks

Census Questionnaire Assistance
Paper Data Capture
Forms Printing and Distribution

Sheila Proudfoot
Alexa Jones-Puthoff
Alexa Jones-Puthoff

IT Infrastructure
Systems Engineering and Integration
Security, Privacy and Confidentiality

Pete Boudriault
Raphael Corrado

Language Services
Internet Self-Response
Content and Forms Design

Jennifer Kim
Alexa Jones-Puthoff
Jennifer Kim

Response Processing

Raphael Corrado

2017 Census Test

Deborah Stempowski

Page 2 of 17

2017 Census Test Timeline – (10/25/2016)
10/28/2016
Full Package
to OMB

7/26/2016
Pre-Submission
to OMB

4/29/2016
Finalize GOSC
4/12/2016
Site Selection
announcement
& Deploy website

Sample Size
National Sample – 80,000

12/30/2016
OMB Approval

Content Reinterview Sample – 15,000
3/20/2017
Begin CQA

5/6/2016
SRR

11/14/2016
Printing and Mailing
TRR1

6/3/2016
CDR

5/16

4/1/2017

Census Day

6/16

7/16

8/16

9/16

10/16

11/16

12/16

4/10/2017 - 7/10/2017
Content Reinterview

1/17

4/1/2016

2/17

3/17

4/17

5/17

3/20/2017 - 5/12/2017
Self Response

5/5/2016
PBR

8/1/2016
ORR Kick-Off

1/3/2017
Self Response
TRR2

4/3/2017
Mailing 3
3/23/2017
Mailing 2
4/20/2017
3/20/2017
Mailing 1

Mailing 5
4/10/2017
Mailing 4

Page 3 of 17

6/17

7/17
7/31/2017

2017 Census Test

Focus of the
Test

Required to
Support the Test

NOT Included in
the Test

SUPPORT
Program Management
Program
Management

Systems
Engineering
and
Integration

Security,
Privacy, and
Confidentiality

Content and
Forms Design

FRAME
Geographic Programs

Infrastructure

Census/Survey Engineering
Language
Services

Decennial
Service
Center

Field
Infrastructure

RESPONSE DATA
Forms Printing and Distribution

Decennial
Logistics
Management

IT Infrastructure

PUBLISH DATA

Update
Enumerate

Nonresponse Followup

Data Products and
Dissemination

Local Update of
Census Addresses

Paper Data Capture

Group Quarters

Response Processing

Address Canvassing
(in-office only)

Integrated Partnership &
Communications

Enumeration at Transitory
Locations

Federally Affiliated Americans
Count Overseas

Internet Self-Response

Census Questionnaire
Assistance

Redistricting Data
Program
Count Review

Count Question
Resolution
Archiving

Non-ID Processing

TEST AND EVALUATION

OTHER CENSUSES
Island Areas Censuses

Coverage Measurement
Design and Estimation

Coverage
Measurement Matching

s Test Timeline20me

Page 4 of 17

Coverage Measurement
Field Operations

Evaluations and
Experiments

Key Information
Sample Selection
Description

Sites



Test Focus
Scope and Limitations
Operations

Focus of the test is:
 Test the feasibility of collecting tribal enrollment information
The 2017 Census Test cannot be generalized to the entire nation. The results do not predict national trends or rate estimates expected in the 2020 Census.
Test Focus Operations - Goals, Objectives, Success
Criteria and Research Questions:


Overall Test
Assumptions

A nationwide sample of 80,000 households for a self-response only operation, oversampled for areas with relatively higher concentrations of people estimated to identify as
AIAN (Source: Mike Bentley)
A subsample of 15,000 households for a reinterview operation.

Content and Forms Design

Support Operations - with Goals, Objectives, Success Criteria
and Research Questions:


Non-ID Processing

Support Operations Only – Summary on how the operation
will support the test:












Program Management
Systems Engineering and Integration
Security, Privacy and Confidentiality
Language Services
IT Infrastructure
Geographic Programs
Forms Printing & Distribution
Paper Data Capture
Internet Self-response
Response Processing
Census Questionnaire Assistance

1. Mailing package will include a paper questionnaire


Internet Choice Panel receives the paper questionnaire in Contact 1 (and again in Contact 4 if they have not yet responded)



Internet First Panel receives the paper questionnaire in Contact 4 if they have not yet responded

2. The National sample test is conducted through the Self Response contact strategy only [no nonresponse followup]

Page 5 of 17

Test Focus Operations with Test Objectives and Research Questions
Goals

Success Criteria
(should be quantifiable)

Objectives

Research Questions
Identify which are implementation vs research
questions?

Content and Forms Design (Jenny Kim)
Test feasibility of collecting Tribal Enrollment
information (offer option of collecting tribal
enrollment via Internet Self Response (SR),
Telephone SR (CQA), and Paper SR




Test Tribal Enrollment question.
Have a reinterview component to
assess the validity of the tribal
enrollment question fielded
(conducted by the NPC Call
Center).

The tribal enrollment reinterview
shows that the tribal enrollment
question captures tribal enrollment
in at least one federally- or staterecognized tribe(s).

Page 6 of 17

 What are the item and unit nonresponse
rates from the initial survey?
 What is the consistency rate for tribal
enrollment status when comparing the
initial survey response to the
reinterview?

Support Operations with Goals, Objectives, Success Criteria and Research Questions
Goals

Objectives

Success Criteria (should be
quantifiable)

Research Questions



Maximum number of Non-ID
responses match to a valid address
record in the Census universe during
real time processing.



What were the results from real-time
matching and geocoding Stateside
addresses during self-response?



What were the results from Asynchronous
Non-ID Processing (e.g., how many
additional matches and geocodes were
derived)?

Non-ID Processing (Evan Moffett and Frank McPhillips)
Conduct Real Time Non-ID Processing
(RTNP)



Implement real-time address
processing (standardization, MAF
matching, geocoding) for Stateside
addresses.

Conduct post-RTNP automated
matching and geocoding (also known as
Asynchronous Non-ID Processing)



Utilize administrative records data

to enhance respondent-provided
address data, and then make a
further attempt to match to an
MTdb record and/or derive a
census block geocode. This will
occur on a transactional basis for
each case not matched during RTNP
(i.e., individually, not in batches).

Additional Non-ID responses are
matched to MTdb records and/or
assigned to census blocks

Conduct manual Non-ID processing
concurrent with self-response
processing



Complete all outstanding manual

processing before the first cut for
UE workload, and keep up with
daily internet and CQA response
turnaround by the nominal date
when UE or NRFU operations would
have started (this will simulate a
2020 environment where Non-ID is

Manual processing catches up with

the backlog of cases not resolved
during automated processing by the
time the initial UE or NRFU cut
would have been taken. This will
give us a measure of how much Non- 
ID can reduce the UE or NRFU
workload before the operations

Page 7 of 17

What additional matches were derived
during manual Non-ID processing that
could have reduced the UE and NRFU
workload (e.g., how many, geographic
distribution, address characteristics, etc.)
What additional/updated geocodes were
derived during manual processing?

Support Operations with Goals, Objectives, Success Criteria and Research Questions
Goals

Objectives

Success Criteria (should be
quantifiable)

reducing UE and NRFU workload as
quickly as possible, given that selfresponse will remain open during
UE and NRFU).
Conduct office-based address
verification (OBAV) for eligible Non-ID
cases



Attempt to verify the existence and
census block location of all eligible
addresses from Non-ID processing
using geographic reference sources
in an office-base environment in
order to reduce field address
verification workload. Reference
sources may include available
online mapping and imagery
services.

Research Questions

even start.



The number of addresses verified for 
eligible Non-ID cases in the officebased operation is maximized.

Page 8 of 17

How many of the Non-ID cases eligible for
address verification could be verified in an
office-based operation?

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

Program Management (Deidre Hicks)
Define and implement program
Implement:
management policies, processes, and  Governance
the control functions for planning and  Strategic Communication
implementing the 2017 Census Test.
(internal/external)
 Change Management
 Knowledge Management
 Acquisition and Sourcing
Management
 Budget Management
 Schedule Management
 Human Capital Management Risk
and Issues Management
 Performance Measurement

Systems Engineering and Integration (Pete Boudriault and Daniel Lewis)
Gain experience moving to new IT
infrastructure – cloud computing

The Census Enterprise Data
Collection and Processing (CEDCaP)
Commercial Off-the-Shelf (COTS)
Platform and the real-time non-ID
processing systems will be hosted in
a commercial cloud.
Census will obtain metrics regarding
Page 9 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

cost versus performance as a result
of the testing to inform decisions
regarding future cloud
implementation, our ability to scale
nationally for 2020, etc.
Gain experience moving to new IT
infrastructure – services
Ensure the integration of new
systems into the Field Test that will
support the 2020 Census.

Utilize an enterprise data model.

Final list of systems supporting the field test will be determined once capability
requirements have been finalized and allocated to systems.

Integrate the following new systems
into the Test:
•

•
•

CEDCaP COTS Platform Enterprise
Census And Survey Enabling
Platform (ECaSE)
CQA
PEARSIS (Production Environment
for Administrative Records
Staging, Integration and Storage)

Each new system meets all functional
and non-functional requirements in
support of the Test.

Page 10 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

Security, Privacy, and Confidentiality (Pam Mosley, Pete Boudriault)
All planned security systems listed in
the scope for Security, Privacy and
Confidentiality are in place and
operating.

The following systems are in place
and operating:
•
•
•
•
•
•
•

Enterprise IT infrastructure and
Telecommunications Systems
Enterprise Security Operations
Center (SCC)
Enterprise Intrusion Detection
Systems (IDS)
Enterprise Security Vulnerability
management systems
Enterprise Audit Log Repository
Enterprise Operating System (OS)
Security Baseline Management
Application Security Baseline
Management System.

Language Services (Jenny Kim)
Support language needs of nonEnglish respondents.

Provide functionality enabling nonEnglish speakers to respond
 Internet self-response
Page 11 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

instrument (Spanish)
Paper (Spanish)
CQA aids for the CQA agents
in Chinese, Vietnamese,
Korean, Russian, Arabic;
Spanish instrument
Provide mailing materials in Spanish
that include:
 Letters, postcards



IT Infrastructure (Raphael Corrado and Pete Boudriault)
Provide the information technologyrelated infrastructure support to the
2017 Census Test, including:
•

Enterprise systems and
applications.

•

Decennial specific systems,
applications, and interfaces.

Geographic Programs (Evan Moffett and Ross Davis)
MAF/TIGER will provide MAF Extracts
and Geographic Reference Files to
support the operation.

Page 12 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

Forms Printing and Distribution (Alexa Jones-Puthoff and Mark Wolfram)
Print and distribute Internet
invitation letters, reminder postcards,
and questionnaire mailing packages
to support the 2017 Census Test
mailing strategy and enumeration of
the population.

Paper Data Capture (Alexa Jones-Puthoff and Heidi Brady)
Capture and convert data from the
2017 Census Test paper
questionnaires, including document
preparation, scanning, Optical
Character Recognition, Optical Mark
Recognition, Key from Image, and
checkout.

Internet Self-Response

(Alexa Jones-Puthoff and Jason Reese)

Test full mail contact strategies (5
mailings) in the mailout areas of the
national sample.



Refine final mail contact strategy
prior to the 2018 End to End Test.

Page 13 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Response Processing










Objectives

Issues/Constraints

(Raphael Corrado and Acting Branch Chief for Response Processing)

Establish the testing enumeration
universe
Manage the self-response
enumeration strategy
Distribute workload files required
for self-response enumeration
operation
Track self-response enumeration
status by case and support
determining the course of selfresponse enumeration based on
established business rules.
Perform required response data
collection process editing and
race/Hispanic origin coding
Perform required post-data
collection processing actions in
order to prepare the data for final
decennial response file



Evaluate the ability to support
the Data Collection Phase from a
response processing perspective
including Internet, Paper, and
CQA modes.
 Evaluate the ability to check for
sufficient data and identify datadefined persons.
 Evaluate support for the Non-ID
processing operation to
successfully transmit and receive
geocoding data from GEO.
 Evaluate the ability to conduct
required response data
processing steps, such as
standardizing demographic data
responses, race coding,
residence coding, determining
population counts, etc. in
preparation for future
unduplication work.
 Evaluate the ability to produce
Page 14 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals







Perform required steps to create
a Census Unedited File for data
analysis purposes, including
unduplicating data via the
primary selection algorithm and
performing count imputation.
Serve as the final test data
repository for input to required
analysis

Test methods for detecting
fraudulent responses

Objectives





Issues/Constraints

final data for analysis that has
completed an unduplication
effort using the primary selection
algorithm methodology
Evaluating the performance of
count imputation and creating a
Census Unedited File to be used
for test analysis and results.

Examine responses from
internet, CQA, and paper in an
attempt to identify potential
fraud, using multiple methods.



There is likely low motivation for submitting fraudulent responses during a
Census Test (as opposed to the 2020 Census), so we will likely have to rely on
simulated fraudulent responses to help test and refine our processes.

Page 15 of 17

Support Operations Only – Summary of How the Operation will Support the Test
Goals

Objectives

Issues/Constraints

Census Questionnaire Assistance (Sheila Proudfootand Kevin Zajac)
 Provide contact center
infrastructure and staff to handle
inbound assistance calls from
respondents (no web chat or
email capabilities in 2017)
 Assist respondents in completing
the Census questionnaire,
including capturing responses
 Answer questions about Census
processes and operations
 Automate certain tasks using
Interactive Voice Response (IVR)
self service solutions
 Assess the interface(s) between
the CQA contractor and Census
Bureau’s systems

Page 16 of 17

Document History
Version

Date

Changes

1.0

May 4, 2016

1.1

September 21, 2016

Draft for review incorporating updates since May.

2.0

September 30, 2016

Baseline 2 incorporating updates since May.

3.0

October 19, 2016

3.1

November 21, 2016

Baseline prior to Project Baseline Review (PBR)

Refinement of scope per 2020 Census Decision Memo 2016.21
Added detail for response processing.

Page 17 of 17