2018EndtoEndCensusTest_NRFU_Study Plan

Study Plan for the 2018 End-to-End Census Test Operational Assessment Nonresponse Followup Integrated Project Team Draft Pending Final Census Bureau Executive Review and Clearance.
	Version 0.7 August 4, 2017

[This page intentionally left blank.]

Table of Contents

I. Introduction 3

II. Background 3

III. Assumptions 8

IV. Methodology 10

V. Scope of Assessment Content and Questions-To-Be-Answered 13

VI. Knowledge Management Resolutions 26

VII. Risks & Limitations 26

VIII. Measures of Success 28

IX. Data Requirements 30

X. Division Responsibilities 31

XI. Study Plan & Assessment Report Schedule 32

XII. Issues That Need to be Resolved 34

XIII. Review/Approval Table 35

XIV. Document Revision and Version Control History 35

XV. Glossary of Acronyms 35

XVI. Field Operational Control System Alerts 37

XVII. Enterprise Event Codes used During NRFU 38

XVIII. References 40

1. Introduction

The 2018 End-to-End Census Test (2018 E2E CT) is an important opportunity for the U.S. Census Bureau to ensure an accurate count of the nation’s increasingly diverse and rapidly growing population. It is the first opportunity to apply much of what has been learned from census tests conducted throughout the decade in preparation for the nation’s once-a-decade census of population and housing. The address canvassing operation for the 2018 E2E CT will be held in three locations: Pierce County, Washington; Providence County, Rhode Island; and the Bluefield-Beckley-Oak Hill, West Virginia area. The remaining operations in scope for the test will take place in Providence County only. Peak operations will commence with the self-response operation in March 2018, followed by the nonresponse followup operation beginning in May 2018.

Providence County is an ideal community to simulate a microcosm of the 2020 Census experience, as its demographics mirror those of the nation. The county has a population of over 600,000 with more than a quarter-million housing units. It includes historically hard-to-count populations and has a large Hispanic presence. Providence County was the single site that provided a thorough ability to test all of the systems and operations planned for the 2018 End-to-End Census Test. As such, we remain confident that the 2018 End-to-End Census Test is sufficiently robust to test the systems and operations that must be tested.

The 2018 E2E CT will begin on August 28, 2017 with the start of In-Field Address Canvassing, and it will conclude on July 31, 2018 with the end of NRFU Reinterview (NRFU RI). Census Day for the test will be April 1, 2018 (U.S. Census Bureau, 2017b).

The 2018 E2E CT will test and validate the 2020 Census operations, procedures, systems, and field infrastructure to ensure proper integration and conformance with functional and non-functional requirements. The test also will produce a prototype of geographic and data products, and will validate the 2020 Census design and cost estimate (U.S. Census Bureau, 2017a). The 2018 E2E CT results are based on one test site that was purposely selected and cannot be generalized to the entire United States. Additionally, because it is not conducted in a “full decennial census environment,” the results may not replicate the results to be obtained in the 2020 Census.

This study plan documents how the Nonresponse Followup (NRFU) operation will be assessed, as guided by questions to be answered.

2. Background

The goal of the NRFU operation is to determine the housing unit status for non-responding addresses, and to enumerate those that were occupied on Census Day (April 1). Historically, the NRFU operation has been the largest and most expensive field operation of the decennial census. During the 2010 Census, the operation consisted of in-person interviews by enumerators to all addresses that did not return a paper census questionnaire. The NRFU workload consisted of approximately 47 million housing units with more than 500,000 enumerators who completed at least one interview. The cost of the 2010 NRFU operation was about $1.6 billion (U.S. Census Bureau, 2012).

One of the key developments since the 2010 Census for the NRFU operation was the creation of an enterprise solution that uses information technology and an electronic interface to collect response data, while automating and integrating the processes of case assignment, case management, data collection, and post-data collection. The new system replaced paper-based operations, providing a faster, more accurate, and more secure means of data collection for the 2020 Census. Other NRFU improvements since 2010 include enhanced contact strategies for multiunits (e.g., apartment buildings, condominiums), simplified procedures for conducting interviews with proxy respondents, refined case removal strategies, and development of a maximum attempt-day threshold for household visits (U.S. Census Bureau, 2017e).

The goals of the NRFU operation for the 2018 E2E CT are to:

• Use administrative records (AR) to inform and reduce the NRFU workload,

• Evaluate refined field operations, including a reengineered operational control system that optimizes case assignments and routing,

• Measure the efficiency and effectiveness of the field staff structure and workload, and

• Evaluate the cost of the operation and the quality of response data obtained.

The NRFU operation will start on May 9, 2018, and end on July 24, 2018. A sample of all NRFU cases will be selected for a quality control Reinterview during NRFU RI, with the aim to detect and deter data falsification by enumerators. NRFU RI will start on May 10, 2018, and end one week after the end of NRFU, on July 31, 2018 (U.S. Census Bureau, 2017c).

After the NRFU operation starts, a supplemental workload will be added to NRFU. The supplemental workload will consist of new addresses from the U.S. Postal Service’s Delivery Sequence File (DSF). New addresses added during the Update/Leave operation that do not self-respond will also become part of the NRFU workload.

1. Post-2010 Census NRFU Testing

Post-2010 Census testing of operational improvements and cost-saving measures for NRFU began with the 2013 Census Test in Philadelphia, Pennsylvania. The aim of the test was to pilot several novel methods that had the potential to reduce costs associated with the NRFU operation. The methods explored in the 2013 Census Test included: 1) using administrative records to reduce the NRFU workload; 2) reducing the number of contact attempts made by enumerators; 3) using an adaptive case management strategy to control in-person enumeration visits; and 4) making initial enumeration contact attempts via telephone. The results of this test reaffirmed the potential of these methods to reduce costs for the 2020 Census. Recommendations from the 2013 Census Test included refinement of the contact strategies for proxies, and optimization and prioritization of cases for enumerators (U.S. Census Bureau, 2014).

The 2014 Census Test took place in parts of Montgomery County, Maryland, and Washington, D.C. The NRFU component consisted of four panels designed to continue research to improve efficiency and reduce costs of the operation by modifying contact strategies for in-person visits; streamlining operations to promote efficiencies; and using administrative records to reduce the NRFU workload and to inform, replace, or augment self-response. The 2014 Census Test found that the use of administrative records reduced the NRFU workload and improved the determination of unit status. Recommendations from the test included assessing the quality of proxy interviews, improving the automated data collection instrument, and investigating refined contact strategies for restricted access cases (Poehler et al., 2015).

The 2015 Census Test was conducted in Maricopa County, Arizona, and was a proof-of-concept field test for a reengineered 2020 Census. The NRFU operational objectives were as follows: 1) testing a new field control system and new field data collection tool; 2) testing a real-time operational control system to integrate and manage operations; 3) using smart phone technology for census field enumeration (including the Bring Your Own Device – or BYOD – initiative); and 4) continuing to use administrative records and third-party data to inform and supplement field data collection efforts. Recommendations from the test included the need to consider a new approach to enumeration of multiunit households, refined instrument pathing for proxy and non-interviews, and elimination of the BYOD option for future tests (Hatcher, 2015).

The 2016 Census Test was conducted in portions of Harris County, Texas, and Los Angeles County, California. The objectives of the NRFU operation during this test were to continue refinement of reengineered field operations, improve the efficiency and effectiveness of field staff and workload management, utilize administrative records to reduce the NRFU workload, and incorporate reengineered quality assurance methods. Two new aspects of the NRFU operation for the 2016 test were the implementation of manager visits (MV) at multiunits to obtain the occupancy status of individual units and an enhancement to the enumeration application that prompted the enumerator to begin attempting a proxy interview, when the case became eligible. The results of the test indicated that manager visits effectively reduced the nonresponse followup workload in multiunits and the burden on respondents. The test also showed that additional emphasis was required during enumerator training on the importance of attempting proxy interviews (Gibb et al., 2017).

2. 2018 NRFU Operational Design

The 2018 NRFU operation will include both elements from prior tests that effectively reduced costs and improved data quality, and new enhancements designed to test functionality that will be necessary for the 2020 Census.

Key aspects of prior tests that will continue to be part of NRFU during the 2018 E2E CT include:

• Use of administrative records and third-party data to identify and remove vacant units and non-housing units from the NRFU workload.

• A 1-in-5 subset of cases that undergo AR modeling will be sent to the field to assess the accuracy of the AR determinations.

• Use of administrative records and third-party data to identify occupied housing units, and set one field attempt for such cases.

• A fixed contact strategy for NRFU cases with no status from administrative records that allows a maximum of six attempt days, where cases become eligible for enumeration by proxy on the third attempt day after an unsuccessful household attempt (U.S. Census Bureau, 2017d).^{1 2}

• Capability to extend contact attempt days in hard-to-enumerate areas based on a low resolution rate compared to other areas.

• Automated proxy interview prompts by the enumeration application once a case becomes eligible for enumeration by a proxy respondent.

• Manager visits at multiunits to identify vacant units and non-housing units with a minimum number of contact attempts and to keep respondent burden low.

• Collection of the previous address of inmovers who did not reside at the housing unit on Census Day.

• Capability to stop field work, pause field work, or reassign field work for special situations (e.g., dangerous situations, natural disasters).

• Quality control (QC) program that includes edits throughout the enumeration application to minimize errors, post-training assessments and observations for low-scoring enumerators, control system alerts that detect egregious and anomalous enumerator behavior, and Reinterview/QC program to detect falsification of NRFU interviews, manager visit interviews, and Field Verification cases.

New aspects of NRFU for the 2018 E2E CT include:

• Expansion of addresses that will be eligible for nonresponse followup to include addresses in the Update/Leave TEA.³ In prior tests, only non-responding addresses in the Self-Response TEA were eligible for NRFU.

• Refined stopping rules for cases with an AR modeled status (occupied, vacant, or not a housing unit) based on the presence or absence of an Undeliverable as Addressed (UAA) code from the United States Postal Service (USPS) after postcard mailings. Anytime the AR modeled status and the presence or absence of a UAA code conflict, the case will be sent back for additional field work (i.e., AR ‘vacant’ status with no UAA after the postcard mailing, suggesting the unit may be occupied) (U.S. Census Bureau, 2017d).

• A Field Verification (FV) case workload that attempts to verify the location of housing units that respond to the census without a pre-assigned ID and whose addresses are not in the Census Bureau’s address frame. This sub-operation will not involve an interview with a respondent.

• A Re-collect case workload that determines whether cases flagged as potentially fraudulent are actually fraudulent and provides feedback on the performance of models used in the Fraud Detection System. Re-collect cases will not be eligible for NRFU RI.

• A workload of self-reported vacant cases that require verification in the field.

• A workload of self-responses with only a population count that require fieldwork to obtain more detail about the members of the household.

• A reverse check-in workload for paper questionnaires that do not pass data sufficiency checks and require fieldwork to resolve them.

• The addition of new addresses to the field workload in the midst of the NRFU operation from the U.S. Postal Service’s Delivery Sequence File (DSF) and from the UL operation.

• Refined contact strategies for restricted access and refusals, the two most common types of non-interview situations from the 2016 CT. For restricted access in multiunits, enumerators in the 2018 E2E CT will have an information sheet to show managers that describes the purpose of the manager visit. Refusal cases will be reassigned to a different enumerator after the same enumerator receives a respondent refusal on two different contact attempt days.

• Units within multiunits will not be eligible for enumeration until two manager visits have been attempted. After a manager visit is completed, individual units will be eligible for enumeration based on when respondents are likely to be home (weekday evenings and weekends).

• NRFU cases that have been attempted when the respondent was not home will be available for the enumerator to work, if the respondent returns home while the enumerator is still in the area.

• Two new field alerts that notify the supervisor when an enumerator: 1) does not make attempts to interview a proxy respondent after a case becomes proxy-eligible, and 2) opens uncompleted cases from their caselist and does not attempt an interview.

• Additional training for enumerators on strategies for completing proxy interviews, and the importance of such interviews when a member of the household is unavailable.

• Operational close-out procedures will include the relaxation of NRFU sufficiency rules and rules on the “cool down” period after leaving a Notice of Visit (NOV), and extension of the number of attempt days in certain areas to lower the rate of unresolved cases.

New aspects of NRFU RI for the 2018 E2E CT include:

• An improved analytic sampling method to identify NRFU RI cases. This method uses administrative records, data on population count, phone number, housing unit status, interview duration, and coordinates collected through the Global Positioning System (GPS) to select completed NRFU cases for RI. Cases selected for RI will be matched and compared to their NRFU production counterparts based on housing unit status, population count, and roster names. The resulting match rates will be compared to the rates from previous analytic sampling.

• NRFU RI will include manager visits. Field Verification cases will have a Quality Control (QC) component, as well, but do not involve an interview. Re-collect cases will not have an RI component, but will be proceduralized like RI cases (i.e., they will have three maximum attempts and never be proxy-eligible).

• Running the Sampling, Matching, Reviewing, and Coding System (SMaRCS) for multiple field operations simultaneously.

• All cases sampled for NRFU RI and MV RI with a valid phone number will receive three initial Reinterview attempts via telephone, conducted by the Census Questionaire Assistance (CQA) program. Cases that are not resolved via telephone after three attempts will receive up to three field attempts.

3. Assumptions

This NRFU operational assessment study plan assumes:

• The 2018 E2E CT will occur in the Providence County, RI test site for peak operations.

• Households that self-respond to the census (with or without an ID) and match an address on the Master Address File (MAF) will be removed from the NRFU workload.

• Administrative records and third party data will reduce the NRFU workload.

• The initial NRFU workload in the Self-Response TEA will consist of approximately 411,000 housing units.

• The initial NRFU workload in the Update/Leave TEA will consist of approximately [ ] housing units.⁴

• The NRFU Reinterview workload in both TEAs will consist of approximately [ ] housing units.

• The supplemental NRFU workload will consist of approximately [ ] housing units.

• NRFU field work will last for about ten weeks.

• NRFU RI field work will last for about eleven weeks, extending one week past the end of NRFU.

• Field Verification cases will be part of the NRFU workload.

• Re-collect cases will be part of the NRFU workload.

• Paradata and payroll data will provide the necessary information to calculate cost per attempt and measure data quality.

• Detailed budget data will be available to measure variance between expected and actual costs.

• The Enterprise Censuses and Surveys Enabling (ECaSE) platform Enumeration (ENUM) application and operational control systems will handle case assignment and management.

• MOJO will optimize case assignments based on the enumerators who are available for work, their location, the location of open/unresolved NRFU cases, and routing algorithms that take into account the best time to contact respondents at each address.

• Integration between all systems will work properly.

4. Methodology

The initial NRFU universe for the 2018 E2E CT includes all addresses in the Self-Response and the Update/Leave TEAs that did not respond to the census by the date the universe was created (a few days prior to the start of the operation).

After the initial NRFU universe is created, all NRFU cases undergo administrative records modeling, which assigns to each case one of four modeled statuses:

1. Occupied,

2. Vacant,

3. Not a housing unit, or

4. No determination.

These modeled statuses define the contact strategies and stopping rules for all cases. Addresses with an AR status of ‘vacant’ or ‘not a housing unit’ are sent a final mailing to encourage self-response in case the AR status was incorrect. If the postal service returns a UAA code for an AR ‘vacant’ or ‘not a housing unit’ address, that address is removed from the NRFU workload. If the Census Bureau does not receive a UAA code for an AR ‘vacant’ or ‘not a housing unit’ after a certain period of time, that address is added back to the NRFU field workload.

Addresses with an AR status of ‘occupied’ receive one contact attempt. If the enumerator is not successful in resolving the case on that day, it will be mailed a postcard prompting self-response. If the postal service returns a UAA code for that mailing, the case will be sent back for additional field work.

Finally, addresses with no AR status (‘no determination’) will receive a maximum of six attempt days. They will become eligible for enumeration by a proxy respondent on the third attempt day after an unsuccessful attempt on that day with a member of the household. There will be a maximum daily limit of three proxy attempts per case.

For any hard-to-enumerate areas within the test site with a low field resolution rate compared to other areas, the operational control system will allow the number of contact attempt days to be extended beyond six days.

In early June, AR modeling will be performed a second time on any unresolved NRFU cases, when additional data will be incorporated into the model. Any cases with at least one contact attempt and an AR status of ‘occupied’ from the second round of modeling will be pulled from the field workload and enumerated via administrative records.

NRFU enumerators will attempt manager visits at multiunits to identify vacant units and non housing units with the purpose of reducing the operational workload and burden on managers.

During the 2018 E2E CT, there will be a target field staff ratio of one Census Field Supervisor (CFS) per 20 enumerators.

A sample of NRFU and MV cases will be subjected to a quality control reinterview, in order to detect and deter enumerator falsification of response data. These cases are known as NRFU RI and MV RI, respectively. For these cases, the RI enumerator will attempt to contact the same respondent from the original interview. This may be a member of the household, neighbor, or some other proxy respondent. If the original respondent confirms that they completed the original interview, the RI enumerator will recollect roster names only. If the respondent was not contacted or cannot remember whether they completed the original interview, the RI enumerator will conduct a full interview with the respondent.

For all cases selected for RI, the original interview and RI data will be compared through computer and clerical matching, as well as field investigation to determine whether the original enumerator conducted the interview correctly, followed proper procedures, and collected valid and complete data. There are four components to the RI program: analytic, random, supplemental, and rework.⁵ The Original interview and RI data will be compared using the Sampling, Matching, Reviewing, and Coding System (SMaRCS), which provides computer matching, clerical matching, final coding, and supplemental RI selection.

Field Verification cases will also have a QC component that will consist of an independent attempt to verify the location of the address. Re-collect cases will not have an RI component. NRFU RI will utilize the same mobile application as NRFU to collect reinterview data, and the same field staff will perform both operations.

In addition to some legacy systems from prior tests, NRFU and NRFU RI will rely upon several newly-developed Census Enterprise Data Collection and Processing (CEDCaP) systems that will be used for the first time during the 2018 E2E CT. These new systems replace the Multiple Operational Control System (MOCS) and Census Operations Mobile Platform for Adaptive Services and Solutions (COMPASS) systems from the 2016 Census Test, as well as some of the original functionality of MOJO. The following systems represent those that are critical for data collection during NRFU for the 2018 E2E CT.

New CEDCaP Systems	Function
Concurrent Analysis and Estimation System (CAES)	Enterprise modeling platform that executes statistical modeling for administrative records.
Enterprise Censuses and Surveys Enabling platform (ECaSE) - Enumeration (ENUM)	Supports field data collection for enumeration work.
Enterprise Censuses and Surveys Enabling Platform (ECaSE) Operational Control Systems (OCS)	Creates the initial universe, receives status updates and response data, and maintains operational workloads as data collection proceeds. Contains two components: Survey OCS and Field OCS.
Non-CEDCaP Legacy Systems	Function
Census Questionnaire Assistance (CQA)	Conducts agent-initiated data collection for outbound RI calls.
Decennial Applicant Personnel and Payroll System (DAPPS)	Process and tracks selection, hiring, and payroll for field staff.
MOJO Optimizer	Makes optimized enumerator work assignments through the ECaSE ENUM application based on the enumerators who are available for work, their location, the location of open/unresolved NRFU cases, and best time to contact information.
Production Environment for Administrative Records Staging, Integration, and Storage (PEARSIS)	Provides administrative records data to CAES to perform modeling.
Sampling, Matching, Reviewing, and Coding System (SMaRCS)	Supports quality control by selecting a sample of RI cases and testing whether enumerators are using validated procedures and collecting accurate data.
Unified Tracking System (UTS)	Data warehouse that combines data from a variety of census systems and creates reports for management of operations and assessment of data after the test is complete.

5. Scope of Assessment Content and Questions-To-Be-Answered

This analysis will be broken out by the two TEAs, AR status, and each type of NRFU case (e.g., production NRFU, manager visit, Field Verification, Re-collect, self-reported vacant, Reinterview), as applicable.

1. Collect summary NRFU operational data to inform 2020 Census cost parameters.

Overall Metrics

1. What was the self-response rate prior to the start of the NRFU operation? How did this compare to the expected self-response rate of 60.5 percent?

2. What was the self-response rate between the start and end of the NRFU operation (May 9 to July 24), by whether or not a Notice of Visit (NOV) was left?

3. How did the actual start and end dates for the NRFU operation compare to the planned start and end dates?

4. What was the size of the initial NRFU universe created on May 7? How many NRFU cases were sent to the field on May 9, the first day of the operation?

5. How many cases were added to the initial NRFU universe from the following: DSF refresh, UL adds, field/office adds, reverse check-ins, AR reverse check-ins?

6. What was the percentage of AR cases eligible for removal from the NRFU workload, by AR unit status and by round of AR modeling (first in May and second in June)?

7. At the end of the NRFU operation, how many cases received contact attempts, by case type and final resolution?

8. What were the number of attempt days, attempt days where contact was made, and attempt days where contact was made and the case was resolved: overall, by hour, and by day of the week?

Note: For hour and day of the week detail, we will create stacked bar charts.

Table 1. NRFU Attempt Days, Contacts, and Resolved Cases by Attempt Day
Attempt Day	Number of Cases Attempted	Contact Made	Percent Contacted	Resolved	Percent Resolved Of Contacted
1
2
3
4
5
6
7+
Total

Source(s): ECaSE Enum

9. What was the average number of attempts made per hour, by week of the operation?

10. Within a given attempt day, what was the average number of attempts made per case?

Table 2. NRFU Average and Median Number of Attempts by Attempt Day
Attempt Day	Number of Cases Attempted	Total Number of Attempts	Average Number of Attempts	Standard Deviation	Median Number of Attempts
1
2
3
4
5
6
7+
Total

Source(s): ECaSE Enum

11. What was the number of resolved cases by attempt day (including cases not resolved in NRFU): overall, by hour, and by day of the week?

Note: For hour and day of the week detail, we will create stacked bar charts.

Table 3. Number and Percent of NRFU Cases with a Resolved Status by Attempt Day
NRFU Resolved	Attempt Day	Number	Percent Overall
	Total NRFU Resolved
	1
	2
	3
	4
	5
	6
	7+
No NRFU Resolved Status	Total No NRFU Resolved Status
	Self-Response Received
	NRFU Exit Status
	NRFU Unresolved
Total

Source(s): ECaSE Enum and OCS Status Data

12. What was the number of resolved cases by resolved status: overall and by day of the week?

Note: For day of the week detail, we will create stacked bar charts.

Table 4. Number and Percent of Resolved NRFU Cases by Resolved Status
Resolved Status	Number	Percent
Occupied
Occupied: Household Respondent
Occupied: Proxy Respondent
Vacant
Proxy Vacant
Observed Vacant
Not a Housing Unit
Proxy Not a Housing Unit
Observed Not a Housing Unit
Total

Source(s): ECaSE Enum

13. What was the average interview length of completed interviews: overall, by resolved status, and by household size for occupied units?

Note: For household size detail, we will create separate tables.

Table 5. NRFU Average Interview Length by Resolved Status
Resolved Status	Average	Standard Deviation
All completed Inerviews
Occupied
Occupied: Household Respondent
Occupied: Proxy Respondent
Proxy Vacant
Proxy Not a Housing Unit

Source(s): ECaSE Enum

Cost Metrics

14. What was the total cost of the NRFU operation, and how did it compare to the planned budget?

15. How did the estimated cost per enumerator and per attempt compare to the actual cost?

16. How many training hours did enumerators charge?

17. How many miles did enumerators charge: on average, overall, and by week?

18. How many hours did enumerators charge: on average, overall, and by week?

2. What was the unresolved rate at the end of the NRFU operation, overall and by case type? How did this compare to the unresolved rate from prior tests? From the 2010 Census? What was the unresolved rate in what would have been Early NRFU areas?

3. Capture metrics related to enumeration of multiunits.

1. How many manager visit cases were there? How many individual units were associated with the manager visit cases?

2. How often were different enumerators assigned to manager visits within the same BCU on the same attempt day?

3. How many attempt days were there for manager visits?

4. How many attempt days were there at individual units within multiunits?

5. How many vacant units and non-housing units were identified by manager visits?

Table 6. Number and Percent of Manager Visit Statuses for Individual Units
Manager Status for Individual Unit	Number	Percent
Occupied
Vacant or Not a Housing Unit
Vacant
Not a Housing Unit
No Manager Status
Total

Source(s): ECaSE Enum

6. What impact did pre-identifying vacant units and non-housing units have on the number of attempt days at multiunits?

7. What impact did pre-identifying vacant units and non-housing units have on the cost of enumerating multiunits?

8. How often was a unit which was identified by a manager visit as ‘occupied’ later resolved as ‘vacant’ or ‘not a housing unit’ by an enumerator?

9. How often was a unit which was categorized as ‘occupied’ by AR modeling later identified by a manager visit as ‘vacant’ or ‘not a housing unit’?

10. What was the final housing unit status for individual units within multiunits, by the status from the manager visit?

Table 7. Final NRFU Status for Individual Units by Manager Visit Status
	Manager Visit Status
	Occupied		Vacant		Not a Housing Unit		No Manager Status
Final Housing Unit Status	Number	Percent	Number	Percent	Number	Percent	Number	Percent
Occupied
Occupied: Household Respondent
Occupied: Proxy Respondent
Vacant
Proxy Vacant
Observed Vacant
Not a Housing Unit
Proxy Not a Housing Unit
Observed Not a Housing Unit
No NRFU Resolved Status
Self Response Received
Exit Status Obtained
NRFU Unresolved
Total Overall

Source(s): ECaSE Enum

11. How often were there attempt days for units identified as ‘vacant’ or ‘not a housing unit’ during the manager visit?

12. What was the average number of attempt days for units identified by a manager visit as ‘occupied’ or with ‘no manager status’?

13. What was the average interview length for completed manager visits: overall and by number of NRFU units within the multiunit?

14. What was the average interview length for completed interviews with units identified by a manager visit as ‘occupied’ or with ‘no manager status’: overall and by household size for occupied units?

4. Verify NRFU contact strategies.

Overall Metrics

1. How often did the number of attempt days extend beyond the number set for that case?

2. How often was a case worked by more than one enumerator on the same attempt day? Did cases that were worked by more than one enumerator on the same attempt day have a higher or lower resolution rate compared to other cases?

3. What was the average number of proxy attempts: overall and by the final case status?

4. How often did a case which received a status of ‘duplicate’ or ‘multiunit missing unit designation’ receive additional contact attempt days?

5. How often did inmovers with no knowledge of the previous resident(s) receive further household attempts on subsequent days?

6. How many enumerators worked a case, by the final resolution? How many attempt days were there on cases worked by only one enumerator?

7. What was the outcome of the NRFU verification attempt for self-reported vacants, by the initial self-reported vacancy reason and the final NRFU housing unit status?

Table 8. Number and Percent of Self-Reported Vacant Cases by Final Housing Unit Status
Final Housing Unit Status	Number	Percent
Occupied
Occupied: Household Respondent
Occupied: Proxy Respondent
Vacant
Proxy Vacant
Observed Vacant
Not a Housing Unit
Proxy Not a Housing Unit
Observed Not a Housing Unit
No NRFU Resolved Status
Self-Response Received
Administrative Record Confirmed
Exit Status Obtained
NRFU Unresolved
Total

Source(s): ECaSE Enum

Table 9. Final Housing Unit Status for Self-Reported Vacants by Vacancy Reason
	Final Housing Unit Status
	Occupied		Vacant		Not a Housing Unit		NRFU Unresolved
Vacancy Reason	Number	Percent	Number	Percent	Number	Percent	Number	Percent
For rent
Rented, not occupied
For sale only
Sold, not occupied
For seasonal, recreational, or occasional use
For migrant workers
Other
Total

Source(s): ECaSE Enum

Observed Status Metrics

8. How often did the same enumerator make both the first and second validation for observed ‘vacant’ units and observed ‘not a housing unit’ cases? Regardless of who performed the second validation, how often did it yield the same result as the first?

Table 10. First and Second Enumerator-Observed Outcomes

	Same Result		Different Result		Total
	Number	Percent	Number	Percent	Number	Column Percent
Same Enumerator
Different Enumerator
Total

Source(s): ECaSE Enum

9. How often did a case observed by an enumerator as ‘vacant’ or ‘not a housing unit’ on two separate attempt days, receive additional attempt days?

10. How often did a case observed by an enumerator as ‘vacant’ or ‘not a housing unit,’ with a subsequent proxy interview, receive additional attempt days?

11. How often were proxy attempts made on the same attempt day that an enumerator observed a case as ‘vacant’ or ‘not a housing unit’?

12. What was the final status for any case with at least one attempt resulting in an observed ‘vacant’ or observed ‘not a housing unit’ status?

Table 11. Number and Percent of Final NRFU Status by Initial Observed Status
	Initial Observed Status
	Vacant		Not A Housing Unit
Final Status	Number	Percent	Number	Percent
Occupied
Occupied: Household Respondent
Occupied: Proxy Respondent
Vacant
Proxy Vacant
Observed Vacant
Not a Housing Unit
Proxy Not a Housing Unit
Observed Not a Housing Unit
No NRFU Resolved Status
Self Response Received
Exit Status Obtained
NRFU Unresolved
Total Overall

Source(s): ECaSE Enum

5. How many cases had any of the following non-interview situations on any attempt day, by their final status: dangerous situations, refusals, restricted access, duplicates, and not a housing unit situations?

6. Analyze the enumerator alerts sent to supervisors by the operational control system.

1. What were the number and type of alerts handled by Census Field Supervisors (CFSs)?

2. Did the number and type of alerts change as the operation progressed?

3. By type of alert, how often were they responded to?

4. How often did an enumerator have the same alert sent to their CFS multiple times?

5. At the end of the operation, how many unresolved alerts were there by type of alert?

7. Were any NRFU cases not assigned for fieldwork during the test for any reason? If yes, explain why. [Include background on “ungeocoded” cases in 2016 CT.]

8. What was the number and duration of field training sessions? How many replacement training sessions were there?

9. What was the number of field staff by position: authorized, invited to training, available for work, assigned work, worked cases, and replacements?

10. How did the actual field staff ratio compare to the target ratio? How did the field staff ratio change over the course of the operation? How well did the ratio work, according to enumerators and CFSs?

11. How were the following lessons learned from the 2016 CT addressed for the 2018 E2E CT?

1. Were changes to the contact strategies for multiunits operationalized as intended?

2. Did enumerators attempt to interview proxy respondents on the appropriate attempt day, for the appropriate types of cases?

3. Were CFSs better able to manage their workloads related to their assigned enumerators, manage review cases, and the volume of field alerts?

4. Were enumerators able to resolve more non-interview situations?

5. Were we able to reduce the unresolved rate?

12. What were the lessons learned from the 2018 E2E CT?

13. Did system integration between ECaSE Enum, Field OCS, and Survey OCS perform as expected?

14. Did the MOJO Optimizer perform as expected?

15. Did we receive all of the response data and paradata we requested? Was it available when we expected it to be?
    
    

16. Summarize the NRFU Quality Control (QC) operation.

    1. What percentage of addresses were selected for the QC operation?

Table 12. Number and Percent of Cases Selected for QC Operation

	Number Worked	Number Selected	Percent
Addresses

Source(s): SMaRCS

2. Of the addresses selected for QC, how many were resolved by CQA? How many were resolved in the field?

Table 13. QC Cases by Type of Resolution

	Resolved by CQA	Resolved in the Field	Unresolved
Addresses

Source(s): SMaRCS

3. How many enumerators had at least one hard fail during the QC operation?

Table 14. Number and Percent of Enumerators with at Least One Hard Fail

Number of Enumerators	Enumerators with Hard Fails	Percent

Source(s): SMaRCS

4. How many of the selected addresses passed/failed the QC operation?

Table 15. Number and Percent of Addresses by QC Outcome

	Number	Percent
Addresses that Passed
Addresses that Failed
Addresses that Hard Failed

Source(s): SMaRCS

5. How well did the QC sampling methods perform?

6. Were we able to successfully process QC cases and perform automated and clerical matching of QC data?

17. Present standard demographic tables

Table 16. Standard Assessment Demographic Table for Sex in NRFU Interviews

Sex	Number	Percent
Total Population		100.0
Male
Female
Missing

Source(s): DRF

Table 17. Standard Assessment Demographic Table for Age and Sex in NRFU Interviews

Age and Sex	Number	Percent
Total Population		100.0
Under 5 years
5 to 9 years
10 to 14 years
15 to 19 years
20 to 24 years
25 to 29 years
30 to 34 years
35 to 39 years
40 to 44 years
45 to 49 years
50 to 54 years
55 to 59 years
60 to 64 years
65+ years
Missing
Male
Under 5 years
5 to 9 years
10 to 14 years
15 to 19 years
20 to 24 years
25 to 29 years
30 to 34 years
35 to 39 years
40 to 44 years
45 to 49 years
50 to 54 years
55 to 59 years
60 to 64 years
65+ years
Missing
Female
Under 5 years
5 to 9 years
10 to 14 years
15 to 19 years
20 to 24 years
25 to 29 years
30 to 34 years
35 to 39 years
40 to 44 years
45 to 49 years
50 to 54 years
55 to 59 years
60 to 64 years
65+ years
Missing

Source(s): DRF

Table 18. Standard Assessment Demographic Table for Race and Ethnicity in NRFU Interviews

Race and Ethnicity	Number	Percent
Total Population		100.0
White, Alone
Hispanic, Latino, or Spanish, Alone
Black or African American, Alone
Asian, Alone
American Indian or Alaska Native, Alone
Middle Eastern or North African, Alone
Native Hawaiian or Other Pacific Islander, Alone
Some Other Race, Ethnicity, or Origin, Alone
Two or More
Write-In Only*
Missing

*Write-in accepted if write-in box was filled and no other race categories were selected

Source(s): DRF

Table 19. Standard Assessment Demographic Table for Relationship in NRFU Interviews

Relationship	Number	Percent
Total Population		100.0
Householder
Opposite-sex Husband/Wife/Spouse
Opposite-sex Unmarried Partner
Same-sex Husband/Wife/Spouse
Same-sex Unmarried Partner
Biological Son or Daughter
Adopted Son or Daughter
Stepson or Stepdaughter
Brother or Sister
Father or Mother
Grandchild
Parent-in-law
Son-in-law or Daughter-in-law
Other Relative
Roommate or Housemate
Foster Child
Other Nonrelative
On Extended Roster*
Related to Householder
Not Related to Householder
Both
None
Missing

Source(s): DRF

Table 20. Standard Assessment Demographic Table for Tenure in NRFU Interviews

Tenure	Number	Percent
Total Housing Units		100.0
Owned with a mortgage or a loan
Owned without a mortgage or a loan
Rented
Occupied without payment of rent
Missing

Source(s): DRF

6. Knowledge Management Resolutions

1. Knowledge Management Recommendation Number: 147

Document Title:

2010 Census Field Verification Operational Assessment Report

Project Level Disposition Code:

Pursued for 2018 E2E CT

Team Action Plan:

The 2018 E2E CT will introduce a Field Verification (FV) workload within an automated environment for the first time this decade. Field Verification cases will be worked on a flow basis during the NRFU operation with other types of cases. FV cases will be eligible for quality control procedures.

2. Knowledge Management Recommendation Number: 1141

Document Title:

2020 Census Research and Testing: 2014 Census Test Nonresponse Followup Panel Comparisons and Instrument Analysis

Project Level Disposition Code:

Pursued for 2018 E2E CT

Team Action Plan:

During the 2018 E2E CT, NRFU will test refined use of administrative records (AR) to reduce the workload and make contact attempts more efficient. In 2018, the presence or absence of Undeliverable as Addressed (UAA) codes from the U.S. Postal service received after mailings will be used in conjuction with the AR status to define further field attempts. Anytime where the two sources are not in agreement about the unit status of a particular address, field work will resolve the discrepancy.

7. Risks & Limitations

The following sections present various risks and limitations associated with this operational assessment.

1. General Program Risks for All 2018 E2E CT Assessments

1. As of January 2017, funding for the 2018 Census End-to-End Test has not been finalized or guaranteed.

IF the amount of funding is different than the amount requested, THEN NRFU activities will be conducted as a proportional sample relative to the amount of funding available for the operation (compared to the request.)

2. Major disasters (e.g., earthquake, flood, tornado, epidemic, and terrorist attack) can affect the population and prevent people from self-responding to the 2018 E2E CT or being contacted by field staff. Major disasters can disrupt operations at key facilities (e.g., Headquarters, National Processing Center, Regional Census Centers, and Area Census Offices) and supporting infrastructure (e.g., Post Offices and telecommunications).

IF a major disaster occurs during the implementation of the test, THEN operations may not be able to be executed as planned, leading to increased costs, schedule delays, and lower quality data.

2. System development and integration for the 2018 E2E CT is running behind schedule and may affect system readiness to support the field operations and the associated test objectives. 

IF system development continues to run behind schedule, THEN the field operations for the 2018 E2E CT will not be able to perform as planned.

2. Risks Specific to This 2018 E2E CT Assessment

1. Many aspects related to the NRFU operational design and the infrastructure necessary to support it are based on workload assumptions. A key input to those workload assumptions is the self-response rate.

IF the 2018 E2E CT self-response rate falls below expectations, THEN the initial NRFU workload will be higher than expected and the infrastructure to support an increased field data collection volume may be insufficient.

2. The NRFU workload will be impacted by other operations that are striving to develop and improve the coverage and quality of the address frame used for the 2018 E2E CT.

IF there is an increase in the NRFU operational workload due to the results of the upstream address frame operations, THEN the expected cost savings from the NRFU operation may not be realized.

3. The completeness and quality of this operational assessment are dependent upon the data available at the conclusion of the 2018 E2E CT.

IF data are not made available on time, or in the expected structure, THEN the analysis may be delayed or scaled back.

IF data are not made available, THEN the analysis may not be completed as anticipated.

4. Paradata are important because they contain information about the times of day interviews were conducted, how long the interviews took, how many times there were contacts with each interviewee or attempts to contact the interviewee, the reluctance of the interviewee, and the mode of communication (such as phone, Web, email, or in person) etc.

IF paradata from NRFU cases is not captured, stored, or transferred properly, THEN downstream operations may not be as efficient and NRFU cannot do its own evaluation.

3. Limitations

1. This assessment of the NRFU operation during the 2018 E2E CT will be based on housing units selected from particular local areas, and cannot be generalized to the entire nation. The results will not predict national trends or rate estimates expected in the 2020 Census.

2. During the 2018 E2E CT, enumerators will work a combination of NRFU (in two different TEAs), manager visit, Field Verification, Re-collect, and NRFU RI cases (including MV RI and FV QC). For parts of the analysis, it will not be possible to distinguish between these different types of cases (i.e., enumerators report miles charged overall rather than by case type). However, it is expected that the bulk of most enumerators’ assignments will be standard NRFU cases.

8. Measures of Success

The following criteria will be used to define successful completion of the NRFU and NRFU RI operations:

1. Improved procedures are developed and implemented for gaining entry to restricted access cases.

2. Proxy procedures increase completion rates and reduce multiple visits to the same proxy respondent.

3. The rate of unresolved cases is lower than that of the 2016 CT.

4. Field staff effectively collect data from inmover cases, in addition to the enumeration of the original case address.

5. Field staff verify the address of Non-ID self-responses for addresses that are not in the MAF during Field Verification.

6. NRFU RI, manager visit RI, and FV QC identify cases of enumerator falsification.

7. The test is completed with no corruption or confusion of data from the different types of NRFU cases within SMaRCs.

9. Data Requirements

Data Source	Description of Data Needed	Anticipated Availability
Decennial Response File (DRF)	Respondent-provided data to be used for post-processing purposes	TBD
Decennial Applicant Personnel and Payroll System (DAPPS)	Payroll data related to operational costs	TBD
Universal Tracking System (UTS) reports	High-level reports used by managers to evaluate the test	TBD
Sampling, Matching, Reviewing, and Coding System (SMaRCS)	Data for cases selected for NRFU RI and their matched results	TBD
Enumeration instrument auxiliary data and paradata	All output data from the enumeration instrument and other data not needed for response-processing purposes	TBD
Enumerator and CFS debriefing results	Feedback from enumerators and CFSs about various aspects of the operation	TBD
Observation reports	Reports created by shadowing enumerators in the field	TBD
ECaSE alert data	Data describing characteristics of alerts sent to supervisors to assess enumerator productivity	TBD
ECaSE management review data	Data describing all management review actions	TBD
ECaSE assignment data	Data describing all field assignments sent to enumerators	TBD
Field staff data	Data that links enumerators to their supervisors	TBD
Multiunit grouping data	Data describing cases identified for grouping into manager visit cases	TBD
Data on cases that could not be geocoded	Data for cases that could not be geocoded and were not sent out to the field	TBD

10. Division Responsibilities

Division	Division Members	Description of Responsibilities
Decennial Census Management Division (DCMD)	Sarah Gibb Troy Wands Farouk Nabourema	Coordinating/assisting with study plan and analysis report writing; Coordinating study plan and analysis report review, and making updates as needed; Assisting with data acquisition
Decennial Statistical Studies Division (DSSD)	Ryan King Robert Fitzsimmons Mary Frances Zelenak Rafael Morales RJ Marquette	Acquiring data; Analyzing response data; Analyzing instrument data; Analyzing quality control data; Writing study plan and analysis report
Field Division (FLD)	Amy Fischer Rhonda Cleveland	Collecting data in the field; Providing content for analysis report (e.g., staffing, cost, etc.); Reviewing study plan and analysis report
Geography Division (GEO)
Census Enterprise Data Collection and Processing (CEDCaP)
Office of Innovation and Implementation (OII)

11. Study Plan & Assessment Report Schedule

Below are the standard schedule activities for the development of the study plan and operational assessment report. Activities highlighted and in bold block-face are the key milestone activities. Durations, start dates, and finish dates in bold block-face are “actuals”; those not in bold block-face are the baseline.

Activity ID	Activity Name	Duration	Start	Finish
NRFU Assessment Study Plan
First Draft
18NRD-21040	Prepare First Draft of NRFU Assessment Study Plan	36	3/16/17	5/4/17
18NRD-21050	Distribute First Draft of NRFU Assessment Study Plan to the Assessment Sponsoring DCMD ADC and Other Reviewers	1	5/5/17	5/5/17
18NRD-21060	Incorporate DCMD ADC and Other Comments to NRFU Assessment Study Plan	4	6/20/17	6/23/17
Initial Draft
18NRD-21160	Prepare Initial Draft NRFU Assessment Study Plan	18	5/31/17	6/23/17
18NRD-21070	Distribute Initial Draft NRFU Assessment Study Plan to Evaluations & Experiments Coordination Branch (EXCB)	1	6/23/17	6/23/17
18NRD-21080	EXCB Distributes Initial Draft NRFU Assessment Study Plan to the DROM Working Group for Electronic Review	1	6/26/17	6/26/17
18NRD-21090	Receive Comments from the DROM Working Group on the Initial Draft NRFU Assessment Study Plan	5	6/26/17	6/30/17
	Schedule the NRFU Study Plan for the IPT Lead to Meet with the DROM Working Group	1	5/16/17	5/16/17
18NRD-21100	Discuss DROM Comments on Initial Draft NRFU Assessment Study Plan	1	7/6/17	7/6/17
	Discuss DROM Comments on Initial Draft NRFU Assessment Study Plan (Cont.)	1	7/28/17	7/28/17
Final Draft
18NRD-21110	Prepare Final Draft of NRFU Assessment Study Plan	11	7/31/17	8/14/17
18NRD-21120	Distribute Final Draft of NRFU Assessment Study Plan to the DPMO and the EXCB	1	8/15/17	8/15/17
18NRD-21130	Discuss Final Draft of NRFU Assessment Study Plan with the 2020 PMGB	6	8/16/17	8/23/17
	Incorporate 2020 PMGB Comments for NRFU Assessment Study Plan	10	8/24/17	9/7/17
18NRD-21110	Prepare FINAL NRFU Assessment Study Plan	5	8/24/17	9/7/17
18NRD-21120	Send Final NRFU Assessment Study Plan to the EXCB	1	9/8/17	9/8/17
18NRD-21130	EXCB Staff Distributes the NRFU Assessment Study Plan and 2020 Memorandum to the DCCO	3	9/11/17	9/13/17
	DCCO Staff Process the Draft 2020 Memorandum and the NRFU Assessment Study Plan to Obtain Clearances (DCMD Chief, Assistant Director, and Associate Director)	30	9/14/17	10/25/17
	DCCO Staff Formally Release the NRFU Assessment Study Plan in the 2020 Memorandum Series	1	10/26/17	10/26/17
NRFU Assessment Report
First Draft
	Receive, Verify, and Validate NRFU Assessment Data	20	8/6/18	8/31/18
	Examine Results and Conduct Analysis	20	9/3/18	9/28/18
18NRD-21180	Prepare First Draft of NRFU Assessment Report	15	10/1/18	10/19/18
18NRD-21190	Distribute First Draft of NRFU Assessment Report to the Assessment Sponsoring DCMD ADC and Other Reviewers	1	10/22/18	10/22/18
18NRD-21200	Incorporate DCMD ADC and Other Comments to NRFU Assessment Report	7	10/23/18	10/31/18
Initial Draft
18NRD-21210	Prepare Initial Draft NRFU Assessment Report	8	11/1/18	11/12/18
18NRD-21230	Distribute Initial Draft NRFU Assessment Report to Evaluations & Experiments Coordination Br. (EXCB)	1	11/13/18	11/13/18
18NRD-21240	EXCB Distributes Initial Draft NRFU Assessment Report to the DROM Working Group for Electronic Review	1	11/14/18	11/14/18
18NRD-21250	Receive Comments from the DROM Working Group on the Initial Draft NRFU Assessment Report	10	11/15/18	11/29/18
	Schedule the NRFU Report for the IPT Lead to Meet with the DROM Working Group	10	11/30/18	12/13/18
18NRD-21260	Discuss DROM Comments on Initial Draft NRFU Assessment Report	1	12/14/18	12/14/18
Final Draft
18NRD-21270	Prepare Final Draft of NRFU Assessment Report	25	12/17/18	1/23/19
18NRD-21280	Distribute Final Draft of NRFU Assessment Report to the DPMO and the EXCB	1	1/24/19	1/24/19
18NRD-21290	Schedule and Discuss Final Draft NRFU Assessment Report with the 2020 PMGB	14	1/25/19	2/13/19
18NRD-21300	Incorporate 2020 PMGB Comments for NRFU Assessment Report	10	2/14/19	2/28/19
Final Report
18NRD-21220	Prepare FINAL NRFU Assessment Report	10	3/1/19	3/14/19
18NRD-21310	Deliver FINAL NRFU Assessment Report to the EXCB	1	3/15/19	3/15/19
	EXCB Staff Distribute the FINAL NRFU Report and 2020 Memorandum to the DCCO	3	3/18/19	3/20/19
	DCCO Staff Process the Draft 2020 Memorandum and the FINAL NRFU Report to Obtain Clearances (DCMD Chief, Assistant Director, and Associate Director)	30	3/21/19	5/1/19
	DCCO Staff Formally Release the FINAL NRFU Report in the 2020 Memorandum Series	1	5/2/19	5/2/19
	EXCB Staff Capture Recommendations of the FINAL NRFU Report in the Census Knowledge Management SharePoint Application	​1	5/3/19	5/3/19

12. Issues That Need to be Resolved

The following are issues related to the NRFU operation that are in the process of being discussed, decided upon, and documented, but still need to be completed prior to the 2020 Census:

• Rules for determining when a NRFU response is sufficient need to be defined and documented (and how these rules may change during the closeout phase).

• Assignment rules for NRFU cases based on their event code, status, and disposition need to be defined for the operational control systems.

• Business rules for optimization of NRFU case assignments need to be provided to MOJO.

• Closeout procedures for the NRFU operation need to be defined.

13. Review/Approval Table

Role	Electronic Signature	Date
Fact Checker or independent verifier
Author’s Division Chief (or designee)
DCMD ADC
DROM DCMD co-executive sponsor (or designee)
DROM DSSD co-executive sponsor (or designee)
Associate Director for R&M (or designee)
Associate Director for Decennial Census Programs (or designee)
2020 PMGB

14. Document Revision and Version Control History

VERSION/EDITOR	DATE	REVISION DESCRIPTION	EAE IPT CHAIR APPROVAL
v0.4/Troy Wands	05/04/2017	First Draft
v0.5/Sarah Gibb	05/18/2017	NRFU IPT comments incorporated
v0.6/Sarah Gibb	06/23/17	Added new operation dates and additional information from EAE’s content guidelines; incorporated comments from Maryann Chapin
V0.7/Sarah Gibb	08/04/17	DROM comments incorporated; removed Early NRFU references; updated test site information; added KMRs; updated event code list and alert list in appendices

15. Glossary of Acronyms

Acronym	Definition
2018 E2E CT	2018 End-to-End Census Test
ADC	Assistant Division Chief
AR	Administrative Records
BCU	Basic Collection Unit
CAES	Concurrent Analysis and Estimation System
CEDCaP	Census Enterprise Data Collection and Processing
CFS	Census Field Supervisor
COMPASS	Census Operations Mobile Platform for Adaptive Services and Solutions
CQA	Census Questionnaire Assistance
DAPPS	Decennial Applicant, Personnel and Payroll Systems
DCCO	Decennial Census Communications Office
DCMD	Decennial Census Management Division
DPMO	Decennial Program Management Office
DRF	Decennial Response File
DROM	Decennial Research Objectives and Methods Working Group
DSF	Delivery Sequence File
DSSD	Decennial Statistical Studies Division
ECaSE	Enterprise Censuses and Surveys Enabling Platform
ECaSE ENUM	Enterprise Censuses and Surveys Enabling Platform Enumeration
ECaSE FOCS	Enterprise Censuses and Surveys Enabling Platform Field Operational Control System
ECaSE SOCS	Enterprise Censuses and Surveys Enabling Platform Survey Operational Control System
EXCB	Evaluations & Experiments Coordination Branch
FLD	Field Division
FV	Field Verification
FD	Fraud Detection
GPS	Global Positioning System
IPT	Integrated Project Team
IRS	Internal Revenue Service
MAF	Master Address File
MOCS	Multiple Operational Control System
MOJO	MOJO Optimizer
MV	Manager visit
NRFU	Nonresponse Followup
NRFU RI	Nonresponse Followup-Reinterview
PEARSIS	Production Environment for Administrative Records Staging, Integration, and Storage
PMGB	Portfolio Management Governance Board
R&M	Research & Methodology Directorate
SMaRCS	Sampling, Matching, Reviewing, and Coding System
TEA	Type of Enumeration Area
UAA	Undeliverable as Addressed
USPS	United States Postal Service
UTS	Unified Tracking System

16. Field Operational Control System Alerts

Alert Name	Definition
No Timesheet	If an employee worked on a particular day, but did not submit timesheet by 11:59pm.
Overtime Claimed	Anytime OT was entered on a timesheet.
Potential Overcharge Miles	Anytime miles claimed exceeds expected miles (within certain tolerance).
Potential Overcharge Hours	Anytime hours claimed exceeds expected hours (within certain tolerance).
Look Ahead Availability	When CFS team does not have enough workers with availability for 3 work days from today and there is work to be assigned.
Payroll Not Approved (CFM)	Payroll submitted at least 2 work days ago and no action has been taken.
Many Days No Work	When an employee has not attempted any assignment in 3 work days.
Long Distance Flag	When work was attempted/completed on an assignment for a particular day and the employee set off the long distance flag check.
Short Interview	Any short interview completed by the employee defined as: Interview length is two minutes or less.
High Partial Rate	When an employee has a high rate of partial interviews (i.e., Sufficient Partial Rate Mean + 1.5 * Standard Deviation).
High Refusal Rate	When an employee has a high refusal rate (i.e., Refusal Rate Mean + 1.5 * Standard Deviation).
Work Not Started	When an employee was scheduled to work and has an assignment that has been pushed to their device for that work day, but they have not attempted a case after 45 minutes plus their estimated drive time to their assignment.
Low Completed Cases Rate	When an employee has a low rate of completed cases, defined as Mean – 1.5 * Standard Deviation, compared to peers in a similar geographic area.
High Completed Cases Rate	When an employee has a high rate of completed cases, defined as Mean + 1.5 * Standard Deviation, compared to peers in a similar geographic area.
Unconfirmed Delete by Observation	When an employee has two or more deletes that are not confirmed either by a proxy or a second enumerator.
Working Before Assigned Hours	Identifies when an employee attempts a case more than 30 minutes before their assigned start time.
Unconfirmed Vacancy by Observation	When an employee has two or more vacant outcomes that were not confirmed either by proxy or a second enumerator.
High POP 1 Rate	When an employee has a high POP 1 rate when compared to peers in a similar geographic area (i.e., POP 1 Rate Mean + 1.5 * Standard Deviation).
Low Attempts/Hour	When an employee has completed a low number of attempts/hour compared to peers in a similar geographic area, over the last 7 calendar days (i.e., Mean Attempts/Hour - 1.5 * Standard Deviation).
High Attempts/Hour	When an employee has completed a high number of attempts/hour compared to peers in a similar geographic area, over the last 7 calendar days (i.e., Mean Attempts/Hour + 1.5 * Standard Deviation).
Proxy Attempts	When two or more cases were proxy eligible but the employee did not enter any proxy attempts into the enumeration application.
Reopening Cases	When an employee opens two or more inactive cases but did not complete them.

17. Enterprise Event Codes used During NRFU

Enterprise Event Code Description	Event Code	NRFU Event Code Description
Response acquired	1.010	Complete
Response acquired by proxy	1.021	Complete by proxy
Form provided to interviewer	1.030	Form provided to enumerator
Partial or break-off with insufficient information	1.040	Insufficient Partial
Partial or break-off with insufficient information by proxy	1.041	Insufficient Partial by Proxy
Partial or break-off with sufficient information	1.050	Partial sufficient
Partial or break-off with sufficient information by proxy	1.051	Partial sufficient by proxy
Refusal & break-off	3.001	Refusal
Not attempted	3.020	Not attempted
Non-contact	3.021	No one at home
Non-contact proxy attempt	3.022	Proxy not at home -- can recontact
Occupied by Manager visit	3.025	Occupied by MV
Other eligible	3.050	Other eligible
Language barrier	3.062	Language barrier
Hearing barrier	3.063	Hearing barrier
Nothing known about respondent or address	4.000	Nothing known about address
Non-contact	4.011	No one answers -- telephone
Non-contact proxy	4.012	Proxy cannot provide information -- do not attempt proxy again
Non-contact Manager	4.013	No manager exists to be interviewed
Non-contact Manager not present	4.014	A manager cannot be found at this time
Unable to reach	4.030	Restricted access
Unsafe area	4.031	Dangerous address
Unable to locate	4.032	Unable to locate
Other unknown eligibility	4.090	Other unknown eligibility
Unknown eligibility by Manager visit	4.091	Unknown Eligibility by MV
Nonresidence	5.040	Nonresidence
Group quarters	5.043	Group quarters
Vacant	5.047	Vacant -- needs verification
Vacant by proxy	5.048	Vacant by proxy
Vacant by Manager	5.049	Vacant by MV
Multi-unit, missing unit designation	5.051	Multi-unit, missing unit designation
Delete	5.062	Delete -- needs verification
Delete by Manager	5.063	Delete by MV
Duplicate listing	5.080	Duplicate listing
Does not exist	5.081	Does not exist
Demolished	5.082	Demolished
Uninhabitable	5.084	Uninhabitable
Empty mobile home site	5.085	Empty mobile home site
Other not eligible	5.090	Other not eligible
Map spot verified	13.000	Map spot verified
Map spot unverified	13.001	Map spot unverified
Map spot other	13.002	Map spot other

18. References

Hatcher, Jr., W. W. (2015), “Assessment of the 2015 Census Test,” July 30, 2015.

Gibb S., King, R., Fitzsimmons, F., Wands, T., & Nabourema, F. (2017), “2020 Research and Testing: 2016 Census Test Nonresponse Followup Assessment Report,” Draft, March 20, 2017.

Poehler, E., Cronkite, D., Sanchez, P., Wakim, A., Dusch, G., Walrath, H., King, R., & Jones, J. (2015), “2020 Census Research and Testing: 2014 Census Test Nonresponse Followup Panel Comparisons and Instrument Analysis,” Version 1.0, July 21, 2015.

U.S. Census Bureau (2012), “2010 Census Program for Evaluations and Experiments: 2010 Census Nonresponse Followup Operations Assessment Report,” 2010 Census Planning Memoranda Series, No. 190., April 23, 2012.

U.S. Census Bureau (2014), “2020 Research and Testing: 2013 Census Test Assessment,” Version 2.1, May 28, 2014.

U.S. Census Bureau (2017a), “2018 End-to-End Census Test: Goals, Objectives, Success Criteria (GOSC) and Research Questions,” Version 2.4, April 24, 2017.

U.S. Census Bureau (2017b), “2018 End-to-End Census Test: One-Pager,” Version 1.5, April 24, 2017.

U.S. Census Bureau (2017c), “2018 End-to-End Census Test Plan,” Version 0.3, March 31, 2017.

U.S. Census Bureau (2017d), “2018 NRFU Contact Strategies,” May 4, 2017.

U.S. Census Bureau (2017e), “2020 Census Detailed Operational Plan for the Nonresponse Followup Operation,” Version 0.9, April 14, 2017.

1 All NRFU cases will undergo AR modeling before the start of the operation to define their contact strategies. AR modeling will assign one of four statuses to each case: 1) occupied, 2) vacant, 3) not a housing unit, or 4) no determination.

2 The term “attempt day” is used throughout this document to refer to a unique day in which a NRFU case receives contact attempts by an enumerator. Within an attempt day, a case may receive multiple contact attempts; however, most cases will only have one contact attempt per attempt day.

3 A Type of Enumeration Area (TEA) is a set of blocks that are expected to be enumerated via the same methodology (e.g., self-response).

4 NRFU workload estimates for the Update/Leave TEA, for NRFU Reinterview, and for the supplemental NRFU workload are currently unknown.

5 For more information on the four components of RI, see the “2018 End-to-End Census Test: Quality Control Plan for the Nonresponse Followup Operation,” June 30, 2017.