OMB Appendix C Pilot evaluation logical framework

Pilot Evaluation

Narrative

Indicators of success

Means of verification

Assumptions

Purpose of Pilot

1.

1a. Survey data is of good
quality (e.g. limited missing or
invalid data).

1a. Item response analysis

•

1b. Lab data (e.g. Positive
controls are within acceptable
standard deviation, negative
controls are negative)

•

To determine if study procedures
allow team to answer research
questions

1b. Lab data is of good quality
(e.g. limited missing or invalid
data).

2.

Low Pressure Event
Outcomes

1.

To familiarize team members with
study implementation and
procedures

CDC team and utility select
appropriate exposed and
unexposed areas.

1c. LPE exposure characteristics
are available for each event.

1c. Item response analysis

2a. CDC team conducts 1-2 onsite training sessions.

2a. & 2b. Observed behavior and
on-site training activities

2b. All team members follow
study standard operating
procedures.
1a. Exposed and unexposed
areas are matched on census
block or tract, housing type, pipe
size and material, and origin of
water.

1a. Census maps and utility
water systems information

1b. CDC team receives customer
contact information within 1week of the event.

2.

LPE form is filled out completely at
the time of repair.

3.

Water sample collection team
follows standard operating
procedures.

2. Water sample information is
available for all 6 samples, chain
of custody information is
complete, low pressure event
information is complete.
3a. Utility personnel collect
water samples from the exposed
area within 24 hours of the LPE,
using CDC protocols, and CDC
receives samples within 30
hours (EPA standard holding
time for bacterial analytes).
3b. Utility personnel collect
water samples from the
unexposed area within 48 hours
of the LPE and CDC receives

•

•
•

1b. Contact data management
database used to monitor
receipt of customer information
from utilities and secure FTP site
used for transfer of customer
information
2. LPE form (with chain of
custody information on last
page)

3a. & 3b. CDC lab chain of
custody information received
with shipments (on LPE form),
lab data management database

•

Utility and CDC team
communicate throughout
the pilot
Utility follows study
procedures
CDC team follows study
procedures

Team members follow study
procedures
No technical issues with
data management
databases
No external issues that
would prevent timely
shipment of water samples

4.

Lab team follows testing, standard
operating procedures, and
maintains chain of custody logs.

samples within 30 hours after
collection.
4a. CDC Environmental
Microbiology Laboratory
processes and begins analysis
the same day the samples are
received from the water utility
(36 drinking water samples from
6 LPEs, with 6 samples collected
by utility collaborators for each
event).
4b. All water sample chain of
custody logs (on LPE forms) are
up to date.

5.

CDC team follows survey mailing
procedures and tracks household
participation.

4c. Lab data management
database is up to date.
5a. CDC contacts at least 200
affected and 400 unaffected
households following
approximately 6 low pressure
events.

4a. CDC lab chain of custody
information on LPE form, lab
data records, and lab data
management database

4b. CDC lab chain of custody
information received with
shipment (on LPE form)
4c. Lab data management
database
5a. & 5b. Contact data
management database used to
monitor receipt of customer
information from utilities

5b. Contact data management
database is up to date with
mailing schedule.

6.

Survey procedures encourage
overall participation and equitable
study participation in exposed and
unexposed groups.

5c. Survey data management
database is up to date with
household participation
information.
6a. Overall survey unit response
rate is 60%.

5c. Survey data management
database used to monitor survey
responses and to house the web
data
6a. – 6c. Pilot non-response bias
evaluation

6b. < 20% difference in unit
response rates in exposed and
unexposed areas.
6c. < 90% non-response for
survey items.
6d. CDC obtains household
respondent data within a 6week time window.

6d. Survey data management
database used to monitor survey
responses and to house the web
data